Skip to content

Reduce stored push document size and rollup scan cost across sink backends #1691

Description

@dcoric

Discussion issue, applies to every sink backend (fs, mongo, and the PostgreSQL sink proposed in #1687).

Every push row stores the complete diff twice. getDiff writes it to both steps[].logs (via step.log(diff)) and steps[].content (via step.setContent(diff)), and writeAudit persists the whole action document as-is in all backends. Large pushes therefore produce multi-megabyte rows, and everything that touches push documents pays for it:

  • The repo activity rollup scans all push rows on every repos page. Indexing can make the scan cheap on PostgreSQL, but it remains O(number of pushes) on every backend, and on mongo/fs it reads the full documents.
  • List endpoints and exports ship or filter over fat documents.
  • Storage grows with diff size rather than with metadata size.

Ideas worth weighing, none of which are backend-specific:

  1. Store the diff once, not twice: drop the duplicate from steps[].logs or make setContent and the log reference one value.
  2. Move diff content out of the push document into its own store or table, loaded only by the push-detail view.
  3. A retention or truncation policy for step logs and diff content on old, settled pushes.
  4. Maintain the repo activity rollup incrementally (a counters document or table updated on push transitions) instead of recomputing it by scanning all pushes.

Filing as a design discussion before any implementation; the immediate, index-level mitigations for the PostgreSQL sink are tracked separately.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions