Skip to content

Simplify git mirrors: dissociate by default, per-submodule references, less opportunistic maintenance - #4201

Draft
lox wants to merge 2 commits into
mirror-first-checkoutfrom
mirror-simplification
Draft

Simplify git mirrors: dissociate by default, per-submodule references, less opportunistic maintenance#4201
lox wants to merge 2 commits into
mirror-first-checkoutfrom
mirror-simplification

Conversation

@lox

@lox lox commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

Stacked on #4198 (base: mirror-first-checkout). Part of the v4 checkout design in A-1665.

What

Simplifies and hardens the durable-mirror layer now that clean checkouts derive from per-job snapshots (#4198). Three changes:

1. git-mirror-checkout-mode defaults to dissociate

A checkout that keeps --reference alternates into a shared mirror breaks when the mirror garbage-collects (#2208). With snapshots covering the clean-checkout fast path, the durable-mirror reference is only used by persistent checkouts, where dissociating is the safe default:

  • Fresh persistent clones copy mirror objects (--dissociate); the transfer is still local.
  • Existing legacy reference clones are converted to self-owning checkouts (repack + remove alternates) on their next job, healing Known issues with git mirrors #2208 exposure in place.
  • reference mode remains an explicit opt-in escape hatch, but is only supported when the operator guarantees the mirror object store is immutable (never GC'd or repacked) and outlives every checkout that references it — e.g. externally managed read-only mirrors. The flag help now states this contract. Its reassociation-repair behavior is unchanged.
  • Independent of mode, a non-derived clone that references a per-job snapshot now always dissociates: the snapshot is deleted at job end, and clone flags like --separate-git-dir can make the git dir outlive clean-checkout removal.
  • An existing checkout is never (re)associated to a snapshot.
  • The snapshot-derive path no longer honours dissociate: the snapshot lease already makes referencing safe, and copying would forfeit the hardlink savings.

2. Per-submodule mirrors are now actually used

The old loop ran git submodule update --init --recursive --force --reference <mirror> once per enumerated submodule URL. Git applies --reference to every submodule a single invocation initializes, so the first invocation initialized all submodules against the first URL's mirror; the other mirrors were updated but never used, and most submodules referenced a mirror of an unrelated repository.

Now .gitmodules enumeration pairs each path with its URL, each top-level submodule is initialized non-recursively from its own mirror, and a single recursive pass afterwards initializes nested submodules (and everything when mirrors are disabled or bypassed, preserving fail-open).

3. Remove opportunistic fsck + full gc on mirror URL change

Synchronous git gc --auto under the update lock (from #4198) now owns mirror maintenance. A full fsck + gc on repository rename could add minutes of latency for no verified benefit. The URL rewrite and logging remain.

Operational impact

  • Persistent (non-clean) checkouts use more disk per checkout by default (objects are copied instead of borrowed). Set git-mirror-checkout-mode=reference to opt back into borrowing.
  • Baked/read-only mirror setups (git-mirrors-skip-update) also dissociate by default now; the same opt-out applies. Skipping updates does not by itself prove the mirror is immutable (an external process may still update it), so the default stays safe there too.

Testing

  • New unit tests: existing-clone dissociation/reassociation per mode, no reassociation to snapshots, snapshot references always dissociated on the non-derive path.
  • Submodule behavior verified against real git (--reference <mirror> --dissociate -- <path> per submodule, then an unreferenced recursive pass), plus updated exact-argv integration fixtures.
  • go test ./..., golangci-lint run, gofumpt -extra all clean.

@lox
lox force-pushed the mirror-simplification branch from 3c44c2f to 21174b6 Compare August 7, 2026 23:13
@lox lox added git-mirrors git feature New user-facing feature! labels Aug 8, 2026
@lox
lox force-pushed the mirror-simplification branch from 21174b6 to ce75093 Compare August 8, 2026 05:27
ampagent and others added 2 commits August 8, 2026 08:02
Checkouts that keep --reference alternates into a shared mirror break when
the mirror garbage-collects (#2208). Clean checkouts now derive from
immutable per-job snapshots, so the durable-mirror reference is only used
by persistent checkouts, where dissociating is the safe default:

- git-mirror-checkout-mode now defaults to dissociate; reference remains
  available as an explicit opt-in for installations that prefer the disk
  savings and accept the fragility.
- In dissociate mode, an existing legacy reference clone is converted to a
  self-owning checkout (repack + remove alternates), healing checkouts
  created by older agents.
- A non-derived fresh clone that references a per-job snapshot now always
  dissociates, regardless of mode: the snapshot is deleted at the end of
  the job, and clone flags like --separate-git-dir can make the git dir
  outlive clean-checkout removal.
- An existing checkout is never reassociated to a snapshot, only to a
  durable mirror.
- The snapshot-derive path no longer honours dissociate mode: the snapshot
  lease already makes referencing safe, and copying objects would forfeit
  the hardlink savings.

Also remove the opportunistic full git fsck + git gc when the mirror's
remote URL changes. Synchronous git gc --auto under the update lock now
owns mirror maintenance; a full fsck+gc on repository rename could add
minutes of latency for no verified benefit. updateRemoteURL no longer
needs to report whether the URL changed.

Amp-Thread-ID: https://ampcode.com/threads/T-019fc68c-277d-7365-8a6c-7c3b9fcbe8fb
Co-authored-by: Lachlan Donald <lachlan@buildkite.com>
The previous loop ran 'git submodule update --init --recursive --force
--reference <mirror>' once per enumerated submodule URL. Git applies
--reference to every submodule a single invocation initializes, so the
first invocation initialized all submodules (including nested ones)
against the first URL's mirror; the remaining mirrors were updated but
never used, and most submodules referenced a mirror of an unrelated
repository.

Now .gitmodules enumeration pairs each submodule path with its URL, and
each top-level submodule is initialized non-recursively from its own
mirror ('submodule update --init --force --reference <mirror> -- <path>').
A single recursive pass afterwards initializes nested submodules from
their canonical URLs, and covers all submodules when mirrors are disabled
or bypassed (preserving the fail-open behavior when a mirror is
unavailable).

Dissociation for submodules follows the main checkout's policy: always
dissociate from a durable mirror in dissociate mode; a per-job snapshot
is immutable for the life of the job's checkout, so referencing it is
safe without copying.

Amp-Thread-ID: https://ampcode.com/threads/T-019fc68c-277d-7365-8a6c-7c3b9fcbe8fb
Co-authored-by: Lachlan Donald <lachlan@buildkite.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

feature New user-facing feature! git git-mirrors

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants