Simplify git mirrors: dissociate by default, per-submodule references, less opportunistic maintenance - #4201
Draft
lox wants to merge 2 commits into
Draft
Simplify git mirrors: dissociate by default, per-submodule references, less opportunistic maintenance#4201lox wants to merge 2 commits into
lox wants to merge 2 commits into
Conversation
lox
force-pushed
the
mirror-simplification
branch
from
August 7, 2026 23:13
3c44c2f to
21174b6
Compare
lox
force-pushed
the
mirror-simplification
branch
from
August 8, 2026 05:27
21174b6 to
ce75093
Compare
Checkouts that keep --reference alternates into a shared mirror break when the mirror garbage-collects (#2208). Clean checkouts now derive from immutable per-job snapshots, so the durable-mirror reference is only used by persistent checkouts, where dissociating is the safe default: - git-mirror-checkout-mode now defaults to dissociate; reference remains available as an explicit opt-in for installations that prefer the disk savings and accept the fragility. - In dissociate mode, an existing legacy reference clone is converted to a self-owning checkout (repack + remove alternates), healing checkouts created by older agents. - A non-derived fresh clone that references a per-job snapshot now always dissociates, regardless of mode: the snapshot is deleted at the end of the job, and clone flags like --separate-git-dir can make the git dir outlive clean-checkout removal. - An existing checkout is never reassociated to a snapshot, only to a durable mirror. - The snapshot-derive path no longer honours dissociate mode: the snapshot lease already makes referencing safe, and copying objects would forfeit the hardlink savings. Also remove the opportunistic full git fsck + git gc when the mirror's remote URL changes. Synchronous git gc --auto under the update lock now owns mirror maintenance; a full fsck+gc on repository rename could add minutes of latency for no verified benefit. updateRemoteURL no longer needs to report whether the URL changed. Amp-Thread-ID: https://ampcode.com/threads/T-019fc68c-277d-7365-8a6c-7c3b9fcbe8fb Co-authored-by: Lachlan Donald <lachlan@buildkite.com>
The previous loop ran 'git submodule update --init --recursive --force
--reference <mirror>' once per enumerated submodule URL. Git applies
--reference to every submodule a single invocation initializes, so the
first invocation initialized all submodules (including nested ones)
against the first URL's mirror; the remaining mirrors were updated but
never used, and most submodules referenced a mirror of an unrelated
repository.
Now .gitmodules enumeration pairs each submodule path with its URL, and
each top-level submodule is initialized non-recursively from its own
mirror ('submodule update --init --force --reference <mirror> -- <path>').
A single recursive pass afterwards initializes nested submodules from
their canonical URLs, and covers all submodules when mirrors are disabled
or bypassed (preserving the fail-open behavior when a mirror is
unavailable).
Dissociation for submodules follows the main checkout's policy: always
dissociate from a durable mirror in dissociate mode; a per-job snapshot
is immutable for the life of the job's checkout, so referencing it is
safe without copying.
Amp-Thread-ID: https://ampcode.com/threads/T-019fc68c-277d-7365-8a6c-7c3b9fcbe8fb
Co-authored-by: Lachlan Donald <lachlan@buildkite.com>
lox
force-pushed
the
mirror-simplification
branch
from
August 8, 2026 08:04
ce75093 to
693637e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #4198 (base:
mirror-first-checkout). Part of the v4 checkout design in A-1665.What
Simplifies and hardens the durable-mirror layer now that clean checkouts derive from per-job snapshots (#4198). Three changes:
1.
git-mirror-checkout-modedefaults todissociateA checkout that keeps
--referencealternates into a shared mirror breaks when the mirror garbage-collects (#2208). With snapshots covering the clean-checkout fast path, the durable-mirror reference is only used by persistent checkouts, where dissociating is the safe default:--dissociate); the transfer is still local.referencemode remains an explicit opt-in escape hatch, but is only supported when the operator guarantees the mirror object store is immutable (never GC'd or repacked) and outlives every checkout that references it — e.g. externally managed read-only mirrors. The flag help now states this contract. Its reassociation-repair behavior is unchanged.--separate-git-dircan make the git dir outlive clean-checkout removal.dissociate: the snapshot lease already makes referencing safe, and copying would forfeit the hardlink savings.2. Per-submodule mirrors are now actually used
The old loop ran
git submodule update --init --recursive --force --reference <mirror>once per enumerated submodule URL. Git applies--referenceto every submodule a single invocation initializes, so the first invocation initialized all submodules against the first URL's mirror; the other mirrors were updated but never used, and most submodules referenced a mirror of an unrelated repository.Now
.gitmodulesenumeration pairs each path with its URL, each top-level submodule is initialized non-recursively from its own mirror, and a single recursive pass afterwards initializes nested submodules (and everything when mirrors are disabled or bypassed, preserving fail-open).3. Remove opportunistic
fsck+ fullgcon mirror URL changeSynchronous
git gc --autounder the update lock (from #4198) now owns mirror maintenance. A fullfsck+gcon repository rename could add minutes of latency for no verified benefit. The URL rewrite and logging remain.Operational impact
git-mirror-checkout-mode=referenceto opt back into borrowing.git-mirrors-skip-update) also dissociate by default now; the same opt-out applies. Skipping updates does not by itself prove the mirror is immutable (an external process may still update it), so the default stays safe there too.Testing
--reference <mirror> --dissociate -- <path>per submodule, then an unreferenced recursive pass), plus updated exact-argv integration fixtures.go test ./...,golangci-lint run,gofumpt -extraall clean.