[A-1638] Establish uncached Agent CI baseline - #4252
Draft
jamiemonserrate wants to merge 14 commits into
Draft
Conversation
Three changes to the Cache v2 setup, all aimed at what the warm builds so far did not cover. Drop the branch and commit parts from the Go build cache keys. A key ending in the commit can never match on a new commit, so `cache save` always reached building_archive and uploaded, even when the restore had just hit via fallback. Ending the key at the dependency checksum lets an unchanged graph resolve exactly, and save then short-circuits on "cache already exists" without building an archive -- the shape bk_cli and protobuf_tools already use. Across seven binary-matrix jobs in 14022 the save phase ran 6.0-20.5s (median 9.1s), so roughly 4.7min of runner time per build for the 27-job matrix. The trade is that an entry is sealed until go.mod changes and drifted packages are recompiled; go.mod moves on about 11% of commits, and a one-commit-stale cache still cross-compiled in 3.4-4.5s against ~90s cold. Every warm build measured so far (14016, 14019, 14022) was triggered by a comment-only commit, so none of them exercise real source drift -- that still needs confirming. Move the linux/amd64 build cache writer from lint to shard 0 of the amd64 test job. Lint's cache holds only what go mod tidy, go generate, assertzapper and golangci-lint compiled, never the compiled test binaries, so the amd64 test jobs restored a cache that could not speed up a test compile. It was the one platform whose aggregate runner time went up in 14016 (+27s) while ARM64 saved 2m35s and Windows 1m14s. A test run compiles the packages and their tests, so lint now restores that superset and saves only gomodcache. Cache clicommand/ACKNOWLEDGEMENTS.md.gz. go-licenses walks the whole dependency tree on each of the 27 binary jobs to regenerate one file that is a pure function of the dependency graph and the build target, at a very consistent 13.1-14.9s per job. Reuse is opt-in via an env var so local runs always regenerate. Also adds the agent's own os/arch to target_gocache, since a cross-compile cache holds host-compiled tool artifacts that two builder architectures sharing a key would thrash. Build cache key prefixes go to v2, so those four caches are cold on this build with no fallback; the build after this one is the one to measure. gomodcache is deliberately unchanged and stays warm. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Comment-only change: documents the e2e_gocache writer, the one entry that had no rationale. Cache keys are byte-identical to 37f97ef, so nothing is invalidated and this build runs against the caches 14027 populated. What this should show, against 14027: - target_gocache and acknowledgements restore as exact hits (fallback_used=false) rather than via fallback, and both saves short-circuit on "cache already exists" without building an archive. - The binary matrix loses the ~13-15s go-licenses walk per target, since generate-acknowledgements.sh reuses the restored 19 kB file. - The amd64 test job restores the 214 MB / 9,464-entry cache it saved in 14027, which includes the compiled test binaries the older lint-written cache (206 MB / 7,743 entries) never had. What it does not show: this is another comment-only commit, so there is no source drift. It measures the saving from exact keys, not the recompilation cost of a sealed entry. That still needs a warm build on a commit with real Go changes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The warm run showed lint's assertzapper pass going from 0.8s to ~31s, consistently across three builds (14028/14029/14030), while golangci-lint in the same job did not move. Lint's infrastructure had actually got faster, so this was not machine speed: it was the cache contents. Two changes in 37f97ef interacted badly. Moving the linux/amd64 build cache writer to the test job meant lint began restoring a cache written by a job that never runs its analyzers, so their results were absent and the module was re-analysed from scratch. Ordering does not rescue this -- lint and the test jobs have no depends_on between them and both start at about the same second, so the test job restores long before lint has done any analysis. Making the keys exact removed the escape hatch. Under a per-commit key an entry was rewritten every build, so a cache could accumulate whatever the latest writer held. Ending the key at the dependency checksum means the save short-circuits once the entry exists, so the entry is frozen at whatever the first build after a dependency bump wrote -- in this case the test job, permanently. No later build can add the analyzer results. So the two producers need two keys. lint_gocache is owned and consumed by the lint step alone; gocache stays with the test jobs, which are the only producer of compiled test binaries. Each cache is now shaped for its consumer, and neither can be frozen in the wrong shape. The header gains the general rule, which is what was missing: one producer per key, and the producer must leave behind what the consumers need. Also corrects the "superset" claim in cache.yml and tests.sh, which was wrong -- a test-written cache is a superset for compilation but not for analyzer results. lint_gocache is a new key, so the next build is a cold miss for lint and that job will be slower than the ~71s just measured; the build after it is the one to read. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Every warm Cache v2 build so far (14016, 14019, 14022, 14028-14030) was triggered by a comment-only commit, so none of them exercise source drift. Ending the build cache keys at the dependency checksum means an entry is sealed until go.mod changes, and packages that churn in between are recompiled rather than accumulated into the cache. That recompilation cost is the one thing the measurements have not covered, and it is what decides whether dropping the per-commit key part was the right call. This adds an exported no-op to internal/job so the next build recompiles real code. Why here: internal/job is the second most frequently changed package in the repo (396 commits in the last 12 months, behind clicommand's 552), so the resulting recompilation is representative of an ordinary commit rather than a best or worst case. It also has the widest cascade of the churn hotspots -- agent, clicommand and both integration packages all depend on it -- so the measurement includes recompiling the largest package in the tree and its test binaries. Why an exported symbol rather than editing an existing body: a body-only change need not reach dependents, because Go's export data carries inlinable bodies but not all bodies. A new exported symbol always changes export data, so every dependent genuinely recompiles. Nothing calls it, so it cannot affect test output. Revert this once the drift measurement is recorded. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…change" This reverts commit b54aced. The measurement it existed for is recorded. Build 14032 carried the probe against warm caches; 14030 is the equivalent build without source drift. The binary matrix went 15.95m to 18.61m, so one commit's worth of drift costs about 5.9s per job across the 27 targets. The per-commit save it replaced ran a median 9.1s per job, so the recompilation is cheaper than the upload it removed, and 18.61m with drift still beats the 23.42m the old per-commit keys cost without any. Ending the build cache keys at the dependency checksum was the right call. The probe also confirmed the lint cache fix from a8648d7: assertzapper went from 30.7s reading a test-written cache to 6.0s reading its own, and the 6.0s residue is the drift this probe deliberately introduced. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Establishes an uncached CI timing baseline for the Cache v2 dogfooding work by removing the existing host-mounted Go build and module caches from the Docker-based jobs.
GOCACHEandGOMODCACHEcontinue to point to/gocacheand/gomodcache. Without the bind mounts, those locations are fresh container-local directories. This keeps the experiment focused on the effect of host cache persistence rather than changing Go cache paths at the same time.The affected path filters now include
.buildkite/docker-compose.yml, ensuring that Compose changes exercise lint, protobuf generation, and the full tests/coverage group.This PR will remain a draft while the pipeline runs and we record the uncached job timings.
Context
Changes
lintandagentCompose services.Testing
go test ./...). Buildkite employees may check this if the pipeline has run automatically.go tool gofumpt -extra -w .)docker compose -f .buildkite/docker-compose.yml config -q.buildkite/pipeline.ymlparses as YAMLgit diff --checkThe functional verification for this draft is the Buildkite pipeline itself. Its job durations will become the uncached baseline for the follow-up Cache v2 integration.
Deployment
No application deployment impact. If merged, Docker-based CI jobs will start with fresh Go caches rather than reusing caches from the agent host.
Rollback
Restore the four bind-mount entries in
.buildkite/docker-compose.ymlto return CI to the existing host cache behavior.Disclosures / Credits
Codex made the small configuration changes and drafted this PR with direction from Jamie Monserrate.