Reduce CI binary build fan-in

[lox]

Description

The binary build matrix previously used one static Buildkite step key, so every dependency on build-binary waited for all 22 cross-platform binaries, even when the consumer downloaded only the Linux amd64 and/or arm64 artifacts.

Split the matrix into consumer-aligned cohorts and persist compilation/module caches for the e2e container without allowing Go test-result reuse.

Context

In post-#4209 main build 13869, E2E execution took 2.53 minutes, but its trigger first spent 3.39 minutes waiting on dependencies. A replay of 72 passed main builds estimated the removable binary fan-in at 1.18 minutes median / 2.06 minutes p90, measured from completion of both required core Linux binaries to completion of the full binary matrix.

Measured impact

In one directly comparable PR-build observation, build 13871 completed in 4.63 minutes versus 7.00 minutes for build 13868: a 2.37-minute / 34% lower wall-clock duration. This is a single-build comparison, not an estimate of typical speedup; total duration is affected by runner queueing and external-service variance.

Build 13871's graph-specific timings were:

• Both core Linux binaries ready: 0.55m
• Final binary cohort ready: 2.58m
• Core-to-all-binaries completion gap: 2.03m, the dependency delay removed for core-only consumers in this run
• E2E dependency wait: 0.57m, versus 3.60m in build 13868
• E2E child queue wait: 1.41m
• E2E child execution: 2.66m

The first cache-enabled E2E execution was effectively neutral against main build 13869's 2.53-minute execution, so no cache speedup is claimed. E2E uses -count=1 so successful test results are never reused; only Go compilation and module-download caches persist.

Build 13871 produced all 22 expected binary artifacts and all 22 expected attestation filenames. Its 2 core, 6 package, and 14 remaining binary jobs passed, as did E2E, version checking, bk CLI testing, all six Docker builds and twelve architecture-specific Docker tests, Debian/RPM packaging, and GitHub release assembly.

Simpler alternative comparison

Draft #4214 tested extracting only Linux/amd64 while leaving the release matrix fan-in unchanged. Its source diff is roughly half the size and E2E finished 2.87 minutes earlier, but the unchanged release consumers became runnable in one burst and queued for 2-3.2 minutes. Its build 13874 finished in 8.86m, versus 8.28m for this PR's similarly queue-constrained build 13872.

That single comparison was 0.58m / 7% slower overall: the draft's non-E2E tail finished at 8.85m versus 6.85m here. This supports retaining the consumer cohorts when overall completion time, rather than only early E2E feedback, is the goal.

Changes

• Split the binary matrix into three static-key cohorts while preserving all 22 targets and attestation filenames:
  • core Linux: amd64 and arm64
  • package Linux: architectures consumed by Debian/RPM builders
  • remaining release targets
• Narrow each consumer to only the cohorts containing the artifacts it downloads.
• Persist Go build and module caches for the E2E container.
• Disable Go test-result reuse with -count=1, preserving E2E execution semantics.

Testing

• Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
• Code is formatted (with go tool gofumpt -extra -w .)

No Go source changed. Validation performed:

• Parsed .buildkite/pipeline.yml with the repository's own pipeline uploader in dry-run mode with parse warnings rejected.
• Parsed .buildkite/pipeline.yml and .buildkite/docker-compose.yml as YAML.
• Programmatically expanded the old and new matrices and verified exactly 22 unique producing targets, 22 unchanged unique attestation filenames, and the same two intentional skips.
• Audited artifact consumers against their narrowed dependencies.
• Ran bash -n .buildkite/steps/e2e-tests.sh and git diff --check.
• Ran the complete Buildkite pipeline for build 13871 and verified its jobs and uploaded artifacts through the Buildkite API.

Fresh final-commit build 13872 passed. Its E2E child executed for 2.99 minutes with -count=1; all 22 binaries, all 22 attestation filenames, and every narrowed artifact consumer were verified again. Queue pressure raised total wall clock to 8.28 minutes, while the graph still removed 2.15 minutes of binary fan-in.

Disclosures / Credits

Amp analyzed recent Buildkite timing data, implemented the dependency and cache changes, and validated the pipeline and artifact graph. OpenAI's Oracle provided a read-only review of the release dependency design and identified the need to disable Go test-result caching.

[buildsworth-bk-app]

I didn't find any issues. This changes the release pipeline's artifact dependency graph, so I'm leaving it for human sign-off.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 13154, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.

About buildsworth

Model: gpt-5.6-sol with xhigh thinking.

How to request a review: Comment @buildsworth-bk review on the PR, or request buildsworth-bk as a reviewer.

Risk labels (how buildsworth classifies risk) — buildsworth classifies risk itself from the diff. To let it approve, grant L2 approval by mentioning @buildsworth-bk (see L2 approval grant):

• L1 — Low risk (dep bumps, docs/copy, lockfiles, small presentational fixes). buildsworth may approve by default.
• L2 — Standard risk (new UI, additive API fields, refactors). Approved only with an L2 grant; otherwise comment-only.
• L3 — High risk (auth, migrations, payments, secrets, perf-critical paths). Human review always required.

[buildsworth-bk-app]

I didn't find any issues in the updated revision. This changes the release pipeline's artifact dependency graph, so I'm leaving it for human sign-off.

Want to dig deeper?

Paste this into your agent to explore the findings from this review's Buildkite build:

Download the buildsworth logs from build 13161, then answer my questions about the findings.

Install the reading-buildsworth-logs skill to run this.