meta(ci): reduce critical-path latency and runner work without weakening coverage

[davidgmbb]

Objective

Reduce time to a trustworthy CI result and the runner work spent obtaining it, preserving the complete supported compiler/configuration/test contract.

This is a coordination and acceptance meta-issue, not a replacement for #333, #335, #408, #409, #603, #604, or #413. Keep implementation, ownership, and exact-source evidence in those existing issues and the two newly recorded investigations, #707 and #708. Do not create another scheduler, coverage inventory, benchmark service, or generic optimization epic.

Primary implementation priority: #333, targeting Windows x86-64 and Intel macOS together. Investigate #707's configuration startup and #708's Windows driver-test tail alongside it. Complete existing qualification work instead of reimplementing landed changes.

Read AGENTS.md, docs/agents/build.md, docs/agents/testing.md, and docs/agents/workflow.md first. This campaign preserves the single optimized Release unity translation unit and native object, all supported platform/configuration coverage, and the native build driver's orchestration authority.

Evidence and scope of the baseline

Research inspected source at 1a6def6f69faba8a9c6e515f28a0da029a4c8417; the filing refresh observed main b8dd9857c0b8b5dcfc4f1e45159cf4155d26b19f, tree cdb29eb88812caa306bc118add664f2be2c4e9af, after merged #704.

The timing observation is a different, explicitly pinned revision: successful main/push Buster CI 35145639100, attempt 1, source 76dbb1014cf87eab5d8aae32fe8b0bdabee84b8e. It ran September 16, 2026, 20:17:15–20:41:43 UTC: 1,468 s / 24m28s.

Lane	Complete job	Combination step	Evidence
macOS x86-64	24m16s	22m46s	104960818514
Windows x86-64	23m37s	22m24s	104960818543
Linux x86-64	18m05s	17m28s	104960818506
Linux AArch64	16m53s	16m14s	104960818474
macOS x86-64 native	12m17s	Separate mode/differential work	104960818435
Clang analyzer shards	8m36s	Separate analyzer work	104960818135

This is one successful observation, not a before/after experiment, stable median, or complete runner-cost census. Separate workflow-level CI completion from all-required-workflow completion and from human review/merge time. The analyzer number is for a push; #616's same-revision selection means it cannot be transferred to a PR/merge-group comparison cohort.

Windows finished at 20:40:55 and Intel macOS at 20:41:36: only 41 seconds apart. Under that observed schedule, making only the Mac lane arbitrarily faster removes at most that gap from the latest prerequisite finish. Improving both exposes Linux x86-64 as the next roughly 18-minute bound. These are scheduling deductions, not additive speedup promises. The critical jobs' creation-to-start delays were only two and four seconds in this sample; capacity problems in other cohorts remain unmeasured.

Work map: retain owners and distinguish landed implementation from acceptance

Workstream	Existing implementation / evidence	Remaining campaign action
#333: desktop combination partitioning	#410 already separated native suites; #612's desktop coverage prerequisite is merged. The existing continuation says a partition candidate was being built.	Continue that ownership; implement deterministic, disjoint/exhaustive compiler/configuration partitions and measure end-to-end latency plus cost. No competing scheduler branch.
#335: effective coverage	Merged #612 supplies authoritative desktop row identities and strict completion evidence; it does not complete the whole mobile contract.	Reuse the desktop interface for #333. Preserve mobile/native execution classifications; keep broader coverage closure under #335.
#707: Zig configuration barrier — NEW	Windows Zig generation 139.810883 s, versus 1.35–3.45 s for its other compiler trees; Intel Mac Zig generation 60.329349 s. Download-archive setup is separate.	Attribute actual configure child work, then remove demonstrated redundancy or qualify a narrowly scoped optimization. Configuration/build overlap stays coordinated with #333.
#708: Windows sanitized Debug driver-test tail — NEW	compiler_driver_tests takes 414.4824948 s, with 28,772 assertions and zero failures, in the required sanitized Debug row.	Attribute fixtures and child operations; preserve coverage and shared worker budgets. Do not mistake a long successful module for a correctness failure or removable duplicate.
#604: native producer profile	Published screening selects debug-info-only profile C for further qualification, with producer-build reductions around 14–22%; guarded/full-job acceptance was explicitly incomplete.	Finish the existing candidate's guarded compiler-throughput and full-native-job qualification. Preserve local profiling defaults, frame pointers, optimized unity, diagnostics and matching normal/recovery generation.
#603: analyzer comparison policy	Merged #616 removes eligible same-revision duplication. #603 is open/reopened; historical dependency closure and further relevance policy remain distinct.	Freeze genuine reference dependency closure and qualify any additional selection policy conservatively. Always retain required candidate analysis, unity coverage, failure controls and aggregation.
#408: native differential	Bounded workers and production rollout #618 are merged.	Retain prior source-bound evidence and finish/reconcile remaining issue acceptance; do not implement four-worker rollout again or sum its historical saving with another cohort.
#409: evidence packaging	#414 and #468 already bundle diagnostics and publish complete, failure-aware evidence.	Reconcile remaining matched package-plus-upload acceptance; do not write another packer.
#413: Apple Clang unity compilation	Historical trace exists; its main source target overlaps #36-owned direct-native code and has changed.	Fresh current-source profiling and ownership qualification before a source optimization. Not a prerequisite to every independent CI change.

#333 design constraint: Intel Mac direct scheduling

The retained Intel Mac log uses separate direct build/test invocations for unsanitized Release, sanitized Debug/Release, GCC Debug and Zig Debug. Windows uses the four-slot superbuild with one-job tree quotas. The inspected coverage policy explicitly treats Intel macOS as direct scheduling.

Investigate why that exception exists before changing it. Within #333, compare the retained direct path, a supported pooled alternative, and semantic external partitions against the same authoritative required rows. Do not assume flipping the exception is safe, demand platform-identical scheduling for its own sake, or split the Release compiler into multiple objects. More CI jobs are only useful when queue/setup/duplicate work and resource pressure do not erase the benefit.

Corrected reliability work, not new duplicate issues

#685 is not established as a cleanup-only failure. The complete-log diagnosis preserved in merged #704 identifies Debug payload deadline failures followed by Release success in all three originally cited failing attempts. The batch was already failed before final cleanup. #686 supplied production status/lifecycle work; #704 now exposes configuration and payload/cleanup/overall status separately.

Reuse those changes. Do not weaken cleanup or accept a final Release marker as evidence that Debug passed. Coordinate with #676's existing debug-location work where actually relevant; no Windows root cause is inferred from an Android report.

Keep separately owned mobile/coverage and wrapper work (#331/#332/#334/#335, #394/#636, #701) visible, but do not make completion of every broader feature ticket a new dependency of this bounded speedup campaign. Count retry waste and failed/incomplete attempts as operational outcomes, not fast successful samples.

Delivery sequence and coordination

A — Establish one comparable cohort. Reuse tools/github_ci_time.py, existing step/module timings, and retained manifests. Record all job pages and attempts, the exact required workflow set, and the chosen cost/latency goals before timing candidates. Extend existing tools only when a required field is missing. No dashboard or daemon is needed.

B — Work the co-critical desktop lanes. Advance the existing #333 continuation against merged #612. Run #707/#708's read-only attribution in parallel, then agree ownership of any actual shared-code repair. Keep complete compiler/configuration trees together where splitting would duplicate expensive producers; measure rather than assume the optimal boundary.

C — Finish bounded existing savings. Continue #604's selected profile and #603's remaining comparison work. Preserve #408/#409's landed implementations and distinct acceptance. Reassess analyzer and Linux bottlenecks after each integrated step instead of optimizing from stale rankings.

D — Validate the combined result. Individually useful PRs may land independently after their own review and gates. Remeasure the integrated rollout against an equivalent baseline; record remaining bottlenecks and rejected candidates. Never calculate whole-workflow savings by adding unrelated phase reductions.

The campaign goal is to move the two approximately 24-minute desktop lanes below the next demonstrated bottleneck, then re-rank. This is a direction, not an approved numerical budget or measured result. Record the practical latency target and permitted runner-cost change before the first acceptance cohort; any deliberate cost increase requires a visible justification, not post-hoc threshold widening.

Non-negotiable validation contract

• Coverage: driver-owned required row sets; deterministic unique shard ownership; exact source/driver/toolchain binding; completion rejects missing, duplicated, stale, partial, skipped, failed or cancelled required coverage. Aggregate assertion counts alone are not equivalence evidence. Preserve independent references, sanitizers/fuzz, self-host, mode, platform and negative controls.
• Aggregation: preserve explicit status checking after every dependency outcome. GitHub's documented needs behavior can skip dependent jobs after failure/skip unless their condition permits continuation. A final job merely being declared is not a fail-closed proof. Keep optional/inapplicable companion jobs distinct from missing required work, and do not infer every companion workflow is covered by CI complete.
• Resources: account for outer build jobs, inner Ninja jobs and test workers together; preserve BUSTER_TEST_JOBS and the existing matrix budget. Record actual memory/CPU observations; virtual arena reservations are not measured RSS. No unrestricted second worker pool or benchmark-host admission bypass.
• Sampling: at least three comparable successful complete runs per baseline/candidate arm as required by ci: shard the desktop combination matrix to reduce the critical path #333, with every attempted failure/cancellation/incomplete run also retained. Predeclare attempt/cohort rules; no rerun-until-green selection. Separate PR, push, merge-group, cold/warm cache, runner image and toolchain populations. Three samples are a minimum operational check, not automatic statistical significance or a defensible p95 estimate.
• Metrics: workflow creation to terminal gate; job creation-to-start queue delay; execution span; sum of actual job execution durations; configure/build/link/test/package/upload phases; duplicated producer work; cache transfer/verification cost; CPU and peak simultaneous RSS where available. Keep wall, CPU and runner seconds separate. Report unavailable fields as unknown, not zero. Broader exact-head required-workflow latency and retry work remain separate columns.
• Performance: use matched uninstrumented builds for acceptance; diagnostic traces only explain attribution. When producer flags/source change, validate the resulting compiler's throughput as well as host build cost. Preserve the applicable existing performance contract; this meta-issue approves no new thresholds.
• Reproducibility: exact reproduction and rollback per PR; retained raw logs, coverage identities and rejected attempts, with durable preservation of selected acceptance evidence before artifact expiry. Do not rely solely on a ChatGPT attachment or an expiring green badge.

Completion checklist

• Publish the current owner/continuation and bounded deliverable for each selected workstream; retain ci: shard the desktop combination matrix to reduce the critical path #333's existing ownership and ci: expose authoritative desktop coverage manifests #612's authority.
• Record the baseline cohort, exact source/toolchain/coverage identities, target latency and runner-cost tolerance before candidate acceptance.
• Land at least one measured coverage-preserving critical-path improvement and verify its complete integrated CI result; a documentation-only meta-issue is not completion.
• Give ci: investigate and reduce the desktop Zig configuration barrier #707 and ci(windows): attribute and reduce the sanitized Debug driver-test tail #708 a current-source attribution and explicit disposition: implemented/qualified, duplicate routed to its owner, evidence-backed no-change, or deferred with owner/reason.
• Reconcile selected ci: qualify an explicit native Release producer profile instead of inheriting local profiling defaults #604/ci: stop repeating the full analyzer baseline campaign in ordinary correctness runs #603/test: reduce native differential latency with bounded case workers and reusable ABI caller objects #408/ci: bundle desktop differential evidence losslessly before artifact upload #409 acceptance without mistaking a partial PR for full parent closure; keep build: reduce measured Apple Clang Release unity compile cost without splitting #413 conditional on a useful independent current profile.
• Publish whole-workflow and aggregate runner-cost results, all attempts and limitations, proof of unchanged required coverage, and failure/cancellation/resource controls.
• Record the next actual bottleneck and link any remaining scoped owners; close this bounded campaign by an explicit evidence-backed disposition, not by automatically closing its broader children.

No production dispatch/deletion, #36 support manifest, retirement budget, dedicated-host service, branch protection, runner purchase, dependency upgrade, or unrelated source refactor is authorized by this tracking issue. No code or CI run was produced while filing it. New execution in this task was limited to local archive hashing/decoding and evidence inspection; the timing results remain from the pinned retained run.

[davidgmbb]

Campaign advance — landed partition, current attribution, and active evidence path

[Created by ChatGPT]

The issue’s original sequencing is now stale in one important respect: the primary #333 implementation has landed.

Current owners and bounded deliverables

Workstream	Current state	Bounded next deliverable for this campaign
#333	Implemented and closed by merged #712 (fb008bb377d5bbf198b6b673c6797cdc5d8fead3). The six desktop lanes are now twelve release/checks shards with exact disjoint/exhaustive row ownership and a fail-closed 23-job completion gate.	Build the matched integrated baseline/candidate cohort; do not reopen or build a competing scheduler. Count the duplicated checkout/setup/archive cost as well as elapsed time.
#707	Open. PR #718 adds bounded retention of per-tree CMake evidence and an immutable opt-in cmake_profile dispatch input; it changes no matrix ownership or compiler result.	Land the evidence path after exact-head integration succeeds, then obtain separate Windows x86-64 and Intel macOS profiles plus controlled cold/warm Zig compilation-cache runs. No configure speedup is claimed yet.
#708	Instrumentation merged in #723. First successful current Windows run 35164305219 has now been decoded and summarized on #708.	Add child/operation attribution first to compiler_driver_test_native_frame_vectors and compiler_driver_test_machine_fallback; only then qualify a repair with repeated uninstrumented runs.
#604 / #603 / #408 / #409	Existing ownership remains unchanged.	Re-rank these only after the integrated #712 layout is measured; do not add historical phase savings from different cohorts.

New #708 evidence and disposition

Run 35164305219, attempt 1, executed all six required Windows rows successfully. Its compiler_driver_tests module intervals were 156.682 s for unsanitized Release, 163.345 s for sanitized Release, and 484.588 s for sanitized Debug.

The sanitized-Debug interval is concentrated rather than diffuse:

• compiler_driver_test_native_frame_vectors: 258.587 s, about 53.4% of the module;
• compiler_driver_test_machine_fallback: 69.537 s, about 14.4%;
• together: about 67.7% of the module;
• the next tier is wide_vector_boundaries 33.661 s, many_native_arguments 24.122 s, and native_i128_divide 23.749 s.

This is an explicit deferred-with-owner disposition, not a no-change decision: #708 owns child compile/execute/process/filesystem attribution inside the first two fixtures. Inclusive wall time under the concurrent four-tree matrix is not removable CPU time, and no scheduling or coverage change is justified by fixture totals alone.

#718 integration state

PR #718 currently points at 20c58c8d20de9f62a217949add544aa46b3e3ba9. GPU acceptance, native-retirement contract, compiler throughput, self-host fixed point, iOS lifecycle, workflow lint, every completed desktop shard, and the collector itself are green. It has no review submissions or unresolved review threads.

Buster CI run 35169885325 attempt 2 is still terminalizing. One independent UEFI job recorded a retained failure: the AArch64 fast positive control timed out after 120 s before firmware reached the boot-device message, while all four x86-64 cases, AArch64 none/mir-stack/quality, and the AArch64 fast negative control passed. The failure is outside #718’s changed files and the artifact is retained as uefi-boot-35169885325-2 (10475808499). It will be retried as the individual failed job after the current run becomes terminal; the failed attempt remains part of the record.

Acceptance status

This advances current ownership and gives #708 a current-source attribution, but it does not satisfy #709’s performance-completion gates yet. There is still no predeclared matched 3+3 uninstrumented cohort for the integrated #712 layout, no aggregate runner-second comparison, and no demonstrated whole-workflow speedup. The next measurement should use the landed shard identities and retain every attempt, with PR/push/merge-group, runner image, toolchain, cache state, queue delay, execution span, per-shard phase time, and total runner time kept separate.

[davidgmbb]

First integrated post-#333 observation: latency improved, runner work regressed

[Created by ChatGPT]

I compared the issue's pinned successful baseline, run 35145639100 at source 76dbb1014cf87eab5d8aae32fe8b0bdabee84b8e, with the first successful main/push run of the landed #712 partition, run 35165575110 at source fb008bb377d5bbf198b6b673c6797cdc5d8fead3.

This is a useful integrated observation, not the predeclared matched 3+3 acceptance cohort required by this issue. The sources differ and hosted-runner noise is uncontrolled.

Metric	Pinned baseline	First integrated #712 run	Delta
Workflow creation to terminal run state	1,468 s / 24m28s	1,318 s / 21m58s	-150 s / -10.2%
Critical desktop job	macOS x86-64, 1,456 s / 24m16s	macOS x86-64 checks, 1,302 s / 21m42s	-154 s / -10.6%
Windows x86-64 critical job	1,417 s / 23m37s	checks, 1,132 s / 18m52s	-285 s / -20.1%
Sum of all job execution spans	9,802 s / 2h43m22s	11,729 s / 3h15m29s	+1,927 s / +19.7%
Sum of the six desktop lanes / twelve desktop shards	6,182 s / 1h43m02s	7,863 s / 2h11m03s	+1,681 s / +27.2%

The landed partition therefore has the expected shape in this sample: better result latency, materially higher runner work. It meets neither a runner-cost acceptance claim nor the issue's combined completion gate. The increase is not automatically a regression in the implementation—#712 deliberately duplicates checkout/setup/tool installation and some producer work across release and checks shards—but it must be treated as a visible cost, not hidden behind the latency result.

Re-ranked bottleneck and bounded next action

The current critical path is now Intel macOS checks at 21m42s. Windows x86-64 checks is next at 18m52s; Linux AArch64 checks is 15m31s, Linux x86-64 checks 15m11s, and macOS AArch64 checks 14m40s in this run.

The next #333/#709 optimization should therefore target duplicated shard setup and producer work without recombining the latency-critical test ownership. In particular:

1. use ci: investigate and reduce the desktop Zig configuration barrier #707's retained configure evidence to quantify duplicated Zig/CMake startup independently on Intel macOS and Windows x86-64;
2. identify which build products can be transferred with exact source/toolchain/configuration binding and cheaper verification than rebuilding;
3. retain disjoint/exhaustive row ownership and the 23-job fail-closed completion contract;
4. reject any cache/artifact transfer whose upload/download/verification cost or stale-input risk erases the measured gain.

A candidate acceptance cohort must still predeclare exact source, event, runner images/toolchains, cache state, all attempts, and permitted runner-cost change. Until then the practical disposition is: #333 landed a one-observation latency improvement; aggregate cost is the next explicit owner, and #709 remains open.