ci: qualify an explicit native Release producer profile instead of inheriting local profiling defaults

[davidgmbb]

Confirmed configuration mismatch on the critical path

Research snapshot: main 14da1343e96ea08007842bc9a67202e36c9ee02c, tree 250ac2b8d77eb48012e5db052a50d35c96207f62 (2026-09-13 Mexico City / 2026-09-14 UTC).

Current native CI steps generate the shared mode/differential producer with:

"$driver" generate --cc clang --config Release --linker DEFAULT

The missing-CMake-cache recovery path uses the same command. Neither supplies --ci or an explicit producer profiling policy.

I read the actual successful Intel macOS native job 103816332108, not just workflow names. Its log confirms:

BUSTER_CI: OFF
BUSTER_UNITY_BUILD: ON
BUSTER_INCLUDE_TESTS: ON
BUSTER_DEBUG_INFO: ON
BUSTER_FRAME_POINTERS: ON
BUSTER_SANITIZE: OFF
BUSTER_LTO: OFF

Host: macos-26-intel, Intel Core i7-8700B, four logical CPUs, 14 GiB RAM, AppleClang 21.0.0.21000101. The documented build policy deliberately enables debug information/frame pointers for local profiling, but disables those defaults in CI; CI also has a different optimized Release policy. This native lane currently inherits the local policy.

Observed cost, not an established fix

In first-attempt successful run 34791451301:

• Native job: 1,758 seconds / 29m18s.
• Execution-mode matrix step: 261 seconds / 4m21s.
• Log interval from machine-info immediately before the build (00:04:06 UTC) to completed producer link (00:07:54 UTC): approximately 229 seconds. Ninja reports one unity object compile and one link. This is a build interval, not an isolated compiler-process measurement.
• The actual 24-leg mode matrix reports only 4.947165 seconds, with zero failures.
• The differential step's subsequent producer build says ninja: no work to do; modes and differential already share this producer correctly.

The cold producer build therefore dominates the mode step, rather than the 24 mode legs. No fraction of those 229 seconds has yet been proven removable by changing debug information or optimization policy. #413's older 137.708-second CI-profile compile is on a different source/measurement configuration and is NOT a valid before/after comparison with this run.

Focused work

Make the native producer's intended build profile explicit and qualify a cheaper profile where it preserves the required diagnostic and correctness contract.

1. Capture actual CMake caches and exact compile/link commands for the existing native producer and canonical CI Release producer on the same source/toolchain. Inventory every difference; do not blindly append --ci, because it is broader than a debug-information switch.
2. First isolate profiling overhead from optimization changes: compare the current optimized unity build against explicit debug-info/frame-pointer choices while keeping optimization and semantic flags identical. Retain enough diagnostics/backtrace capability for CI failures; line-tables-only is an option to evaluate, not an assumed equivalent replacement for full debugger information.
3. Separately evaluate alignment with the established optimized CI Release profile only if justified by its coverage/diagnostic contract and resulting compiler throughput. No blanket -O0/-O1, global deoptimization, removal of tests, or splitting the Release compiler into multiple translation units/objects.
4. Make normal generation and missing-graph recovery use the same qualified policy. Keep one shared producer for modes and differential; do not introduce extra producer builds or stale cross-run compiler caches.
5. Keep native build policy in build.c/CMake and the existing workflow. Preserve mode/differential execution after an independent preceding failure where currently required.

Acceptance

• At least three equivalent successful, uninstrumented cold builds per candidate/profile on the matching hosted runners; report compile, link, bootstrap/configure, mode execution and full native-job wall time separately, plus peak RSS, executable size and exact flags.
• Preserve the single optimized unity object, tests-enabled producer, semantic/warning flags, all current mode/target/configuration cases, independent reference execution, sanitizers, negative controls, evidence integrity and fail-closed completion.
• Record differences between native execution, link-only checks and diagnostic controls; unchanged aggregate counts alone are not sufficient coverage proof.
• Run complete mode/differential/self-host and applicable platform correctness gates on the exact submitted source tree. Measure the resulting compiler's workload throughput so cheaper host compilation does not hide a materially slower compiler.
• Exercise missing-cache recovery and failure/cancellation paths. No profiling policy drift between the two generation sites.
• Keep the local profiling defaults intact. If full debug information is a required property of this lane and no acceptable lower-cost profile wins, retain that evidence and leave the policy unchanged.

Relationship to existing work

#413 owns source-level reduction of Apple Clang's optimized unity compile cost. This issue owns the separate, verified native-workflow producer-profile mismatch and its qualification; coordinate the two, and do not attribute flag changes to a source optimization. #408 owns the much larger differential execution tail. #333 owns combination partitioning. None of these measurements justifies deleting coverage or advertising a whole-workflow speedup.

Targeted CI/profiling issue and open-PR searches, current #413 (no comments), and current workflow/build documentation found no dedicated ticket for this producer-profile mismatch; this was not an exhaustive remote-branch/session audit. No code, workflow, repository settings or host configuration was changed, and no new validation run was dispatched during this research.

Reference for debug-info tradeoffs: Clang's documented debug-information modes.

[davidgmbb]

#604 qualification milestone — screening complete, production candidate selected (not yet qualified)

The corrected four-platform screening is complete at run 34827295920, attempt 1, source f3b460be724a22eb51c2ff4590241e16738d156c, tree 0e918c44cd0808c7592937fa3e072f34f12b1415, workflow blob 447b06373977a09498860f04f8af4f761f02a1b9.

Three jobs completed successfully:

• Linux x86-64 artifact 10342335719, ZIP SHA-256 7a929387aa8ea03506618dae4fd6dd910d472fcac9756737b11b2898a66ab95d.
• Linux AArch64 artifact 10341054816, ZIP SHA-256 595707905cbe551196b4d5499171044923f4e6854aec8293186f0d0ddb37903b.
• macOS x86-64 artifact 10342296705, ZIP SHA-256 715463bc5e2b059688157d7ea37aca30a64b49250e05d445dd587a7afd23339b.

macOS AArch64 completed all 13 producer-profile rows and all three A-vs-C throughput campaigns, but the job remains failed because one independent A-vs-B timed throughput child exited 125 before exec. The fail-closed finalizer correctly rejected 14 actual versus 13 required rows. Partial artifact 10341617416, ZIP SHA-256 6ffcfefb089de29acfeaf4a5099c38e55c4019cf91c55c08343aaab4d94d5972. I retained this as failed and routed the missing child-side errno evidence to #46; it was not rerun merely to obtain green.

Profile C (current native profile plus only -DBUSTER_DEBUG_INFO=OFF) is the supported production candidate. Median complete producer-build reductions versus A were:

Runner	A	C	Change	Peak-RSS change
Linux x86-64	154.984s	121.263s	-21.758%	-64.729%
Linux AArch64	117.551s	94.796s	-19.358%	-67.679%
macOS x86-64	162.825s	139.663s	-14.226%	-8.200%
macOS AArch64	92.265s	72.840s	-21.054%	-7.586%

After path normalization, A/C compile commands differ only by A's -g. C retains BUSTER_CI=OFF, -O3 -DNDEBUG, -fno-omit-frame-pointer, tests/unity enabled, one unity compile/object, the default linker and all semantic flags. Normal and deleted-cache recovery both succeeded with byte-identical normalized caches; each C binary was stable across its three isolated directories; every mode matrix passed.

Diagnostics are now stated against what the native lane actually retains: nonzero/nonempty compiler failures and raw logs. C keeps symbol/unwind data and frame pointers. Native artifacts retain neither the producer executable nor a core, and the workflow does not symbolize a producer PC; local Release defaults remain unchanged for source-line profiling.

All 12 A-vs-C throughput screens completed with 24 workload/mode cases, two rounds, four pairs, identical-output enforcement, and zero reported regressions/inconclusives. They deliberately used --no-guard, so these are screening observations, not acceptance.

Accordingly, C is not yet a qualified winner: each retained result says profile_winner=null and full_native_job_timing_satisfied=false. The minimum remaining gates are guarded compiler throughput plus at least three successful exact-head full native attempts on all four Unix lanes, including modes, full differential, self-host/fixed-point, complete native-job/RSS/evidence/upload timings and aggregate runner work.

Rejected attempts remain separate: 34822192017 (Bash 3.2 empty array and invalid diagnostic parity) and 34823444517 (Linux-only --cpu auto on Apple). No #36/#437 source or persistent host was touched.

[davidgmbb]

Production qualification is complete in #806 on exact submitted head ddce1b86e6201e632e10a826f6a524462e2f6f3a.

The three gates left by the screening milestone are now satisfied:

• guarded generated-compiler throughput: run 35392054038, 40/40 valid pairs, identical-output enforcement, 0 confirmed regressions, gate exit 0;
• three exact-head full-native attempts on all four Unix lanes: 35392053996, 35392532813, 35392539715, plus three passing self-host/fixed-point attempts 35392053972, 35392534747, 35392541872;
• exact-head cold-build/RSS/recovery telemetry and native critical-path/aggregate-runner-work comparison: runs 35393099586 and 35394188368.

All 12 native jobs passed mode matrix, complete differential, result/reproduction, packing and retained upload. All 12 isolated cold builds retained the qualified debug-off/frame-on/O3/tests/unity policy, produced byte-stable binaries per platform, and preserved the same policy after deleting CMakeCache.txt.

The observed native critical-path median was 16.08% below the exact-parent baseline; aggregate native runner work was 1.99% above it. Hosted-runner variance is large, so this is retained transparently and is not advertised as a causal whole-workflow speedup. The attributable same-cohort screening result remains a 14.2%–21.8% producer-build reduction with lower peak RSS on all four platforms.

#806 contains the narrow production change and closes this issue when merged. It has not been merged automatically.