Cache compilation in CI with ccache

[kylelutze]

CI rebuilds every object from scratch on each of the four builds a run performs, across two container images. This wraps the compilers in ccache so the compiles hit a content-addressed store instead.

Only ccache's store is cached

build/ and .sconsign.dblite stay out of it. Those are incremental state whose correctness depends on scons's dependency scanner having recorded every input, and it doesn't record all of them — DET_INIT is read straight from the environment (SConstruct:322), and the Configure results in config.h / options_cache.py aren't tracked either. Persisting that across runs is how a stale object survives a source change.

ccache re-hashes the source, every header it includes by content, the compiler binary and the full command line on every compile. So scons still rebuilds its dependency graph from a fresh checkout every run and decides what to compile from scratch; only the individual g++ -c invocations are reused. It's stricter than scons here — DET_INIT lands in CXXFLAGS, so ccache keys on it where scons doesn't.

CCACHE_SLOPPINESS is deliberately unset: include_file_mtime / include_file_ctime trade content hashing for timestamps, and time_macros would let ccache serve a stale __DATE__/__TIME__ banner for GlobalContainerArgs.cpp:347. CCACHE_COMPILERCHECK=content covers a g++ point release landing inside a base image.

Tripwires

• ccache -s -v is printed per run — a change to a widely included header that still shows near-total direct hits means the cache isn't seeing it.
• The existing harnesses already run against the produced binaries.

A cold build is gh cache delete on the two ccache-ubuntu:* entries, then re-run. On demand rather than on a timer: ccache keys on content, so a stale entry needs a hash collision, not a stale clock, and a weekly cold build spends 14 minutes of two runners to re-assert that.

Cache lifecycle

Only master saves; PR branches can't read each other's caches anyway, and letting every PR write would churn the quota and evict what PRs restore from. Master saves whenever the compiles ran, including runs a later test step failed — the entries are content-addressed, so a failing harness says nothing about whether the objects are the right objects, and a build that compiled the whole tree and then failed the LAN test shouldn't throw the tree away. !cancelled() rather than always() so a cancelled run doesn't upload during the grace period.

CCACHE_MAXSIZE=1G. A single cold build is ~130M per image; each master push touching a widely included header adds roughly another tree's worth before LRU catches up. Entries are keyed by run_id, so GitHub's 10G repo quota evicts the old ones and only the newest prefix match matters.

Enabling

CCACHE=1 in the environment, not a scons option, so it can't stick in options_cache.py. Shared helper in scons/ccache.py, used by both SConstructs. It resolves ccache via shutil.which — scons scrubs PATH for build commands, so the usual /usr/lib/ccache symlink trick would silently do nothing here. Missing ccache with CCACHE=1 set is a hard error rather than a silent uncached build.

Verified locally

With a logging pass-through shim (no ccache on the dev machine):

• CCACHE=1, no ccache on PATH → aborts with CCACHE is set but ccache was not found on PATH.
• test/SConstruct emits ccache g++ -o TestsRunner.o -c … and the object builds.
• Top level: all 37 configure probes route through the wrapper and 3112 compile-DB entries come out wrapped, so Configure is fine with the prefixed compiler.
• CCACHE unset → zero ccache entries in compile_commands.json, and config.h / options_cache.py byte-identical. Default builds untouched.

The first master run will be a cold build that populates the cache; the speedup starts after that.

Not covered

Windows. The msys2 job needs path translation between the MSYS-style CCACHE_DIR and the Windows path actions/cache tars, which I couldn't test locally. The SConstruct side already works for it if we want it as a follow-up.

[Giszmo]

Review of the ccache change. Six findings, most notable first. None is a correctness blocker for the common path (push-triggered fresh run); 1–3 are edge cases around re-runs and manual dispatch that would silently defeat caching rather than fail loudly.

.github/workflows/build.yml

1. L137 – The save key ccache-${{ matrix.image }}-${{ github.run_id }} omits run_attempt. A re-run of the same job reuses the same key, so actions/cache/save@v4 reports "already exists" and silently skips saving any objects refreshed in the re-run.
2. L133 – if: ${{ !cancelled() && ... }} removes the implicit success() gate for the whole job, not just the intended "test step failed" case. If an early step (e.g. apt-get install ccache) fails outright, /ccache is never populated and Save still runs against a missing path, producing a spurious secondary failure that masks the real root cause.
3. L133 – workflow_dispatch was added as a trigger, but the Save condition still checks github.event_name == 'push', so manually dispatching on master (the natural recovery after gh cache delete) never persists the rebuilt cache.
4. L124 – Report the cache statistics uses if: always() while Save deliberately uses !cancelled() to avoid racing a cancellation's shutdown grace period. Same concern, two different treatments in one job.

SConstruct L280

5. ccache.enable() only rewrites CC/CXX, but SCons's default $SMARTLINK resolves the link step through CXX too, so every link (including configure()'s CheckCC/CheckCXX probes) is routed through ccache g++. Harmless (ccache passes non-compile invocations through) but it adds a spawn per link and pollutes the ccache -s -v output the PR relies on as a cache-health signal.

scons/ccache.py L31

6. The "already wrapped" guard checks env[var].startswith(binary) against ccache's resolved absolute path, so CXX='ccache g++' set manually alongside CCACHE=1 is not recognized as already wrapped and gets double-prefixed to ccache ccache g++, a hard build failure with a confusing "no such file" error.

(written by Junior, my agent)