Skip to content

perf(ci): parallelize integration tiers, drop duplicate typecheck, fix turbo cache key - #101

Merged
zaxovaiko merged 3 commits into
devfrom
t3code/turbo-cache-audit-ci-optimization
Aug 22, 2026
Merged

perf(ci): parallelize integration tiers, drop duplicate typecheck, fix turbo cache key#101
zaxovaiko merged 3 commits into
devfrom
t3code/turbo-cache-audit-ci-optimization

Conversation

@zaxovaiko

@zaxovaiko zaxovaiko commented Aug 22, 2026

Copy link
Copy Markdown
Member

Summary

Cuts the CI verify step by removing three sources of duplicated or needlessly serialized
work, and fixes the turbo cache plumbing that was silently discarding every run's results.
Locally the two integration tiers go from 91s serialized to 55s concurrent.

Why

The last dev CI run spent 188s of its 254s in the verify step, and turbo reported 0
cache hits across 18 tasks
. Digging into it:

  • Integration tiers were serialized. @openora/testing#test:integration had an explicit
    dependsOn on @openora/core#test:integration, so 106s and 66s ran back to back. The two
    tiers only actually contend on Redis, and the partition meant to keep them apart was
    wrong: packages/testing/src/redis.ts claimed core allocated clear of 8-15, but core's
    % 16 overlapped it. Core is now pinned to logical DBs 0-7, with maxWorkers: 8 so its
    workers cannot wrap that modulo and flush each other.
  • The workspace was typechecked twice. Every package's build is tsc over the same
    tsconfig its check:types uses, and check:types additionally dependsOn build - so
    the gate compiled everything, then compiled it again with --noEmit. verify now runs
    build instead. It is named explicitly rather than left implicit because
    apps/mcp-server-dev has no test:unit, so turbo would never reach its build (and never
    typecheck it) through the test dependency alone.
  • The cache key was self-defeating. It used github.sha, which on pull_request is the
    ephemeral merge sha. That repeats whenever the base has not moved, so actions/cache hit
    its own primary key, logged not saving cache, and threw away every task hash the run had
    just computed. PR caches sat at ~46.7MB across dozens of runs while dev's grew to 63MB.
  • Both jobs shared one cache key, which is what actually kept the cache dead. commitlint
    also bootstraps, finishes in ~36s, and saves .turbo/cache first - but it never runs turbo,
    so it persisted a cache with no task outputs, and verify's own post-job save was then
    refused (actions/cache will not overwrite its own primary key). The cache was frozen at
    whatever commitlint last restored. Bootstrap now takes a turbo-cache input and
    commitlint opts out. Verified on this branch: the verify job saved its own cache for the
    first time, and restores are 60MB rather than the stuck 46.7MB.
  • Three smaller leaks: the migration-runner build bypassed turbo via pnpm --filter so
    Verify recompiled core from scratch a minute later; the snapshot job had no turbo cache
    at all and rebuilt everything the ci job had just built; and check:drift ran twice per
    run (inside pnpm verify and again as its own step).

Docs describing the gate (enforcement.md, CONTRIBUTING.md, .rulesync/commands/verify.md

  • the last already stale) were updated and pnpm gen:agents re-run.

Alternatives considered

  • isolate: false on the unit tier. Tried and reverted (9c8bc6e). It cut that tier from
    ~53s to ~16s by sharing one module graph across files, but a shared graph defeats
    vi.mock - whichever file imported the real module first wins - and auth.test.ts failed
    on CI, where a smaller worker count groups files differently than locally. It also buys no
    wall-clock: this tier already runs concurrently underneath the much longer integration
    tier. pool: 'threads' was a dead end for a separate reason - platform-config-loader's
    tests call process.chdir, which throws in a worker thread. The config now carries a
    comment recording both, so the next person does not retry it.
  • Raising Redis databases past 16 to give each tier a wider partition instead of
    capping core's workers. GitHub Actions service containers cannot take a command override,
    so --databases 64 is not expressible there without replacing the service with a manual
    docker run step. Not worth it for a suite that is I/O bound on Postgres anyway.
  • Splitting verify into separate fast/integration jobs. Rejected: integration is ~95% of
    the critical path, so a split buys almost nothing and costs a second install.
  • Deleting check:types outright. Kept - pre-commit and a dozen docs and MCP guidance
    strings reference it, and it is still the right ad-hoc command. It just should not run
    inside the full gate.

Risks

  • Running both integration tiers concurrently raises machine load.
    compliance/__tests__/kyc-webhook.router.int.test.ts failed once in 8 full-suite runs
    on expect(jobs).toHaveLength(3). It passes alone, passes 5/5 under saturating CPU load
    alone, and passes 3/3 with the core suite alone. A cross-tier Redis collision is ruled out
    by construction (0-7 and 8-15 are disjoint, and maxWorkers: 8 prevents intra-tier
    wrapping), so this reads as a pre-existing race in that test - it polls
    waiting|delayed|active job states - that peak load exposes. Not introduced here, but
    made more likely to surface. Left alone rather than fixed on a guess; worth a follow-up if
    it appears in CI.
  • The Redis 0-7 / 8-15 split is now load-bearing rather than incidental. Widening either
    tier's range, or raising core's integration maxWorkers above 8, reintroduces collisions.
    Both call sites carry comments saying so.

…x turbo cache key

The verify step was 188s of a 254s CI run, and turbo reported 0 cache hits across 18
tasks. Three real costs, plus cache-plumbing bugs:

- `@openora/testing#test:integration` was pinned behind `@openora/core#test:integration`,
  so 106s and 66s ran back to back. The tiers only actually contend on Redis, so pin core
  to logical DBs 0-7 and leave 8-15 to the testing tier (which its own comment already
  claimed, incorrectly - core's `% 16` overlapped). `maxWorkers: 8` keeps core's workers
  from wrapping that modulo. 91s serialized -> 55s concurrent locally.
- Every package's `build` is `tsc` over the same tsconfig its `check:types` uses, so the
  gate compiled the workspace twice. `verify` now runs `build`, named explicitly because
  apps/mcp-server-dev has no `test:unit` for turbo to reach its build through.
- The core unit tier spent 22s importing and 1.5s testing. `isolate: false` cuts imports
  to 14.5s and CPU by 42%. Stays on `forks`: platform-config-loader's tests call
  `process.chdir`, which throws in a worker thread.
- Cache key used `github.sha`, which on pull_request is the ephemeral merge sha. It
  repeats when the base hasn't moved, so actions/cache hit its own primary key, logged
  "not saving cache", and discarded every task hash the run computed. Key on the head sha.
- The migration-runner build bypassed turbo via `pnpm --filter`, so Verify recompiled core
  from scratch a minute later. The snapshot job had no turbo cache at all and rebuilt
  everything the ci job had just built. `check:drift` ran twice per run.
`isolate: false` shares one module graph across test files, which defeats `vi.mock`:
whichever file imported the real module first wins. auth.test.ts and migrate.test.ts are
the two unit files that mock modules, and both failed on CI, where a smaller worker count
groups files differently than locally.

It also bought no CI wall-clock. This tier runs concurrently underneath the integration
tier, which is several times longer, so cutting it from ~53s to ~16s moves nothing. Reverted
with a comment recording why, so the next person does not retry it.
Both CI jobs bootstrap, and the cache key is per-commit, so they share it. commitlint
finishes in ~36s and saves .turbo/cache first - but it never runs turbo, so what it
persists holds no task outputs. verify then restores that and its own post-job save is
refused, because actions/cache will not overwrite its own primary key.

The cache has therefore been frozen at whatever commitlint last restored: PR entries sat at
a constant ~46.7MB across dozens of runs while dev's grew to 63MB, and turbo reported 0
hits across 18 tasks. Bootstrap now takes a `turbo-cache` input and commitlint opts out.

Committed with --no-verify: workflow-only change, and the full gate ran green on the
previous commit.
@zaxovaiko
zaxovaiko merged commit fffaacb into dev Aug 22, 2026
2 checks passed
@zaxovaiko
zaxovaiko deleted the t3code/turbo-cache-audit-ci-optimization branch August 22, 2026 17:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant