Fix: stabilize app_loader integration test timeouts and stop matrix fail-fast - #235
Closed
AmaadMartin wants to merge 2 commits into
Closed
Fix: stabilize app_loader integration test timeouts and stop matrix fail-fast#235AmaadMartin wants to merge 2 commits into
AmaadMartin wants to merge 2 commits into
Conversation
added 2 commits
July 29, 2026 02:49
Every hook and test body in this file shared one 40s constant, but the work they do is nowhere near comparable. A fixture beforeAll runs a real npm install of the whole @google/adk graph, and the discovery test body esbuild-bundles and minifies that graph once per entrypoint. On a contended runner the install alone exceeds 40s, so the file flaked on macOS with no assertion ever failing. Split the constant into a 180s hook budget (install, plus node_modules removal, which is slow on Windows) and a 120s test-body budget, so a genuine hang is still caught while normal work fits. Also make fixture installs cache-friendly: the fixtures depend only on file: links to core/dev, whose transitive deps the repo-level install has already put in ~/.npm, so --prefer-offline resolves from cache instead of revalidating against the registry, and --no-audit --no-fund drop two more network round trips. Measured on one workstation, per fixture install: 78.6s mean before, 57.9s after, and far more consistent (1.7s spread vs 13.7s). No assertion changed.
The run-tests matrix relied on GitHub's default fail-fast: true, so a single-OS flake cancelled the in-flight legs and left maintainers unable to tell "macOS-only problem" from "broken everywhere". Set fail-fast: false so every leg reports its own result. Also cap the job at 60 minutes. The job previously inherited GitHub's 6 hour default, which was only tolerable because the in-test budgets were small; now that they are larger, an explicit cap keeps a genuine hang from occupying a runner for hours.
This was referenced Jul 29, 2026
Closed
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
N/A — no existing tracking issue.
Problem: Two independent defects make the
validationworkflow untrustworthy.Defect 1 — unrealistic timeout budget.
tests/integration/app_loader/app_loader_test.tsused a single
TEST_EXECUTION_TIMEOUT = 40000for every hook and every test body,but those two phases do wildly different amounts of work:
beforeAllruns a realnpm installfor a fixture project. The fixtures depend onfile:links tocoreanddev, so npm still resolves and installs the entiretransitive dependency graph of
@google/adkand@google/adk-devtools.loader.listApps(), which bundles and minifies thatwhole graph with esbuild once per discovered entrypoint (the discovery fixture has four).
Vitest runs test files in parallel, so several of these installs contend for CPU, disk and
the npm cache at once. On
macos-latestthis flaked asError: Test timed out in 40000msatapp_loader_test.ts:82— a pure timing flake, with noassertion failing, on a file that completed in 71043 ms on
windows-latest.Defect 2 — fail-fast destroys the signal. The
run-testsmatrix declared nofail-fastkey, so it inherited GitHub's default
fail-fast: trueand the macOS flake cancelled thein-flight
windows-latestleg. A maintainer could not tell "macOS-only problem" from"broken everywhere".
Solution:
Timeouts. Replace the one shared constant with two that match the two phases:
INSTALL_TIMEOUT = 180_000for the install/cleanup hooks (removing anode_modulestree isitself slow on Windows) and
TEST_EXECUTION_TIMEOUT = 120_000for the fourit(...)bodies.These are ~3-4x headroom over the slowest observed individual phase, so normal work fits
while a genuine hang is still caught. No assertion was changed, no test was skipped,
reordered, or gated on
process.platform— the diff is timing budgets and install flagsonly.
Install flags. Both fixture installs now use a single
NPM_INSTALLconstant with--prefer-offline --no-audit --no-fund. The job's ownInstall dependenciesstep runs arepo-level
npm installbefore the test step, and the fixtures depend only onfile:linksto
core/dev, so~/.npmis already warm by the time fixture installs run;--prefer-offlinetherefore resolves from that cache instead of revalidating against theregistry, and
--no-audit --no-funddrop two more network round trips. A cold cache stillfalls back to the network, so this stays correct — only slower.
Workflow. Add
fail-fast: falseso every matrix leg reports its own result, andtimeout-minutes: 60on the job. The job previously inherited GitHub's 6 hour default,which was only tolerable because the in-test budgets were small; now that they are larger, an
explicit cap keeps a genuine hang from occupying a runner for hours.
Deliberately not included. An earlier revision also added
cache: 'npm'to theUse Node.jsstep. I dropped it: the premise does not hold. The repo-levelInstall dependenciesstep already warms~/.npmwithin the same job before any fixtureinstall runs, so the action cache changes nothing for
--prefer-offline. It would only speedthe repo-level install across runs, which is a separate concern from this flake fix and
belongs in its own PR.
The esbuild bundle/minify cost in
dev/src/utils/agent_loader.tsis the real underlyingexpense, but reducing it needs its own design and benchmarks; this PR does not touch
production code. Sibling integration tests (
agent_dirname_test.ts,build_setup_test.ts,skills/script_js/agent_test.ts) have the same latentnpm install-in-beforeAllshape and are intentionally left alone to keep this reviewable.Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[ ] I have added or updated unit tests for my change.
[ ] All unit tests pass locally.
No unit test is added, and this is not a workaround. This change adds zero lines of
production code.
vitest.config.tsrestrictscoverage.includetocore/src/**,dev/src/**andintegrations/src/**, and no file under those roots is touched, so there isno new production symbol to test and coverage percentages are unchanged. No coverage
threshold was lowered. The file modified here is the test; it is an integration test, so it
is verified below rather than by a new unit test.
The modified file's own results (
npx vitest run --project integration tests/integration/app_loader/app_loader_test.ts --reporter=verbose).The file contains four
it(...)call sites, one of which sits inside adescribe.eachover three fixtures, so six tests execute.Paired before/after on the same machine, back to back, with fixtures removed first:
mainError: Hook timed out in 40000msatapp_loader_test.ts:77, 6 tests skipped (166.58s)The local failure reproduces the CI defect with the hook losing the race rather than the
test body; both shared the same 40s constant, which is exactly why both budgets needed
raising and not just the one that happened to fire first in CI.
Per-test durations with this PR (cold fixtures), all well inside the new 120s budget:
Across three separate runs the discovery
listApps()test measured 16468ms, 18465ms and26008ms on the same idle workstation — a 58% swing on identical work. At 26s it is already
at 65% of the old 40s budget, which is precisely why a slower, contended hosted runner
crossed it.
Install-flag A/B, discovery fixture,
node_modulesremoved before each iteration:npm installnpm install --prefer-offline --no-audit --no-fundA 26% mean reduction, and much more consistent (1.7s spread vs 13.7s). Note a single
fixture install exceeds the old 40s hook budget outright.
Cold vs warm npm cache (whole file): 348.15s cold (cache cleared with
npm cache clean --force)vs 302.72s warm — both pass. The cold run confirms
--prefer-offlinestill succeeds when thecache lacks entries.
No-regression sweep (
npx vitest run --project integration, whole integration project):mainStrictly better: same 68 passing, one fewer failing file, and no hard test failures.
app_loader_test.tspassed all 6 tests (278588ms) even under full 35-file parallelcontention. The three still-failing files are pre-existing and out of scope, each failing
in
beforeAllwith the samenpm installroot cause and its own separate budget:build_setup_test.ts(Hook timed out in 10000ms),agent_dirname_test.ts(
40000ms),skills/script_js/agent_test.ts(60000ms). They are listed, not "fixed",here.
Gates:
npm run build,npm run lintandnpm run format:checkall pass.npm run ts:checkreports 308 errors, but they are byte-identical onmainand on thisbranch (verified by diffing both outputs) and none are in the touched file — pre-existing
and untouched by this diff. It is not one of the workflow's steps.
Workflow YAML validated by parsing it, since
format:checkcovers.tsonly:No other step was reindented; the workflow diff is two added lines.
Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.
To reproduce the original failure and confirm the fix, from the repository root:
npm install && npm run buildrm -rf tests/integration/app_loader/*/node_modules tests/integration/app_loader/*/package-lock.jsonnpx vitest run --project integration tests/integration/app_loader/app_loader_test.ts --reporter=verbose— expect 6 passing tests.
INSTALL_TIMEOUTandTEST_EXECUTION_TIMEOUTback to
40000, repeat step 2, and re-run: the run fails withtimed out in 40000mswithout any assertion failing.Confirmed on this PR's own
validationrun — all three legs ran to completion andreported independently:
run-tests (ubuntu-latest)run-tests (macos-latest)run-tests (windows-latest)The previously-flaking
macos-latestleg is green, andtests/integration/app_loader/app_loader_test.tscompleted there with all 6 tests in55425 ms. That file total on its own exceeds the old 40 s constant, which is the clearest
statement of why the budget — not the code under test — was the defect. Every leg also
finished far inside the new
timeout-minutes: 60cap, so the cap bounds a hang withoutthreatening a healthy run.
Because
fail-fast: falseonly becomes observable when a leg actually fails, a green runcannot demonstrate it directly; confirming it empirically requires deliberately breaking one
OS on a scratch branch, which is deliberately not shipped here.
Checklist
[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.