Fix: raise app_loader integration-test timeout to 60s to stop CI flake - #256
Closed
AmaadMartin wants to merge 1 commit into
Closed
Fix: raise app_loader integration-test timeout to 60s to stop CI flake#256AmaadMartin wants to merge 1 commit into
AmaadMartin wants to merge 1 commit into
Conversation
The discovery case in app_loader_test.ts is the first test to touch the loader, so it absorbs the whole cold-start cost of AgentLoader discovery: listApps() calls preloadAgents(), which eagerly loads all four discovered entrypoints in parallel, each running a full esbuild bundle plus a dynamic import of the result. That fits in the 40s budget on ubuntu-latest but intermittently overruns it on the slower windows-latest and macos-latest runners, failing the required validation workflow with "Test timed out in 40000ms". Raise the file's single shared TEST_EXECUTION_TIMEOUT to 60000, matching the budget the other subprocess-heavy integration tests already use (e.g. tests/integration/skills/script_js/agent_test.ts). No assertion is changed and no test is skipped or retried.
This was referenced Jul 29, 2026
Closed
Open
Owner
Author
|
Close. Raising the budget to 60s hides the cost instead of moving it, and the discovery test body runs in about 4 ms once #506 lands. This is closed on approach, independent of which PR lands. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
No existing issue.
Problem: The
validationworkflow is a required check and runs the test suite on athree-way OS matrix —
ubuntu-latest,windows-latest,macos-latest(
.github/workflows/validation.yaml:17). Theapp_loaderintegration tests failintermittently on the Windows and macOS runners with:
The failing case is
AgentLoader discovery and loading integration > should discover apps vs agents across directories and standalone files. It is a runner-speed flake, not aproduct defect:
ubuntu-latestpasses consistently on the same commit, re-runs of thesame commit fail on a different OS each time, and the case itself is deterministic — it
calls
listApps()/listAgents()and asserts on fixed name lists.The reason this particular case is the slow one is that it is the first test in its
describeblock to touch the loader, so it absorbs the entire cold-start cost ofdiscovery.
AgentLoader.listApps()callspreloadAgents(), which fans out over everydiscovered entrypoint with
Promise.all(dev/src/utils/agent_loader.ts:490). Thediscoveryfixture has four entrypoints (service_alpha/app.ts,service_beta/agent.ts,standalone_agent.ts,standalone_app.ts), and becauseDEFAULT_AGENT_FILE_OPTIONSsetsbundle: true(dev/src/utils/agent_loader.ts:82-85), each one runs a fullesbuild.build()of the entrypoint plus its@google/adkdependency graph(
dev/src/utils/agent_loader.ts:181) and then dynamically imports the result. The latercases in the block reuse the cached
AgentFileinstances and are cheap.So the flaking case performs four concurrent esbuild bundles, four large dynamic imports,
and the associated temp-dir/symlink filesystem work. That fits within 40s on the Linux
runner but has no headroom for the Windows runner (slow NTFS metadata operations,
on-access AV scanning of freshly written bundle output) or the macOS runner (fewer, slower
vCPUs).
Solution: Raise the file's single shared
TEST_EXECUTION_TIMEOUTfrom40000to60000.Why
60000specifically: it is already the established budget for the subprocess-heavyintegration tests in this repository, so this makes
app_loader_test.tsconsistent ratherthan inventing a new number.
tests/integration/skills/script_js/agent_test.ts:15TEST_EXECUTION_TIMEOUT60000tests/integration/a2a/stream/stream_test.ts:13TEST_TIMEOUT60000tests/integration/a2a/input_required/input_required_test.ts:13TEST_TIMEOUT60000tests/integration/test_api_server.ts:25DEFAULT_TIMEOUT60000tests/integration/skills/script_js/agent_test.tsis the closest structural analogue —same
execAsync('npm install')inbeforeAll, samespawn('npm', ['run', 'start'])subprocess driven through the shared
sendInputhelper, same single shared constantapplied to both hooks and tests — and it already uses
60000.Because the constant is the file's only budget, this lifts all eight call sites at once
(four hooks, four tests). That is intended: the hooks run
npm installand recursivenode_modulesteardown, which are exactly the operations that are slowest on the Windowsrunner, so giving them the same headroom stops the flake from simply relocating from the
test body into a hook.
Scope notes:
it.skip,it.retry,retry:orbail:. The hook and test bodies are byte-identical tomain.core/src/,dev/src/orintegrations/src/. In particularAgentLoaderis deliberately not "optimized"(e.g. lazy
preloadAgents(), disablingbundle, caching esbuild output); that would bea behavioural change to shipped code and needs its own design and review.
vitest.config.tsand.github/workflows/validation.yamlare untouched.package.json/package-lock.jsonare untouched.core/src/**,dev/src/**andintegrations/src/**(vitest.config.ts:109-113), and a test file is not instrumented.Trade-off: a genuinely hung
app_loadertest now takes 60s instead of 40s to surface.There is no cost on the passing path — Vitest timeouts are ceilings, not sleeps, so a test
that completes in 12s still completes in 12s and green-run CI duration is unchanged.
The diff is one file, +3 −1.
Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
No new unit test was added, and that is a deliberate determination rather than an
omission: this change adds zero lines of production code, so there is nothing to cover. A
meta-test asserting
TEST_EXECUTION_TIMEOUT === 60000would test the edit rather than anybehaviour, so it was intentionally not written. The existing integration tests in the
modified file are the test plan, and all of them still pass unmodified.
[x] All unit tests pass locally.
Targeted run from the repository root, on the exact commit pushed:
Result — all 6 cases pass (3 parameterized
App entrypoint with %scases forapp_ts/app_js/app_default, plus the 3AgentLoader discovery and loading integrationcases):
The measurement that motivates the change. Locally the flaking discovery case takes
16.6s (19.9s on a second run). The decisive number, though, comes from CI on this
branch — on
windows-latestthat same case took:~34.7s against the old 40s budget — 87% of the budget consumed, ~5.3s of headroom —
and that figure is highly reproducible across runs (34689ms / 34803ms), so it is a stable
property of the Windows runner rather than a one-off spike. Any ordinary variance (AV
scanning, disk contention, a noisy runner) pushes it over the line, which is exactly the
observed intermittent failure. The whole file takes ~69s on Windows versus ~48s locally.
At 60s the same case has ~25s of headroom (58% of budget used), which is the margin this
PR buys.
Also run on the same commit:
Per the repository JS guidelines the full suite (
npm test/npm run test:coverage) wasdeliberately not run locally; the full matrix run is the CI workflow's job.
Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.
The authoritative verification is the CI matrix itself, because the bug only manifests on
the slower hosted runners:
validationjobs are green for the head commit:run-tests (ubuntu-latest),run-tests (windows-latest),run-tests (macos-latest).on the same head commit at least twice more and confirm all three OS jobs stay green.
Observed on this branch:
ubuntu-latest5m24s,macos-latest6m56s,windows-latest7m52s). The ~45% wall-clock spread between the Linux and Windowsrunners is the runner-speed gap this fix accounts for.
ubuntu-latestandmacos-latestgreen.app_loader_test.tspassed on all three OSes, including Windows (the 34.7smeasurement above). The Windows job as a whole went red, but on an unrelated,
pre-existing flake in a different file:
tests/integration/tools/run_skill_script_tool_test.ts > successfully executes a real Python skill script, which timed out at 5000ms — Vitest's default budget, i.e.that test declares no explicit timeout at all. It is outside the scope of this PR and is
deliberately not touched here; widening this PR to cover it would only blur the change.
run_skill_script_tool_test.tsflake did not recur, confirming it is intermittent andindependent of this change.
app_loader_test.tsagain passed on Windows, with thediscovery case at 34803ms.
Net result across the three runs:
app_loader_test.tspassed on all three OSes in everyrun, and the only red job was caused by a different file.
To reproduce the original failure locally as a deterministic proxy, temporarily lower
TEST_EXECUTION_TIMEOUT(e.g. to5000) and re-run the command above: the identicalTest timed out in Nmsmessage appears, showing the test is slow, not stuck.Checklist
[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.