Fix: attribute cold AgentLoader discovery cost to beforeAll to stop macOS CI flake - #260
Closed
AmaadMartin wants to merge 2 commits into
Closed
Fix: attribute cold AgentLoader discovery cost to beforeAll to stop macOS CI flake#260AmaadMartin wants to merge 2 commits into
AmaadMartin wants to merge 2 commits into
Conversation
added 2 commits
July 29, 2026 11:11
…eAll The discovery test in app_loader_test.ts intermittently timed out at 40s on the macos-latest CI leg. AgentLoader discovery is lazy: the constructor does no work, so the first listApps() call in the test body paid for four concurrent esbuild bundle-and-minify passes over the whole ADK dependency graph. The cost belonged to setup, not to an assertion. Warm the loader with preloadAgents() in the discovery beforeAll, and give the four install/IO-bound fixture hooks their own FIXTURE_SETUP_TIMEOUT budget. Every it() keeps the existing 40s TEST_EXECUTION_TIMEOUT, so a discovery assertion that ever needs more than that still fails loudly.
Address simplicity review: the rationale for FIXTURE_SETUP_TIMEOUT was 8 lines of prose for a one-line constant, and it described the loader warm-up that only one of the five hooks performs. Keep that detail at the call site where it applies and state the budget's rationale in one sentence.
This was referenced Jul 29, 2026
Fix: gitignore the .env that adk create writes so scaffolded agents cannot commit their API key
#364
Open
Closed
Open
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
Closes: #issue_number
Related: #issue_number
Problem:
tests/integration/app_loader/app_loader_test.ts→'should discover apps vs agents across directories and standalone files'intermittently fails on therun-tests (macos-latest)leg of.github/workflows/validation.yamlwithError: Test timed out in 40000ms., and passes on an immediate re-run of the same commit.This is a cost mis-attribution, not a slow assertion.
AgentLoaderdiscovery is lazy: the constructor (dev/src/utils/agent_loader.ts:365) only assigns fields and registers exit handlers, so thebeforeAllthat builds the loader does almost nothing. The first thing the test body does isawait loader.listApps(), andlistApps()begins withawait this.preloadAgents()— so the test pays for the entire cold discovery pass: fourAgentFile.load()calls that each run a fullesbuild.build()withbundle: trueandminify: true(agent_loader.ts:181,:188-189), inlining the whole@google/adkdependency graph, followed by a dynamicimport()of each emitted artifact.That is why this test flakes and the two after it never do:
preloadAgents()short-circuits onagentsAlreadyPreloadedandAgentFile.load()short-circuits on its cached result, so the following tests inherit a warm loader and run in ~1 ms. All the cost is concentrated in whichever test runs first.Solution: Move the one-time cost to where it belongs — warm the loader with
await loader.preloadAgents()in the discoverybeforeAll— and give the install/IO-bound fixture hooks their ownFIXTURE_SETUP_TIMEOUTbudget, sized for the work they actually perform.Every
it(...)deliberately keeps the existing 40 sTEST_EXECUTION_TIMEOUT. After this change the discovery tests are pure assertions against a warm loader, so if one ever needs more than 40 s again that is a real regression and should still fail loudly. The alternative — simply raising the test timeout until the failure stops — would have hidden that signal.Measured on one machine, same command, same load, before vs. after (
npx vitest run --project integration --reporter=verbose tests/integration/app_loader/app_loader_test.ts):should discover apps vs agents across directories and standalone filesshould load App from directory entrypoint and expose App and rootAgentshould synthesize App when loadApp() is called on BaseAgent fileThe baseline run reproduces the flake exactly: the test passed with only 413 ms of headroom under its 40 000 ms budget. Any slower runner — which
macos-latestis — crosses the line.Note this does not make CI faster, and is not intended to: the same work happens, just in the setup phase instead of inside an assertion. Total wall-clock time for the file was unchanged (60.7 s before vs. 65.0 s after for the discovery block in isolation). One intentional side effect: a genuinely broken fixture now surfaces as a
beforeAllfailure with its original stack trace, rather than as a confusing assertion failure in whichever test ran first.This is a test-only change. No file under
core/src,dev/src, orintegrations/srcis touched, no assertion is added, removed, or weakened, no test is skipped or retried, and noany/@ts-ignore/@ts-expect-error/eslint-disableis introduced (preloadAgents()is apublicmethod returningPromise<void>).It is also independent of the open PR that sets a project-wide
hookTimeout/testTimeoutinvitest.config.ts. A project-level timeout is only a default; an explicit third argument toit(...)/ second argument tobeforeAll(...)overrides it, and every test and hook in this file already passes one explicitly. The two changes touch disjoint files and cannot conflict.Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
No new unit test: this change adds no production code, only re-attributes existing work between a test and its setup hook, so there is no new line or branch to cover. Asserting on a test-harness timeout constant would have no value. The existing unit suite for the code under test was run unchanged and stays green:
[x] All unit tests pass locally.
Manual End-to-End (E2E) Tests:
Setup:
npm install && npm run buildat the repository root. The build is mandatory — the fixtures resolve@google/adkand@google/adk-devtoolsthroughfile:links intocore/anddev/, which serve fromdist/.Full affected file, all six tests, with per-test durations:
Baseline comparison — reverting only this commit's test file and re-running the identical command reproduces the flake's cause, with the first discovery test at 39587 ms (413 ms under its 40 s budget) while the two after it report 1 ms each. That asymmetry is the bug; it disappears after the change.
Timeout-wiring sanity check — temporarily setting
FIXTURE_SETUP_TIMEOUTto1makes the run fail withError: Hook timed out in 1ms.attributed to the discoverybeforeAll, confirming the budget is wired to the hook and not to the test. (Probe reverted; not part of the diff.)Repository gates that CI enforces, run on the exact pushed commit:
git status --porcelainis empty after the run — no fixturenode_modules/,package-lock.json, ordist/artifacts leaked into the tree.One unrelated pre-existing issue was found while verifying and deliberately left out of this diff to keep it reviewable:
tests/integration/agent_loader/agent_dirname_test.ts:29runsnpm installfor each of its three fixtures inside abeforeAllbudgeted at the same 40 s, and fails withHook timed out in 40000mson a sufficiently loaded machine. It is the identical anti-pattern in a file this PR does not touch, and is tracked separately.Checklist
[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.