Fix: align integration install hooks on the project-wide hook timeout - #405
Open
AmaadMartin wants to merge 3 commits into
Open
Fix: align integration install hooks on the project-wide hook timeout#405AmaadMartin wants to merge 3 commits into
AmaadMartin wants to merge 3 commits into
Conversation
added 2 commits
July 31, 2026 10:38
The integration vitest project sets hookTimeout: 120000, chosen to cover a cold, network-bound fixture install measured at ~70s. Three sibling suites passed an explicit hook-timeout argument that shadowed that floor *downward* to 40s/60s, so they remained exposed to the failure mode the project budget exists to prevent. A fourth restated the same 120000 in a local constant. Drop every explicit timeout argument from the install and teardown hooks so all of them inherit the one budget, and move the measurements and the trade-off into the doc comment on that budget.
The neighbouring hook-budget comment now states that suites must not pass a timeout argument to a hook. Saying two lines later that hook timeouts still override the project value reads as an endorsement of the thing that caused the bug. Narrow the sentence to the it() case, which does still override and is meant to.
The comment asserted this was the only definition of the budget in the integration project. It is not: webui_test.ts:45 and three a2a suites pass their own beforeAll budgets (20s / 60s), all below this floor. Drop the false claim and keep only the rule this change does enforce, and tighten the neighbouring test-budget comment to one sentence.
This was referenced Aug 1, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
Related: Fix: set project-wide integration hookTimeout/testTimeout in vitest.config.ts google/adk-js#548
Related: Fix: give build_setup integration hooks an explicit timeout to stop CI flakiness google/adk-js#549
Problem: google#548 set a project-wide
hookTimeout: 120000on theintegrationvitest project. That number was chosen for a specific measured reason, recorded intests/integration/build_setup/build_setup_test.ts: a cold, network-bound fixturenpm installhas been measured at ~70s, against ~16s warm.Three sibling suites still passed an explicit hook-timeout argument, and Vitest resolves an explicit argument above the project setting. So those hooks shadowed the repo-wide floor downward, to below the very worst case the 120s figure exists to cover:
agent_loader/agent_dirname_test.tsbeforeAllnpm installagent_loader/agent_dirname_test.tsafterAllnode_modulesteardownapp_loader/app_loader_test.tsbeforeAll×2npm installapp_loader/app_loader_test.tsafterAll×2disposeAll)skills/script_js/agent_test.tsbeforeAllnpm install40 000 < ~70 000, so these are latent flakes today on exactly the failure mode #549 fixed, just with a longer fuse. This is not theoretical: see the measurement in the Testing Plan below, where these hooks actually blew the 40 000 ms ceiling on a loaded machine during verification.Two secondary defects of the same root cause:
TEST_EXECUTION_TIMEOUT— the per-it()budget — as the hook budget. A test asserting on agent output and a cold registry install are unrelated quantities.build_setup_test.tswarned against exactly this in prose, but the warning was never applied to its siblings.hookTimeout: 120000set project-wide by Chore: enforce the node: protocol for Node built-in imports in src (no new deps) #548,build_setup_test.ts's localHOOK_TIMEOUT = 120000restated the project default. Two co-equal definitions of one policy is how the budget drifts apart again.Solution: drop every explicit timeout argument from the install and teardown hooks under
tests/integration/, so all of them inherit the single budget invitest.config.ts. Because theintegrationproject setshookTimeout, omitting the argument is not "untimed" — it is "inherit 120s". Omission is therefore the correct expression of "use the repo-standard budget", and it makes the invariant structural rather than a comment somebody has to remember.The rationale text and both measurements move from
build_setup_test.tsinto the doc comment onINTEGRATION_HOOK_TIMEOUT_MS, so no information is lost and the policy lives with the value.A second commit narrows the neighbouring
INTEGRATION_TEST_TIMEOUT_MScomment, which said "Per-fileit()/hook timeouts still override both". Left as-is it would sit two lines below the new invariant and read as an endorsement of exactly the override that caused this bug. It now speaks only to theit()case, which does still override and is meant to.A third commit removes an overclaim caught in review. The doc comment originally said "This is the only definition of that budget", which is false:
adk_web/webui_test.ts:45and the threea2asuites each pass their ownbeforeAllbudget (20s / 60s), all below this floor. The comment now states only the rule this change actually enforces — that an install or teardown hook must not pass its own timeout argument — and both comments came out shorter than before.Those remaining hooks were not swept, for three reasons. They start a server rather than install a fixture, so the ~70s cold-install measurement that justifies 120s does not apply to them. The
a2aones pass the same constant to both the hook budget and the server's ownstartFailureTimeout, so the number is deliberately coupled there rather than accidentally shadowing policy. Andwebui_test.tsis the subject of open PR #359, which is specifically about naming its hook timeouts — sweeping it here would collide with a live review.One correction worth recording, since
webui_test.ts:85was also raised as a shadowing site: its trailing20000is not a hook budget. It is the third argument todescribe(), which Vitest 3.2.6 types asTestOptions.timeout— a per-test timeout. Verified empirically rather than from the type docs: a probe suite shapeddescribe('...', () => { beforeAll(slow); it(...) }, 20000), run with the projecthookTimeouttemporarily set to50, fails withHook timed out in 50ms— not20000ms. A describe-level number cannot shadow the hook floor, so there is nothing to fix at that line.Why
build_setup_test.tsis touched again, having only just been fixed by #549 — this is a consolidation, not a revert. #548 made its localHOOK_TIMEOUTconstant redundant by setting the same value project-wide; this merges the two co-equal definitions into one so they cannot drift. That file's effective hook budget is unchanged at 120s. The behavioural fix is to the other three suites.Effect: the hook budget rises 40 000 → 120 000 ms for six hooks and 60 000 → 120 000 ms for one; it is unchanged for the two
build_setuphooks. Noit()budget changes anywhere. There is no CI wall-clock cost on a healthy run — a timeout ceiling is only consumed on failure. The cost is strictly that a genuinely hung hook takes up to 120s to surface instead of 40s, which is the trade-off #548 already accepted repo-wide and this applies consistently.Deliberately out of scope (called out so the omissions are not read as oversights):
tests/integration/that start a server rather than install a fixture (a2a/*,adk_web/webui_test.ts) keep their explicit arguments. The a2a ones pair the hook budget with the server's ownstartFailureTimeout, so being explicit there is meaningful rather than accidental shadowing.afterAllteardowns swallow errors via.catch(() => {}), so a cleanup failure cannot fail the suite. That is pre-existing; changing failure semantics under the banner of a timing fix would be the wrong scope.it()budgets are untouched. One test body is still flaky under parallel load (see Testing Plan) — that is a separate, per-test concern.Collision check (required before implementation).
gh pr list --repo AmaadMartin/adk-js --state open --limit 300, thengh pr diff --name-onlyon every plausibly adjacent PR. Six open PRs touch these same hooks: #218, #235, #256, #257, #260, #276. None of them lands this change, and I did not stack on any of them, because:mainthat had no project-widehookTimeout— Fix: stabilise install-bound and server-spawning integration suites #218 and Fix: stop the agent_loader and build_setup integration suites from timing out on their fixture npm install #257 still show the pre-Feat: declare the Node.js floor on the workspace root and derive the checked manifest list (stacked on #544) #549build_setup_test.tsas their base.INSTALL_TIMEOUT = 180_000,FIXTURE_HOOK_TIMEOUT_MS = 180_000,FIXTURE_SETUP_TIMEOUT = 180000). That keeps the shadowing and adds a second source of truth, which is the defect this PR removes; stacking on one would mean immediately deleting its constants.This PR is the minimal consolidation on current
mainand supersedes the timeout portion of those PRs. #276 and #299 take a different and complementary direction (removing the fixturenpm installaltogether); this change does not conflict with that — a hook that no longer installs simply never approaches the budget.Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.
No unit tests are added: this change alters no product code, adds no executable product lines, and the deliverable is test-infrastructure hardening. Fabricating a unit test for a vitest config constant would assert nothing. Coverage is computed over
core/src,dev/src,integrations/srconly (vitest.config.ts), none of which is touched, so coverage does not move. The existing integration suites are the test, and they are verified below.Inheritance proof (the mutation test). The mutation is on the config value the change makes authoritative: set
INTEGRATION_HOOK_TIMEOUT_MS = 1and check which hooks notice.Against the unfixed files (
git stashthe four test files, probe value in place) — this is the run that proves the tests can fail and that the bug is real:Two things this shows. First, the bug: with the project floor set to 1ms, the three hooks carrying an explicit argument ignored it entirely and reported 40 000 ms / 60 000 ms, while the one hook already written without an argument correctly inherited 1ms. Lowering the repo-wide floor has no effect on the opted-out files. Second, and unplanned: those hooks genuinely hit 40 000 ms and 60 000 ms in this run, with three suites installing in parallel. The 40s ceiling is not merely below the measured cold-install figure — it is reachable in practice.
Against the fixed files, same probe value — every install and teardown hook in all four suites now reports
Hook timed out in 1ms, atagent_dirname_test.ts:27and:49,app_loader_test.ts:30,:55and:77, andagent_test.ts:32and:95. One caveat reported honestly: the fourthapp_loaderhook (afterAllat :130) fails withTypeError: Cannot read properties of undefined (reading 'disposeAll')instead of a timeout, because itsbeforeAllwas killed by the probe and never assignedloader, so the hook throws synchronously before any budget can elapse. That is an artifact of the probe cascade, not a missed argument — the hook carries no third argument, and its three siblings in the same file demonstrably inherit.The probe value was then reverted;
git diff main -- vitest.config.tscontains only the doc-comment change and no changed number.Suite runs at the restored 120 000 budget. All four suites, run together:
The single failure is
agent_dirname_test.ts:41, an assertion inside anit()body (expected '\n> dirname-test@1.0.0 start…' to contain "I'm stubby model response!") — the spawned agent had not answered within the per-test 40s budget while four suites competed for the machine. It is not a hook failure and this change does not touchit()budgets. Run on its own the suite is green:That run is itself corroborating evidence for the fix: 228s wall clock of which only ~15s was test bodies (5.3s, 5.1s, 5.0s), leaving ~213s in six install/teardown hooks — ~35s per hook, against the 40 000 ms ceiling this PR removes.
Other gates on the pushed commit:
tsc --noEmitis not green onmain(see #370, which exists to fix that). The count is 280 both with and without this change, verified by re-running against the stashed tree, and zero of them are in the five files touched here.Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.
To reproduce the inheritance proof yourself, from a clean tree:
npm install && npm run build.INTEGRATION_HOOK_TIMEOUT_MS = 1invitest.config.ts.npx vitest run --project integration tests/integration/app_loader/app_loader_test.ts— every install/teardown hook must fail withHook timed out in 1ms. Onmain(before this change) the same hooks instead reportHook timed out in 40000ms, proving the explicit argument shadows the project budget.INTEGRATION_HOOK_TIMEOUT_MS = 120000and confirm the suite passes again. Do not ship the probe value — it breaks the integration matrix on all three operating systems.Checklist
[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.
CI on this branch (commit
bfd20d50, after the review revision) — real test jobs, all green:run-tests (ubuntu-latest)run-tests (macos-latest)run-tests (windows-latest)The pre-revision commit
41ae0c12was also green on all three (4m39s / 6m1s / 9m0s).