Fix: stop per-file timeout arguments from lowering the integration project's budgets - #860
Open
AmaadMartin wants to merge 4 commits into
Open
Fix: stop per-file timeout arguments from lowering the integration project's budgets#860AmaadMartin wants to merge 4 commits into
AmaadMartin wants to merge 4 commits into
Conversation
added 4 commits
August 9, 2026 04:25
…k budget A per-call Vitest timeout argument replaces the project value in both directions, so these files ran their `npm install` hooks on 40s or 60s while the `integration` project grants 120s. Nothing recorded a measurement behind those numbers. Prettier collapses the `it()` calls once the third argument goes; the bodies are unchanged. Read this diff with `git diff -w`.
A `describe(name, body, timeout)` argument applies to every test in the suite and replaces the project's 60s `testTimeout`. No measurement justifies capping this suite lower.
"Matches the largest per-file timeout in the repo" stops being true once the per-file 60000 constants are deleted. Record the bound the budget actually provides instead.
No file under tests/integration passes a hook timeout argument now, so every hook runs on the project's 120s hookTimeout. build_setup's HOOK_TIMEOUT only duplicated that value; its measurement and the server start watchdog invariant move to the constant they describe. Renames TEST_TIMEOUT to SERVER_START_TIMEOUT in the two a2a suites: the constant's only remaining reader is startFailureTimeout.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
Closes: #issue_number
Related: #issue_number
Problem: A per-call Vitest timeout argument replaces the project value in both directions. It never raises it, so a file that passes
40000runs under 40 s even though theintegrationproject grantshookTimeout: 120000.agent_dirname_test.tsran itsnpm installhook on 40 s three times over, once perdescribe.eachfixture. The 40 s and 60 s numbers were not tied to any measurement.Solution: I removed every timeout argument that shadows a project budget, so hooks run on
hookTimeout: 120000and bodies ontestTimeout: 60000. No file undertests/integration/passes a hook argument now. I kept the two per-test caps that a comment already justifies. No budget is reduced anywhere.Audit of every per-call timeout argument under
tests/integration/:agent_loader/agent_dirname_test.tsbeforeAll,it,afterAllapp_loader/app_loader_test.tsbeforeAll, 4it, 2afterAllskills/script_js/agent_test.tsbeforeAll,itadk_web/webui_test.tsbeforeAll, outerdescribea2a/basic/a2a_agent_test.tsbeforeAlla2a/stream/stream_test.tsbeforeAlla2a/input_required/input_required_test.tsbeforeAllbuild_setup/build_setup_test.tsbeforeAll,afterAllbuild_setup/build_setup_test.tsittools/run_skill_script_tool_test.tsitWhy the two keeps are not misses. The
build_setupbodies are a deliberate 20 s cap, and its comment tells readers not to raise it for hook flakes. Therun_skill_script_tool_testbodies are capped at 40 s to stay aboveUnsafeLocalCodeExecutor's 30 stimeoutSeconds, so the executor's own error surfaces first. Both record their measurement, which is the standardvitest.config.tsnow states.build_setup'sHOOK_TIMEOUTonly restated the project value, so I deleted it. Its comment held the measurement behind the 120 s budget, so that text moves toINTEGRATION_HOOK_TIMEOUT_MS, together with the invariant the dropped hook arguments now rely on: the budget must stay above the 60 s server start watchdog intest_api_server.ts, so a server that boots but never announces itself fails with the watchdog's message.I renamed
TEST_TIMEOUTtoSERVER_START_TIMEOUTin the two a2a suites. It is a rename only. The constant's one remaining reader isstartFailureTimeout, so the old name described a budget it no longer sets.vitest.config.tschanges two doc comments and no value. The old test-budget wording, "matches the largest per-file timeout in the repo", stops being true once the 60000 per-file constants go.Prettier collapses each
it(name, body, timeout)call back toit(name, body)once the third argument goes, so most of the diff is re-indentation. Read it withgit diff -w: no assertion, fixture, spawn command, cleanup step or test name changed.Collision check. I listed all 200 open PRs on the fork and matched them against every file here. Two siblings overlap. #791 drops the
beforeAllarguments in the three a2a suites and inwebui_test.ts, and #792 rewrites theINTEGRATION_HOOK_TIMEOUT_MScomment and deletesHOOK_TIMEOUT. I first shipped only the rows they do not cover, but a review found that subset incoherent: it left five hook arguments shadowing the project budget, one of them in a file this PR already edits. This PR now sweeps all of them, so it supersedes both siblings on those lines. Two differences from #791 are deliberate: it deletesstartFailureTimeoutfrom the a2a suites, which removes the watchdog that makes a failed start report itself, and it leaveswebui_test.ts's suite-level 20000. #662 and #766 also touch two of these files, but only theirafterAllbodies.Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.
No new test file. This change adds zero executable statements; it only removes arguments. The resolved timeout is Vitest runner state, so a test asserting "the hook budget is 120000" would assert the config file back to itself. The existing integration tests are the subject of the change and I ran them.
tests/integration/tools/run_skill_script_tool_test.tskeeps its budget and is not in the diff; it passes too (28 passed | 8 skipped together withbuild_setup).npm run build,npm run lintandnpm run format:checkall pass on the pushed commit.CI note. On the first push
run-testspassed on all three runners, after the windows-latest job failed twice and passed on the third attempt. Both failures were a pre-existing flake this PR does not touch: a test with no explicit budget hits Vitest's 5000 ms default while it waits on a child process. The two attempts failed on two different tests,tests/e2e/tools/mcp/load_mcp_resource_e2e_test.ts(projecte2e) andcore/test/code_executors/unsafe_local_code_executor_test.ts(projectunit:core). Neither project is touched here; this PR changes only theintegrationproject. The sameunsafe_local_code_executortimeout failed on the unrelated branchfix/vertex-session-test-single-genai-apiclient.Proof the change is load-bearing. I appended
await new Promise((r) => setTimeout(r, 45_000));to thebeforeAllbody ofagent_dirname_test.tsand ran the file twice.With the 40000 argument restored, all three fixtures fail:
With the argument removed, the same sleep passes on the project's 120 s budget:
Test Files 1 passed (1) / Tests 3 passed (3). I reverted the sleep before committing; it is not part of the diff. The other dropped hook arguments are the same mechanism at a different number.Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.
To see the old failure, add
const T = 40000;and pass it as the third argument of that file'sbeforeAll, then append a 45 s sleep to the hook body.Checklist
[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.