Fix: poll agent engine operations without a trailing sleep - #777
Open
AmaadMartin wants to merge 3 commits into
Open
Fix: poll agent engine operations without a trailing sleep#777AmaadMartin wants to merge 3 commits into
AmaadMartin wants to merge 3 commits into
Conversation
added 3 commits
August 7, 2026 07:56
Both LRO poll loops raced the poll against a fixed sleep with Promise.all, so each iteration cost max(poll, interval). The iteration that observed done still paid its own sleep, adding 1000 ms to every createSession and 5000 ms to every agent engine deploy. Sleep between polls instead. The first poll still fires immediately, the attempt budget is still 30 polls, and the timeout error strings are unchanged.
Four tests: the first poll must settle the promise with the clock still at its start value and no pending timer, and the not-done path must still issue 30 polls separated by 29 interval sleeps before it times out.
…mers Both agent engine deploy timeout tests attached their rejects assertion after the timer advance loop. The loop now drops its trailing sleep, so the deploy rejects one advance earlier and Node reported the rejection as unhandled, which failed the run under load. The assertion and its message are unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
Link to an existing issue (if applicable):
Or, if no issue exists, describe the change:
Problem: Two long-running-operation poll loops race the poll against a fixed sleep with
Promise.all, so every iteration costsmax(poll, interval). The iteration that observesdonestill pays its own sleep.VertexAiSessionService.createSession()therefore blocks about 1000 ms after the session already exists, andadk deploy agent_engineblocks about 5000 ms after the Reasoning Engine already exists.Solution: Sleep between polls instead of alongside them. The loop now delays only when
attempts > 0, so the first poll still fires immediately and the poll that observesdonereturns with no timer pending. The interval moves into a named constant in each file (POLL_INTERVAL_MS, 1000 ms in core and 5000 ms in dev). The attempt budget stays at 30 polls and the timeout error strings are byte-identical, so the worst case only loses the trailing sleep (30 polls separated by 29 sleeps).Collision check: I listed all 100 open PRs on the fork and read the diffs of every PR that touches either file (#759, #596, #590, #586) plus the session-service PRs (#503, #474, #657, #563, #512, #622). None of them changes a poll loop. No overlap, so this branches from
main.The two other hand-rolled poll loops in
core/src/code_executors/agent_engine_sandbox_code_executor.tshave a different defect (they sleep before the first poll) and are out of scope here.Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.
Four new tests, two per loop. Each pair pins both halves of the schedule: the first poll settles the promise with the fake clock still at its start value and
vi.getTimerCount() === 0, and the never-done path still issues exactly 30 polls separated by 29 interval sleeps before it throws the unchanged timeout error.One change to existing tests, in its own commit. The two deploy timeout tests attached their
expect(...).rejectsassertion after the 30-iteration timer advance loop. The loop now drops its trailing sleep, so the deploy rejects one advance earlier and Node reported the rejection as unhandled; the first CI run failed on that with 3140 tests passing andErrors 1 error. I moved the assertion above the advance loop and awaited it after. The assertion, the matcher and the expected message are unchanged. No test was skipped, weakened or deleted.Commands run on the pushed commit:
npm run ts:checkreports the same pre-existing errors before and after this branch (diffof the two runs is empty). This change adds none.CI is green on all four legs. The macOS leg first failed on the known
tests/integration/app_loader/app_loader_test.tsdiscovery timeout, which also fails on an unmodified-main control run and is unrelated to this change; it passed on retry.Measured dead latency,
core/test/sessions/vertex_ai_session_service_test.ts:Six
createSessiontests each took about 1002 ms before, because the defaultcreateInternalmock has nodonefield and those tests do not fake timers. They now take about 1 ms each. The dev file was already fast because itsbeforeEachstubssetTimeoutto run synchronously.Coverage of the changed lines is 100%. Both sides of
attempts > 0and both loop exits are exercised: the v8 report lists no uncovered statement incore/src/sessions/vertex_ai_session_service.ts:185-192ordev/src/cli/deploy/cli_deploy_agent_engine.ts:182-193.Mutation testing. I restored the
Promise.allrace in each file in turn and re-ran the suite. All four new tests failed:AssertionError: expected false to be trueatexpect(resolved).toBe(true). The create is parked on the trailing 1000 ms timer.AssertionError: expected 1 to be +0atexpect(vi.getTimerCount()).toBe(0). The unfixed loop leaves the 30th sleep pending. The poll-count assertions still passed here, sogetTimerCount()is the assertion that carries the signal.Error: Test timed out in 5000ms.The deploy never settles inside its budget.AssertionError: expected "spy" to be called 30 times, but got 4 times.Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.
A real reproduction needs a GCP project and an Agent Engine, so the fake-clock tests above are the evidence. To reproduce by hand:
VertexAiSessionServiceat asessionsstub whosecreateInternalresolves{name: 'operations/op-1', done: false}and whosegetSessionOperationInternalresolves{done: true, response: {...}}.createSession({appName: '12345', userId: 'u'})call.The deploy path reproduces the same way against
client.agentEnginesInternal.getAgentOperationInternal, with 5000 ms instead of 1000 ms.Checklist
[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.