Fix: clean up compiled-agent temp directories synchronously on process exit - #653
Closed
AmaadMartin wants to merge 3 commits into
Closed
Fix: clean up compiled-agent temp directories synchronously on process exit#653AmaadMartin wants to merge 3 commits into
AmaadMartin wants to merge 3 commits into
Conversation
added 3 commits
August 4, 2026 12:33
…s exit A process 'exit' listener must be fully synchronous: Node terminates as soon as the last listener returns and drops any pending promise or queued I/O. The AgentLoader constructor installed an async listener that awaited disposeAll(), so the compiled-agent directory created by AgentFile.load() survived every run. The SIGINT/SIGUSR1/SIGUSR2/uncaughtException handlers call process.exit(), which synchronously emits 'exit', so Ctrl-C leaked too. Add AgentFile.disposeSync()/AgentLoader.disposeAllSync() and call the sync path from the listener. Also dispose the AgentLoader that AdkApiServer builds for itself in stop(); a loader supplied through ServerOptions keeps belonging to the caller.
Guard disposeSync on cleanupDirPath, which load() always assigns next to cleanupFilePath, so the removal target needs no unreachable fallback. Cover the remaining branches: the SIGINT handler calling process.exit(), and disposeAllSync closing the directory watcher.
Spying on a genuine AgentLoader removes the cast to the interface and exercises the real disposeAll().
This was referenced Aug 6, 2026
Owner
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
Closes: #issue_number
Related: #issue_number
Problem:
adk web,adk api_serverandadk deploy ...leave a compiled-agent scratch directory behind in the OS temp directory on every run, and a new one accumulates each time.AgentLoader's constructor installs its cleanup as an async'exit'listener:Node requires an
'exit'listener to be fully synchronous: the process terminates as soon as the last listener returns, and any pending promise or queued I/O is dropped.AgentLoader.disposeAll()awaitsAgentFile.dispose(), which awaitsfsPromises.unlink()andremoveFolder(). None of those awaits ever resolve during exit, so the output directory created byAgentFile.load()(getTempDir('adk_agent_loader'), containing the compiled<agent>.cjs/.mjsand anode_modulessymlink) survives the process.Ctrl-C leaks for the same reason:
SIGINT/SIGUSR1/SIGUSR2/uncaughtExceptionall take the{exit: true}branch and callprocess.exit(), which synchronously emits'exit'— whose listener is broken as above.process.exit()also cuts the current stack, so thetry/finallyblocks that callagentLoader.disposeAll()incli_deploy_cloud_run.tsandcli_deploy_agent_engine.tsare never reached either.Separately,
AdkApiServer.stop()closed the HTTP server and nothing else. When noagentLoaderis supplied throughServerOptions, the constructor builds one itself — that loader and its compiled directories outlive everystart()/stop()cycle of an embedded server.Solution: a synchronous best-effort cleanup path, which is the only construct that can do any work at all inside an
'exit'listener.AgentFile.disposeSync(): voidremoves the compiled output directory withfs.rmSync(dir, {recursive: true, force: true}).force: truemakes a missing target a no-op; thecatchcoversEBUSY/EPERM, which is a real possibility on Windows, and reports through the module's existingAdkLoggerrather than rethrowing — at exit there is no caller left to handle a failure, and one failed unlink must not stop the remaining agent files from being cleaned up or change the exit code.AgentLoader.disposeAllSync(): voidcloses the directory watcher and callsdisposeSync()on every preloaded agent file, mirroringdisposeAll().'exit'listener dropsasync/awaitand callsdisposeAllSync(). The signal handlers are unchanged: they still callprocess.exit(), which synchronously emits'exit', which now performs real cleanup. Ctrl-C stays instantaneous — no timer, no re-entrancy flag, no second-signal escape hatch, no new latency on the interactive path. The rejected alternative (awaitingdisposeAll()in the signal handler with a bounded timeout plus a force-exit on a second signal) adds all of that and still leaves plain'exit'broken.AdkApiServerrecordsownsAgentLoaderandstop()disposes the loader it built itself in afinally, so the loader is released even whenserver.close()reports an error. A loader supplied throughServerOptions.agentLoaderbelongs to the caller and is never disposed.Notes on the implementation:
fs.rmSyncdoes not follow symlinks, so it unlinks thenode_modulessymlink thatlinkProjectNodeModules()creates inside the output directory rather than deleting the project's realnode_modules. This is load-bearing and is pinned by a test.removeFolderSyncwas added tofile_utils.ts: a singlefs.rmSynccall is not a helper cluster, the existingremoveFolderhas different (swallow-and-console.error) semantics, andfile_utils.jsis wholesale-mocked byagent_loader_test.ts— routing through it would make the new tests assert against a mock instead of the real filesystem.disposeSync()guards oncleanupDirPath, whichload()always assigns in the same block ascleanupFilePath, so there is no unreachable?? cleanupFilePathfallback to leave as a permanently-uncovered branch. Behaviour is identical in every reachable state: an agent file that was never compiled owns no artifact, stays usable, andgetFilePath()keeps returning the original source path.dispose()/disposeAll()are untouched and keep their exact semantics.AgentLoader/AgentFileare not exported fromdev/src/index.ts, so the two new methods are not new public API surface.stop()gains theasynckeyword; its declared return type is unchanged.AdkApiServer.stop()now disposes a self-constructed loader. Callers that inject their own loader see no change; everyAdkApiServerbuilt indev/test/server/adk_api_server_test.tstoday injects anagentLoader, so no existing test changes behaviour.Collision check. All 551 open PRs on this fork were listed and every plausibly adjacent one was diffed before starting:
#511 fix/agent-loader-opt-in-process-handlers— refactors the same constructor block into an opt-ininstallProcessHandlers(), but its listener is() => void this.disposeAll(), i.e. still the async-dropped-work bug. It does not fix this leak. The two changes touch adjacent lines and will be reconciled mechanically by whichever lands second; this diff is deliberately minimal to keep that easy, and does not introduce an opt-in flag.#452 fix/adk-api-server-stop-close-all-connections— also editsstop(), but only addscloseAllConnections()inside the close callback; different concern, no loader disposal.#604 fix/atomic-temp-dir-creation/#612— makegetTempDiratomic (mkdtemp); same files, unrelated concern.#633 feat/lazy-agent-discovery— splits discovery from loading; adds no sync disposal.No open PR implements synchronous cleanup or loader ownership in
stop(), so this is built onmainrather than stacked.Two related defects found while investigating are deliberately out of scope and are tracked separately: the
await usingmisuse inadk_api_server.tsthat disposes a cachedAgentFileafter every request, and theadk deploystaging folder (options.tempFolder) that leaks on Ctrl-C.Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.
CI is green on ubuntu-latest, macos-latest and windows-latest. The windows job needed re-runs for two flakes that are unrelated to this diff and are documented on other branches:
tests/integration/app_loader/app_loader_test.ts > should discover apps vs agents ...timing out at 40s (also failed onfeat/lazy-agent-discovery, run 30911147137) andcore/test/code_executors/unsafe_local_code_executor_test.ts > should execute shell code and return stdouttiming out at 5s (also failed onfix/dev-workspace-undeclared-dependencies, run 30828447776 — and this PR changes no file in thecorepackage).12 new test cases, all additive — no existing test was edited, renamed, skipped or deleted.
dev/test/utils/agent_loader_test.tsdisposeSync()removes the compiled output directory, with noawaitbetween the call and the assertion — that ordering is what pins the bug — andgetFilePath()then throws'Agent is disposed and can not be used'.disposeSync()twice is a no-op (the second call issues no furtherrmSync).disposeSync()afterdispose()touches the filesystem again — the mutual-idempotency invariant.AgentFilestays usable afterdisposeSync():getFilePath()still returns the original source path and the source file is untouched.fs.rmSyncfailure is swallowed:disposeSync()does not throw, the directory survives, and the file is still marked disposed.disposeSync()does not follow thenode_modulessymlink —<agentsDir>/node_modules/@google/adkstill resolves afterwards.AgentLoader.disposeAllSync()removes every preloaded agent's directory, again with noawaitin between.'exit'listener the constructor installs cleans up synchronously. The listener is located by diffingprocess.listeners('exit')and invoked directly rather than viaprocess.emit('exit'), because every other loader built earlier in the file has its own listener registered; the listeners this test adds are removed in afinally.SIGINThandler callsprocess.exit()(which is what synchronously emits'exit'), pinning the Ctrl-C chain.disposeAllSync()closes the directory watcher, so the sync exit path leaves no watcher behind.dev/test/server/adk_api_server_test.ts(newdescribe('AgentLoader ownership'))stop()disposes the loader the server created itself. This drives a realesbuildcompile of a dependency-free agent fixture throughGET /list-apps(which is what triggerspreloadAgents();start()alone does not), asserts one directory appeared in a scratch root the test owns, then asserts it is empty afterawait stop().stop()releases its own loader even whenclose()errors — a secondstop()rejects withERR_SERVER_NOT_RUNNINGanddisposeAll()still runs, which is the error path thefinallyexists for.stop()does not dispose a caller-supplied loader (a realAgentLoaderwith a spieddisposeAll).Proof that each new test can fail. Every test was run against mutated source; each mutation was reverted before the next.
'exit'handler back toasync+await this.disposeAll()AssertionError: expected true to be false(the directory survives)disposeSync()body made asynchronous (void fsPromises.rm(...))expected true to be falsefs.rmSync(target, {recursive: true, force: true})→fs.rmSync(this.cleanupFilePath, {force: true})try/catchremoved fromdisposeSync()expected [Function] to not throw an error but 'Error: EBUSY: resource busy or locked' was throwndisposedshort-circuit dropped fromdisposeSync()expected 3 to be 2/expected 4 to be 3(an extrarmSync)AgentFilemarked disposed anywayAgent is disposed and can not be usedthis.watcher?.close()removed fromdisposeAllSync()expected "close" to be called 1 times, but got 0 timesif (exit) process.exit()→this.disposeAllSync()expected [Function] to throw an errorownsAgentLoaderguard removed fromstop()expected "disposeAll" to not be called at all, but actually been called 1 timesdisposeAll()call removed fromstop()entirelyexpected [ 'agent-0' ] to deeply equal []finallyinstop()replaced by sequential statementsexpected "disposeAll" to be called 2 times, but got 1 timesCoverage. 100% of the statements and branches added or changed by this PR are covered (measured with
@vitest/coverage-v8over the two suites):AgentFile.disposeSync,AgentLoader.disposeAllSync, the constructor's exit handler (including theif (exit)branch, which was uncovered before this PR), theownsAgentLoaderfield and both sides of the guard instop().One deviation worth flagging: the plan called for
vi.spyOn(nodeFs, 'rmSync')to exercise the failure branch. That is not possible — Vitest rejects it with "Cannot spy on exportrmSync. Module namespace is not configurable in ESM". The test file therefore installs a pass-throughvi.mock('node:fs')that delegates every export to the real implementation and only makesrmSyncthrow while a flag is set (the same mock recordsfs.watchinstances, becauseFSWatcheris not an export ofnode:fsand capturing the instance is the only way to observe that the loader closed it). Noany, no@ts-expect-error, noeslint-disable, and no type-checker or linter suppression of any kind is added by this PR.Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.
ls <os.tmpdir()>/adk_agent_loaderand note the current entries, if any.npx adk api_server ., thencurl localhost:8000/list-appsso the agent is compiled. Confirm a new<uuid>directory appeared under<os.tmpdir()>/adk_agent_loader.npx adk web .and confirm the same.npx adk deploy cloud_run . --project <project> --region <region>and interrupt it with Ctrl-C mid-deploy; theadk_agent_loaderdirectory is gone. (The separate--temp_folderstaging directory is a known, separately tracked leak and is still expected to remain.)No new integration test is warranted: the existing suite already spawns real
adk web/adk api_serverprocesses and kills them withSIGINT, and this fix stops those runs littering the temp directory without changing any observable server behaviour.Checklist
[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.