Fix: persist skill script output files to the artifact service instead of the agent process cwd - #410
Open
AmaadMartin wants to merge 6 commits into
Open
Fix: persist skill script output files to the artifact service instead of the agent process cwd#410AmaadMartin wants to merge 6 commits into
AmaadMartin wants to merge 6 commits into
Conversation
added 5 commits
July 31, 2026 12:20
Skill script output files were written into the agent process's current
working directory by materializeFiles' implicit default, never persisted
as artifacts, and returned to the model with their raw base64 bytes.
Both skill script tools now hand their executor output to
saveScriptOutputs, which saves each file through Context.saveArtifact
(recording an artifactDelta) and returns only {name, mimeType} per file.
materializeFiles loses its process.cwd() default so the footgun cannot
be re-armed, and the leaked output.txt at the repo root - itself a
product of that default - is deleted.
Adds unit coverage for saveScriptOutputs (encoding normalization, artifact delta recording, versioning on repeat filenames, the no-artifact-service warning path and the partial-save-failure path) and for toBase64Content. Both skill tool suites drop the vi.mock of file_utils and the 'calls materializeFiles with output files from executor' cases, which pinned the removed process.cwd() write. They are replaced by tests that assert the artifact is saved, that the response carries no file bytes, that the process working directory is unchanged, and that a missing artifact service produces an explicit warning.
Replaces the four integration tests that asserted the process.cwd() write and the _2 collision rename - the behaviour this change removes - with real-executor tests that save through a session-scoped InMemoryArtifactService, read the bytes back, prove a repeat run creates artifact version 1 rather than a renamed file, and cover the no-artifact-service warning path.
The script_js CLI end-to-end test read its three generated files straight out of the directory the agent process was started from - the write this change removes. It now runs the CLI with a file-backed artifact service, asserts the three files are absent from that directory, and compares the saved artifacts against the same expected/ fixtures, so the content assertions are preserved rather than dropped.
…ation assertions
UnsafeLocalCodeExecutor skips input files by comparing File.name (which
uses /) against an fs.readdir({recursive:true}) entry (which uses \ on
Windows), so on Windows a skill's own input scripts come back as output
files. Assert the script's output by containment rather than list
equality so these tests pin this change's behaviour and not that
pre-existing executor defect, which is tracked separately.
- Export the existing ScopedArtifactService from common.ts and delete the seven-method duplicate the integration test util had grown, which only existed because the real class was not public. - Drop materializeFiles' Promise<File[]> return and its accumulator: the skill tools were the only consumer of the created-file list, and the one remaining caller discards it. Document the in-place file.name mutation the collision loop performs, which unsafe_local_code_executor relies on. - Collapse saveScriptOutputs' redundant empty-output early return into the no-artifact-service guard and drop describeFile in favour of one up-front name/mimeType projection. - Build a real InvocationContext in the new helper unit test instead of casting a literal. - Add a case pinning that no warning is emitted when a script produced no files and no artifact service is configured.
This was referenced Jul 31, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
No existing issue.
Problem: both skill script tools ended their happy path with the same line —
result.outputFiles = await materializeFiles(result.outputFiles)(
run_skill_script_tool.ts:145,run_skill_inline_script_tool.ts:144) — andmaterializeFilesdefaulted its destination toprocess.cwd()(
core/src/utils/file_utils.ts:17). Three defects follow from that one default:produces is written into the working directory of the agent server process —
a directory no caller named and the session cannot address. It is not
per-session either: two sessions running the same skill write into the same
directory and collide, silently renamed
_2,_3, … by the loop atfile_utils.ts:37-52. The repository carried the proof: a trackedoutput.txt(contents:
hello) at the repo root, left behind by the change thatintroduced skill script execution. Reproduced below — reverting the fix and
running the suites re-creates that exact file plus six siblings.
saveArtifactcall existed,so outputs were invisible to
load_artifacts, to the artifact REST surfaceand to the UI — inconsistent with the built-in code-execution flow, which does
persist executor output files
(
code_execution_request_processor.ts:503-513).CodeExecutionResultwas returned as the tool response, includingFile.content— the encoded bytes of every output file. A script emitting a2 MB PNG puts ~2.7 MB of base64 into the prompt.
Solution: send the bytes where the session can reach them, and send the model
only a manifest.
core/src/tools/skill/script_output_utils.tsexportssaveScriptOutputs(toolContext, result), which saves each output file throughContext.saveArtifact(name, {inlineData})— the same API the code-executionflow uses, which also records
eventActions.artifactDelta[name] = versionandis what surfaces a file to clients. It returns
{stdout, stderr, outputFiles: [{name, mimeType}], warning?}; theSkillScriptResponsetype makes it impossible to put bytes on that response.Saves run concurrently via
Promise.allSettled, so one failing file does notserialise or sink the rest.
toBase64Content(file)incode_execution_utils.ts, co-located with theFiletype it operates on. Artifact payloads must be base64(
FileArtifactServicedoesBuffer.from(data, 'base64')), but the twoexecutors disagree:
UnsafeLocalCodeExecutordeclarescontentEncoding: 'utf-8'for text types whileAgentEngineSandboxCodeExecutorleaves itundefined on content that is already base64. The rule is therefore
"
utf-8→ encode, anything else → pass through". It deliberately does notreuse
getEncodedFileContent, which sniffs withisBase64Encodedandmisclassifies plain text that happens to be valid base64 —
hellois exactlysuch a string, and is exactly what a skill script writes.
materializeFiles(files, dir: string): Promise<void>— theprocess.cwd()default is removed, so the compiler now prevents the next caller from
re-arming the footgun. Its
Promise<File[]>return and thecreatedFilesaccumulator behind it are removed too: the skill tools were the only consumer
of that list, the one remaining caller
(
unsafe_local_code_executor.ts:182, which already passes anfs.mkdtempdirectory) discards it, and no test asserted it. The collision loop and the
in-place
file.namemutation stay —unsafe_local_code_executor.ts:274matches output candidates against the mutated
inputFiles[].nameto skipinput files, and
file_utils_test.tspins the suffix behaviour for duplicatenames within one batch. That mutation is now documented on the function
instead of being implied by the return value.
ScopedArtifactServiceis now exported fromcore/src/common.ts. It is theframework's own bridge between the two already-public artifact interfaces
(
BaseArtifactService→SessionArtifactService), and exporting it deletes aseven-method copy of it that the integration test util would otherwise carry
and that would silently drift from the real class. This is the one public API
addition beyond the two response types.
output.txtat the repo root. It is a leaked script outputfrom the mechanism this PR removes, and the only
grep -rn "output.txt"hitsare unrelated in-test filename literals.
Design notes:
The artifact service is the single destination; no
outputDirknob isadded. One destination means one contract. The artifact service is
session-scoped, versioned, already sanitizes filenames and rejects traversal
(
file_artifact_service.ts:476-516), is reachable by clients and byload_artifacts, and matches the built-in code-execution flow. A configurablehost directory would re-introduce a second, unversioned, cross-session-colliding
path for no demonstrated requirement.
Collisions are handled by artifact versioning, not by
_2/_3renaming.Running a script twice now yields versions 0 and 1 of one artifact.
A persistence failure is never a tool error.
stdout/stderralways comeback. No artifact service configured → the produced filenames are still
reported with an explicit
warning, and onelogger.warn; a rejectedsaveArtifact→ the saved subset plus awarningnaming the failures, witheach rejection reason logged (never returned, so it cannot enter the model
context). The existing
EXECUTION_ERRORcatch was deliberately not widened— a storage outage must not be reported as "Failed to execute script". No new
error codes.
A review round proposed collapsing this to a plain
Promise.alland letting arejection propagate, matching the ten-line loop in
code_execution_request_processor.ts:503-513. Declined, with the reasoninghere rather than in a comment. The same argument the reviewer accepted for
keeping the no-artifact-service warning ("turning a completed script run into
a hard tool error would be worse than reporting the loss") applies unchanged
to a GCS outage or a filename the artifact backend rejects: in both cases the
script ran and its stdout is worth returning. Concretely,
Promise.allherewould be worse than the reviewer's own description of the trade-off — the
tools do
return saveScriptOutputs(...)inside theirtry, and a returnedpromise's rejection is not caught by the enclosing
catchin an asyncfunction, so the failure would surface as an unhandled
runAsyncrejectionrather than as a mildly inaccurate
EXECUTION_ERROR. The processor is not aprecedent for this: it builds an event inside the framework's own error
handling, whereas this runs at a tool boundary that must return a response.
What the review did fix here: the redundant
outputFiles.length === 0earlyreturn is folded into the no-service guard, and
describeFileis replaced byone up-front
namesprojection. A new case,does not warn about a missing artifact service when the script produced no files, pins the behaviour that early return used to provide.Both tools are fixed in one change, on purpose. It is the same one-line
defect in sibling tools sharing the new helper. Splitting it would leave the
two tools with contradictory output semantics for a release, and would produce
a second PR conflicting on the same file. The
srcdiff is ~150 lines across6 files, one logical checkpoint, so it is not stacked.
Not modified:
DEFAULT_SKILL_SYSTEM_INSTRUCTION(the response alreadynames the saved artifacts; editing it would churn prompt-text assertions for no
behavioural gain). Out of scope:
stdout/stderrtruncation, andnormalising
contentEncodingin the executors themselves —toBase64Contenthandles that divergence at the point of use.
Breaking changes (both tool classes are
@experimental, andrunAsyncistyped
Promise<unknown>):outputFilesentries losecontentandcontentEncoding, and awarningfield may appear.
grep -rn "outputFiles" core/src dev/src integrations/srcfinds only the two tools and the code-execution processor, so the impact is
limited to the model-facing contract — which is the point of the fix.
process.cwd(). Anyone relying on that now gets asession-scoped, versioned artifact instead.
materializeFiles(files)now requiresdir, and returnsvoidinstead ofFile[]. It is not exported fromcore/src/index.tsorcore/src/common.ts,and all in-repo callers already pass a directory and discard the return, so
this is internal only.
ScopedArtifactServiceis newly exported (additive).No suppressions of any kind were added — no
any,as any,as never,@ts-expect-error,@ts-ignore,eslint-disable, or coverage-tool ignore, insrcor in tests, and the suppression grep over this diff now returnsnothing. The new unit test originally carried one
{…} as unknown as InvocationContext(the repo's existing fixture pattern forthat type, in 22
core/testfiles); a review round asked whether a real contextwas cheap enough, and it is — the new file now builds
new InvocationContext({invocationId, agent, session, pluginManager, artifactService})from a realLlmAgentandcreateSession. The fourpre-existing casts in the tool suites are left alone; they belong to fixtures
this PR only extends.
npx tsc --noEmitis red onmaintoday with 280 pre-existing errors; on this branch it is 279, and a per-file
diff of the two reports shows the single difference is one removed error in
run_skill_script_tool_test.ts(theas Filecast in the deleted test). Thischange introduces zero new type errors and removes one.
Collision check (run before implementing):
gh pr list --repo AmaadMartin/adk-js --state open --limit 100, plusgh pr diff --name-onlyonevery plausibly adjacent PR. Five overlaps found; none lands this change:
lands in process.cwd()" (stacked on Fix: materialize skill script output into a declared directory, never process.cwd() #298 "make the skill script output
directory configurable") touches nine of the same files and fixes defect 1 by
a competing design: it keeps writing to the host filesystem and adds an
outputDiroption toSkillToolset. It does not persist anything as anartifact and still returns raw
File.contentto the model, so defects 2 and 3survive it. This PR is deliberately not stacked on that branch: the guidance
to stack exists to keep a review digestible, and a child whose diff is dominated
by deleting its parent's central feature (
outputDir/materializeOutputFilesand their tests) is harder to review than the samechange against
main, not easier. The two are mutually exclusive designs forone defect and only one should land — this one, per the "single destination"
note above. Both remove the
materializeFilesdefault identically, so whicheverlands second needs a trivial resolution in
file_utils.ts.repo working tree" touches the same two integration files. It mitigates the
leak in tests; this PR removes its cause. Not stacked on, same reason.
containment) change
materializeFiles' body, which this PR does not touch —they compose cleanly with the signature change here.
it lands first, that hunk drops out cleanly.
Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
That last line is the human-visible form of the bug and is worth keeping: before
this change, running the two skill-script integration suites left
output_from_script*.txt,test_output_*.txtandtest_inline_output_*.txtin the repository root.
CI:
run-testsis green on all three matrix legs —ubuntu-latest,macos-latestandwindows-latest— running the fullnpm run test:coveragesuite (2691 tests), plus
npm run lint,npm run format:checkandnpm run docs:check. Two intermediate red runs are worth naming so they are notmistaken for flakes that were papered over: the first was the real
script_js/agent_test.tsregression described above, the second the real WindowstoEqualover-assertion described above, and both were fixed rather thanretried. One further macOS failure was
tests/integration/app_loader/app_loader_test.ts > should discover apps vs agents across directories and standalone filestiming out at 40000 ms — apre-existing, install-bound timing flake in a file this PR does not touch, which
passed on the same commit on ubuntu and windows and on re-run. After the review
revisions, one windows leg similarly timed out at 5000 ms in
core/test/code_executors/unsafe_local_code_executor_test.ts > should execute shell code and return stdout— a shell cold-start flake in a test this PR doesnot touch (the very effect the existing
TEST_EXECUTION_TIMEOUTcomment in theskill integration suite documents); it passes locally and passed on re-run.
Added:
core/test/tools/skills/script_output_utils_test.ts(new, 10 cases) — noscript output, and no script output with no artifact service (which must not
warn about discarding zero files); save-and-summarise with an explicit
assertion that
Object.keys(outputFiles[0])is exactly['name', 'mimeType']; the threeencoding cases (utf-8 encoded, base64 passed through, undeclared treated as
base64); artifact-delta recording; a repeated filename becoming version 1; the
no-artifact-service warning path; and a partial failure where the second of
three saves rejects, asserting the other two are returned, the warning names
the failed file, the reason is logged rather than returned, and nothing throws.
It drives a real
InMemoryArtifactServicebound to a session, so the assertionsare about bytes that actually round-tripped, not about a recording spy.
core/test/code_executors/code_execution_utils_test.ts— adescribe('toBase64Content')appended next togetEncodedFileContent; the fileis otherwise unrestructured.
saves script output files to the artifact service and omits file bytes from the response,does not write script output files to the process working directory(snapshotsfs.readdir(process.cwd())either side of
runAsync; this is the regression guard for the reporteddefect), and
reports produced files with a warning when no artifact service is configured.UnsafeLocalCodeExecutor, no mocks:saves script output files to the artifact service(loads the artifact backand decodes it to
hello from script file, the end-to-end proof that theencoding normalization is right, and asserts the file is absent from
process.cwd()),creates a new artifact version instead of a renamed file on repeat runs(two versions of one artifact, no_2.txtanywhere), and theno-artifact-service warning path.
tests/integration/tools/artifact_service_test_utils.ts— two small helpersshared by both integration suites:
createSessionArtifactService()(anInMemoryArtifactServicewrapped in the now-exportedScopedArtifactService)and
loadArtifactText(). An earlier revision hand-rolled a seven-methodsession-scoped adapter here because
ScopedArtifactServicewas not public;a review round correctly called that a copy that would drift, so the real
class is exported instead and the duplicate is gone.
Existing tests rewritten in place — declared explicitly, invoking the
documented exception. The repo's guidance is to add a test rather than edit
one, precisely so a reviewer can tell a wrong assertion from an inconvenient
one. Eight cases are edited in place here, and all eight are the documented
"the existing test encodes the wrong behaviour" case: every one of them
asserts either that a file appears in
process.cwd()(or in the CLI's launchdirectory) or that a collision produces a
_2.txtrename — i.e. each pinsexactly the defect being removed, so no untouched original can survive the fix.
The replacements are strictly stronger: they assert the working directory stays
clean, that the artifact bytes round-trip, and that a repeat run increments the
artifact version. The two
vi.mock('../../../src/utils/file_utils.js')fixturesdeleted alongside them are part of the same exception — with the mock in place
the new cwd-snapshot tests could never observe a real write, so keeping it would
have made the regression guard vacuous.
The eight, enumerated:
core/test/tools/skills/run_skill_script_tool_test.tscalls materializeFiles with output files from executorcore/test/tools/skills/run_skill_inline_script_tool_test.tscalls materializeFiles with output files from executortests/integration/tools/run_skill_script_tool_test.tscreates files in process.cwd returned from executiontests/integration/tools/run_skill_script_tool_test.tshandles file collisions by appending a numeric suffixtests/integration/tools/run_skill_inline_script_tool_test.tscreates files in process.cwd returned from executiontests/integration/tools/run_skill_inline_script_tool_test.tshandles file collisions by appending a numeric suffixtests/integration/skills/script_js/agent_test.tsshould run agent with skills successfully(read its three files out of the CLI's launch directory)vi.mock('file_utils')fixtures, per the paragraph aboveDetail on what each pinned and what replaced it:
core/test/tools/skills/run_skill_script_tool_test.tscalls materializeFiles with output files from executormaterializeFilesis called with the cwd default. The tool no longer calls it at all. Thevi.mock('../../../src/utils/file_utils.js')block went with it — and removing that mock is what lets the new cwd-snapshot test see a real write.core/test/tools/skills/run_skill_inline_script_tool_test.tscalls materializeFiles with output files from executortests/integration/tools/run_skill_script_tool_test.tscreates files in process.cwd returned from executionfs.access(path.join(process.cwd(), 'output_from_script.txt'))succeeds — literally the defect. Its inverse is now asserted bysaves script output files to the artifact service.tests/integration/tools/run_skill_script_tool_test.tshandles file collisions by appending a numeric suffix_2rename that artifact versioning replaces. Its subject is now covered bycreates a new artifact version instead of a renamed file on repeat runs, which asserts versions[0, 1]and that no_2.txtexists.tests/integration/tools/run_skill_inline_script_tool_test.tsEvery other test in all five files is untouched. No test was skipped, disabled,
.only'd, or weakened, and no case was deleted without a named replacement.One further test rewritten, caught by CI, not by me. The first push of this
branch went red on
run-tests (ubuntu-latest):tests/integration/skills/script_js/agent_test.ts— a CLI end-to-end test thatspawns
adk runin a fixture directory and then readsephemeral_entanglement.md,index.htmlandsketch.jsout of thatdirectory, because
process.cwd()of the spawned agent process was thefixture directory. It is the same defect one layer up, and it is the best
possible demonstration of it: this suite is why the CLI's launch directory
accumulates model output. Rather than delete the content assertions, the test now
runs the CLI with
--artifact_service_uri file://<mkdtemp>, asserts the threefiles are absent from the fixture directory, and compares the saved artifacts
against the same
expected/fixtures — so every original assertion survives,pointed at the new destination. Locally: passes; with the fix reverted it fails
with
AssertionError: promise resolved "undefined" instead of rejecting(thefile is back in the launch directory).
Two pre-existing Windows defects surfaced while validating this, neither fixed
here. Both are called out at their call sites and queued separately:
UnsafeLocalCodeExecutorskips input files when scanning for outputs bycomparing
File.name(forward slashes, e.g.scripts/hello.js) against anfs.readdir({recursive: true})entry (backslashes on Windows). Thecomparison never matches there, so every input file is reported as an
output — the windows-latest leg returned 12 output files for a script that
writes one. The integration tests therefore assert the script's output by
containment rather than list equality, with a comment naming the cause; the
exact-shape assertion (
toEqual([...]), and theObject.keys(...)check thatno bytes are present) lives in the unit tests, where the executor is a mock.
Note this defect is worsened in kind by nothing in this PR — it previously
copied every skill script into the launch directory, and now saves them as
artifacts instead — but it should be fixed on its own.
getArtifactServiceFromUrimanglesfile://URIs on Windows (below).The CLI test builds its
file://URI as`file://${root.split(path.sep).join('/')}`rather than with
pathToFileURL, becausegetArtifactServiceFromUristrips thescheme with
uri.split('://')[1]: a canonicalfile:///C:/…would leave/C:/…, whichpath.resolvemangles on Windows. That is a pre-existinglimitation of the CLI's URI parsing, is commented at the call site, and is out of
scope here — it has been queued separately rather than fixed in this PR.
Proof each test can fail. Each new test was run against mutated source and
observed to fail:
dir = process.cwd()inmaterializeFilesandresult.outputFiles = await materializeFiles(result.outputFiles); return result;inrun_skill_script_tool.tsdoes not write script output files to the process working directory→AssertionError: expected [ '…', …(30) ] to deeply equal [ '…', …(29) ].saves script output files to the artifact service and omits file bytes…→expected { stdout: 'script stdout', …(2) } to deeply equal { … }. Integration:expected [ { …(4) } ] to deeply equal [ { …(2) } ]— four keys instead of two, i.e. the file bytes back in the response. The run also left seven files in the repository root —output.txt,output_from_script.txt,_2,_3,_4,cwd_regression_output.txt,unsaved_output.txt— reproducing the defect, including the committedoutput.txtthis PR deletes.run_skill_inline_script_tool.tsinsteadtoBase64Contentalwaysreturn file.content;toBase64Content > base64-encodes content declared as utf-8andsaveScriptOutputs > base64-encodes utf-8 file content before saving→expected 'hello' to be 'aGVsbG8='. Integrationsaves script output files to the artifact service→expected '\ufffd\ufffde\ufffd…' to be 'hello from script file'(mojibake from double-decoding).saveScriptOutputsreturns{...result}(the rawFile[])Object.keys(…) === ['name','mimeType']assertion andexpected {} to deeply equal { 'a.txt': +0, 'b.txt': +0 }for the artifact delta.{stdout, stderr, outputFiles: []}with nowarningreports produced files with a warning…case in all three suites.savedreturns the saved subset with a warning when an artifact save fails: the failed file is reported as saved.tests/integration/skills/script_js/agent_test.tsfails withAssertionError: promise resolved "undefined" instead of rejecting, and the three generated files reappear in the fixture directory the agent was launched from.One further mutation, added with the review revisions: dropping the
outputFiles.length > 0guard on the no-service branch failsdoes not warn about a missing artifact service when the script produced no files, which starts reporting "0 output file(s) … discarded". Every mutation inthis table was re-run after the review revisions and still fails as recorded.
Coverage. Measured with
--coveragerestricted to the touched files:core/src/tools/skill/script_output_utils.tscore/src/code_executors/code_execution_utils.tscore/src/utils/file_utils.tsThe
file_utils.tsshortfall is lines 61-64, the pre-existing secondpath-traversal check; this PR changes only that function's signature, return
type and doc comment, adding no executable line to it. No
/* v8 ignore */,istanbul ignoreor any other coverage suppression was added.Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.
The integration tests above are the automated form of this — they drive the real
UnsafeLocalCodeExecutorwith no mocks. To reproduce by hand:scripts/create_file.jsisconst fs = require('fs'); fs.writeFileSync('report.csv', 'a,b');, wirenew SkillToolset([skill], {codeExecutor: new UnsafeLocalCodeExecutor()})into an agent, and run it with an artifact service configured (e.g.
InMemoryRunner). Have the model callrun_skill_script.report.csvappears in the directory you launched the server from,and its bytes are in the tool response.
{stdout, stderr, outputFiles: [{name: 'report.csv', mimeType: 'text/csv'}]},and
report.csvis loadable from the session's artifact service — includingvia the
load_artifactstool.report.csv, not areport_2.csv.stdout/stderrstill come back,outputFilesstill namesreport.csv, andwarningsays the file was discarded. Onelogger.warnis emitted. Nothingis written to disk.
git statusin the repository root after any of the above: clean.Checklist