Skip to content

Feat: cap stdout/stderr size in the skill script tool responses - #518

Open
AmaadMartin wants to merge 4 commits into
mainfrom
feat/cap-skill-script-output
Open

Feat: cap stdout/stderr size in the skill script tool responses#518
AmaadMartin wants to merge 4 commits into
mainfrom
feat/cap-skill-script-output

Conversation

@AmaadMartin

@AmaadMartin AmaadMartin commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

  1. Link to an existing issue (if applicable):
    N/A — no public issue tracks this.
  2. Or, if no issue exists, describe the change:
    Problem: RunSkillScriptTool and RunSkillInlineScriptTool hand the code executor's stdout and stderr straight back to the model with no size limit (core/src/tools/skill/run_skill_script_tool.ts, core/src/tools/skill/run_skill_inline_script_tool.ts, both previously ending their try block with return result;). A skill script that dumps a build log, a large JSON blob, or a looping stack trace puts its entire output into the prompt on the very next model turn. For a long-running agent that is an unbounded context-window and cost hazard, and it breaks the "bound every accumulator" rule that applies to anything growing per invocation.

Solution: Add a small shared helper, core/src/utils/truncate_utils.ts, and apply it to the stdout and stderr fields of both tools' success responses.

  • DEFAULT_MAX_OUTPUT_CHARS = 30_000, applied per stream (stdout and stderr are capped independently, so a large stdout never eats into the stderr budget). The constant lives in skill_toolset.ts beside the option it defaults, because the cap is skill-toolset policy rather than a property of string truncation; truncate_utils.ts stays a generic, import-free helper that the queued follow-ups can reuse without inheriting a constant named after skill script output.
  • truncateMiddle(text, limit) (both parameters required — no call site omits the limit) returns text unchanged when it fits, otherwise keeps ceil(limit/2) characters of head and limit - ceil(limit/2) of tail, replacing the elided middle with \n... [truncated <N> characters] ...\n where <N> is the number of characters removed. The preserved-content budget is therefore exactly limit; the returned string is marginally longer because it also carries the marker.
  • Configurable per toolset: new SkillToolset(skills, {codeExecutor, maxOutputChars: 4_000}). The option is surfaced as public readonly maxOutputChars and read by both tools in this same change, so it is not dead config. The model-facing FunctionDeclarations are unchanged — the cap is an operator control, and a model-settable cap would be a model-controlled way to defeat its own context budget.

Design notes / where this deviates from adk-python (deliberate, per the cross-language-parity rule that local convention wins for things that do not cross a wire boundary):

  • The numeric default matches adk-python's src/google/adk/tools/environment/_constants.py (MAX_OUTPUT_CHARS = 30_000), and the per-stream application matches ExecuteTool.run_async in src/google/adk/tools/environment/_execute_tool.py, which truncates each stream separately.
  • The truncation shape intentionally does not match. adk-python's truncate() (src/google/adk/tools/environment/_utils.py) keeps only the head. The hazard here is specifically a looping stack trace or a failing test suite, where the diagnostically useful content — the failure summary, the innermost frames — is at the end of the stream. Head-only truncation throws exactly that away, so this keeps both ends.
  • This is not a port of an existing adk-python skill-tool behaviour: adk-python's src/google/adk/tools/skill_toolset.py returns script stdout/stderr verbatim too. This borrows the environment toolset's mechanism for the JS skill tools.
  • The unit is characters, not bytes: it is what correlates with prompt-token cost, and byte-slicing UTF-8 can split a multi-byte code point for no benefit.
  • Both tools now return a copy of the executor's CodeExecutionResult (object spread) rather than the same object with outputFiles assigned in place. No caller in the repo depends on that identity — these two return statements are the only consumers — and a test pins that the executor's own result is no longer mutated.

Helper placement: truncateMiddle is generic string handling, not skill-specific, so it lives in core/src/utils/ with a feature-neutral name (truncate_utils.ts), not inlined at the top of either tool file and not prefixed with the feature that happened to need it first. Following the core/src/utils/file_utils.ts precedent it stays internal — it is not added to core/src/common.ts or core/src/index.ts, and its tests import it by relative path, exactly as core/test/utils/file_utils_test.ts does for materializeFiles.

Collision check (required before implementation): gh pr list --repo AmaadMartin/adk-js --state open --limit 1000 returned 418 open PRs; none truncates skill script output (gh pr diff over every plausibly adjacent PR found no truncat/maxOutputChars hit). Several PRs are textually adjacent — #517 (save output files as artifacts), #410 (output files to the artifact service), #437/#353/#298 (explicit script output directory) all touch the same runAsync tails and/or SkillToolset options — but each does a different thing, and they conflict with one another as much as with this PR, so stacking on any one of them would be arbitrary. This branch is based on current main and its diff is confined to the stdout/stderr fields; outputFiles handling is left byte-for-byte equivalent to main so those PRs stay mergeable.

Deliberately out of scope (not silently dropped): saving the full log to the artifact service (needs an artifactService-undefined guard, graceful degradation, a collision-resistant filename scheme, and its own error surface — and would collide head-on with #517/#410); capping the base64 content of outputFiles; capping LoadSkillResourceTool's content field. The last two have been queued as independent follow-ups that reuse this helper.

Testing Plan

Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.

No existing test was modified, skipped, weakened or deleted; every addition is a new it or a new describe block.

npx vitest run --project unit:core \
  core/test/utils/truncate_utils_test.ts \
  core/test/tools/skills/run_skill_script_tool_test.ts \
  core/test/tools/skills/run_skill_inline_script_tool_test.ts \
  core/test/tools/skills/skill_toolset_test.ts
# Test Files  4 passed (4) | Tests  53 passed (53)

npx vitest run --project integration \
  tests/integration/tools/run_skill_script_tool_test.ts \
  tests/integration/tools/run_skill_inline_script_tool_test.ts
# Test Files  2 passed (2) | Tests  19 passed | 4 skipped (23)

npm run build   # ok
npx eslint <the 9 changed files>          # clean
npx prettier --check <the 9 changed files> # clean

New coverage:

  • core/test/utils/truncate_utils_test.ts (new file, 9 cases): below/at/one-over the limit, even-limit split asserted as exact head and tail slices, odd-limit split (limit = 7 -> 4 head + 3 tail), exact removed-character count on a hand-written input, limit === 0 (marker only), negative limit clamped to 0, and the default (DEFAULT_MAX_OUTPUT_CHARS === 30_000, 30,000 chars unchanged / 30,001 truncated).
  • run_skill_script_tool_test.ts: new describe('output truncation') — stdout under the cap verbatim, stdout over the cap, stderr over the cap while a short stdout stays verbatim, both streams capped independently in one call, a custom maxOutputChars, and the executor's result not being mutated. Plus one new case for the pre-existing EXECUTION_ERROR path. This one is a deliberate drive-by, not load bearing for the cap: it exercises the catch block, which this diff does not touch, and it exists because MockCodeExecutor.shouldThrow was declared in this file but never used (the inline sibling already had the equivalent case). It is 17 lines of pure addition covering the error path adjacent to the try block being restructured; say the word and I will split it out.
  • run_skill_inline_script_tool_test.ts: the equivalent describe('output truncation'), driven through the existing confirmed-ToolConfirmation helper so the calls pass the security gate.
  • skill_toolset_test.ts: maxOutputChars defaults to DEFAULT_MAX_OUTPUT_CHARS, and reflects an explicitly passed option.

Measured coverage (--coverage.include limited to the changed files): core/src/utils/truncate_utils.ts is 100% lines / 100% branches / 100% functions. Every line added to the three modified files is covered; the residual uncovered lines in those files are pre-existing (_getDeclaration bodies, unrelated SkillToolset paths) and untouched here.

Proof the tests can fail. Coverage is a floor, not proof, so each new test was run against mutated source and confirmed to fail (all mutations reverted afterwards; the working tree is clean):

# Mutation Result
1 truncateMiddle body -> return text; 14 failed: expected 'abcdef' to be 'abc\n... [truncated 1 characters] ...\nef'
2 Math.ceil(cap / 2) -> Math.floor(cap / 2) 2 failed: expected 'abc\n... [truncated 93 characters] ...' to be 'abcd\n... [truncated 93 characters] ...'
3 marker count text.length - cap -> text.length 12 failed: expected 'abc\n... [truncated 6 characters] ...' to be 'abc\n... [truncated 1 characters] ...'
4 fits-guard <= -> < 4 failed: expected 'abc\n... [truncated 0 characters] ...\nde' to be 'abcde'
5 drop the clamp: Math.max(0, limit) -> limit 1 failed: expected '\n... [truncated 106 characters] ...\n' to be '\n... [truncated 6 characters] ...\n'
6 drop the stderr: line from run_skill_script_tool 2 failed: expected 'eee…' to contain '... [truncated 10 characters] ...'
7 drop the stderr: line from run_skill_inline_script_tool 2 failed: expected 'eee…' to contain '... [truncated 7 characters] ...'
8 SkillToolset ignores the option (always the default) 5 failed: expected '0123456789' to be '01\n... [truncated 6 characters] ...\n89'
9 mutate the executor result in place instead of spreading 1 failed: expected 'yyyyy\n... [truncated 40 characters] ...' to be 'yyyyy…' (50 chars)
10 pass stdout/stderr through untruncated (integration) 1 failed: expected 'SSS…' to contain '... [truncated 110 characters] ...'

Complexity review round 1 (three findings, all addressed in 301569f0, no behaviour change — the same 53 unit + 19 integration tests pass unchanged): the inline tool read the cap as this.toolset?.maxOutputChars ?? DEFAULT_MAX_OUTPUT_CHARS, but toolset is non-optional and maxOutputChars is always assigned, so both guards were unreachable — collapsed to the sibling's this.toolset.maxOutputChars; truncateMiddle's limit default was taken by no caller and existed only for the test that asserted it — made required, and that test became an explicit-limit assertion that the result is exactly limit characters of content plus the marker; and with no default left in the helper, DEFAULT_MAX_OUTPUT_CHARS moved to skill_toolset.ts. It remains internal: core/src/common.ts re-exports SkillToolset by name (export {SkillToolset} from ...), not export *, so the constant is not published. A new mutation was added for the move — changing the constant to 10_000 fails skill_toolset > maxOutputChars > defaults to DEFAULT_MAX_OUTPUT_CHARS when the option is omitted — and every mutation in the table above was re-run against the refactored source and still fails.

CI on the first pushed commit failed docs:check (typedoc --emit none --treatWarningsAsErrors) because a {@link DEFAULT_MAX_OUTPUT_CHARS} in the SkillToolset option doc resolved to a symbol that is not part of the documented public API. Fixed by naming the constant and its value inline rather than exporting the internal module just to satisfy a doc link. run-tests is now green on ubuntu-latest, macos-latest and windows-latest.

Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.

An automated no-mock end-to-end case was added to tests/integration/tools/run_skill_script_tool_test.ts (caps stdout and stderr of a real noisy JavaScript skill script): it runs a real script under UnsafeLocalCodeExecutor that writes 610 characters to stdout and 710 to stderr with maxOutputChars: 500, and asserts the exact head slice, the exact marker (... [truncated 110 characters] ... / ... [truncated 210 characters] ...) and the tail sentinels STDOUT_END / STDERR_END. This is a deliberate, small departure from "no new integration tests": a 500-character cap needs only ~600 characters of real subprocess output, so the case costs ~150 ms rather than the minutes a 30 KB flood would, and it is what proves the option is wired through to a real executor end to end. Mutation 10 above confirms it fails without the fix.

To reproduce by hand:

npm run build
npx vitest run --project integration tests/integration/tools/run_skill_script_tool_test.ts

With a live model: configure an LlmAgent with a SkillToolset whose skill has a script printing well over 30,000 characters, ask the agent to run it, and confirm the tool response carries the head, the marker with a plausible removed-character count, and the tail — and that the model's follow-up references the truncation rather than reasoning as though it saw the whole log. Repeat with maxOutputChars: 500 to confirm the option is honoured end to end.

Checklist

[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.

Amaad Martin added 3 commits August 2, 2026 04:34
Add a shared truncateMiddle helper and apply it to the stdout and stderr
fields of RunSkillScriptTool and RunSkillInlineScriptTool, so a script that
emits a large log cannot put its whole output into the next prompt. The cap
is configurable per toolset via the new SkillToolset maxOutputChars option
and defaults to 30,000 characters per stream.
Add unit coverage for the truncateMiddle helper (boundaries, odd/even
split, removed-character count, zero and negative limits, default cap),
for both skill script tools (stdout, stderr, independent caps, custom
maxOutputChars, no mutation of the executor result), for the new
SkillToolset option, and an end-to-end case that runs a real noisy script
through UnsafeLocalCodeExecutor and asserts the head, marker and tail.
docs:check runs typedoc with --treatWarningsAsErrors, and a {@link} to
DEFAULT_MAX_OUTPUT_CHARS warns because truncate_utils stays internal and
is not part of the documented public API. Name the constant and its value
inline instead of linking to it.
Address the complexity review:
- RunSkillInlineScriptTool read the cap as
  'this.toolset?.maxOutputChars ?? DEFAULT_MAX_OUTPUT_CHARS', but toolset is
  non-optional and maxOutputChars is always assigned, so neither guard could
  fire. Collapse it to the sibling tool's spelling.
- truncateMiddle's 'limit' default was taken by no caller, only by a test of
  the default itself. Make the parameter required.
- With no default left in the helper, the 30,000 cap is skill-toolset policy
  rather than a property of string truncation, so move the constant beside
  the option it defaults and leave truncate_utils a generic helper the queued
  follow-ups can reuse.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant