Skip to content

Fix: honour ADK_DISABLE_GEMINI_MODEL_ID_CHECK in BuiltInCodeExecutor and UrlContextTool - #470

Open
AmaadMartin wants to merge 4 commits into
mainfrom
fix/model-id-check-bypass-code-executor-url-context
Open

Fix: honour ADK_DISABLE_GEMINI_MODEL_ID_CHECK in BuiltInCodeExecutor and UrlContextTool#470
AmaadMartin wants to merge 4 commits into
mainfrom
fix/model-id-check-bypass-code-executor-url-context

Conversation

@AmaadMartin

@AmaadMartin AmaadMartin commented Aug 1, 2026

Copy link
Copy Markdown
Owner

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

  1. Link to an existing issue (if applicable):
    Closes: #issue_number
    Related: #issue_number
  2. Or, if no issue exists, describe the change:
    Problem: ADK_DISABLE_GEMINI_MODEL_ID_CHECK is a documented opt-in escape hatch for users whose model ids do not follow the public gemini-* naming convention (custom endpoints, internal aliases). In adk-js it is only half-wired: isGeminiModelIdCheckDisabled() (core/src/utils/model_name.ts) is consulted by vertex_ai_search_tool.ts, enterprise_web_search_tool.ts and google_maps_grounding_tool.ts, but two call sites that gate on the same concept never read it and hard-throw instead:
  • core/src/code_executors/built_in_code_executor.tsGemini code execution tool is not supported for model internal-model-v1
  • core/src/tools/url_context_tool.tsURL context tool is not supported for model internal-model-v1

With the flag set, ENTERPRISE_WEB_SEARCH.processLlmRequest(...) succeeds on the same request while those two throw. adk-python has the bypass in both places (code_executors/built_in_code_executor.py:46-47, tools/url_context_tool.py:51-67).

Solution: read the existing helper at both gates. No new types, exports, config fields, or public API; process.env is never read directly in src/.

// built_in_code_executor.ts
const modelCheckDisabled = isGeminiModelIdCheckDisabled();
if (llmRequest.model && (isGemini2OrAbove(llmRequest.model) || modelCheckDisabled)) {

// url_context_tool.ts
const bypassModelCheck =
  isGeminiModelIdCheckDisabled() && !isGemini1Model(llmRequest.model);
if (!isGeminiModel(llmRequest.model) && !bypassModelCheck) { ... }
if (!isGemini2OrAbove(llmRequest.model) && !bypassModelCheck) { ... }

Design decisions worth reviewing:

  • Both URL-context guards respect the bypass, not just one. A non-Gemini id such as internal-model-v1 fails both predicates, so bypassing only isGeminiModel would leave the second guard throwing and the escape hatch still broken.
  • The bypass excludes Gemini 1.x (parity wins over transliteration). adk-js folds Python's dedicated Gemini-1.x rejection into the !isGemini2OrAbove guard, so a bare || modelCheckDisabled would let gemini-1.5-pro through with the flag set — which adk-python explicitly forbids (url_context_tool.py:54-55 raises before the bypass is considered). google_maps_grounding_tool.ts:29-35 already encodes the same "Gemini 1.x is rejected ahead of the bypass" shape.
  • No Gemini 1.x carve-out in the code executor (deliberate asymmetry). adk-python's executor gate is purely version-or-bypass, so gemini-1.5-flash + the flag appends the tool there; a plain || reproduces that. The asymmetry with the URL context tool is in the reference implementation, not an oversight, and it tracks capability: code execution works on Gemini 1.5, url_context does not exist there at all. Net effect to be aware of when reviewing: with the flag set, BuiltInCodeExecutor admits gemini-1.5-* and UrlContextTool still rejects it.
  • The llmRequest.model && guard is retained (local convention wins). adk-python has no such guard, so model=None plus the flag would append a code-execution tool there. Keeping it preserves the behaviour pinned by the pre-existing ... not supported for model undefined test and avoids attaching a tool to a request that names no model. The flag suppresses a validation; it does not turn a previous early-return into a positive capability claim. Not observable across the language boundary for any request that actually sets a model.
  • Error strings are byte-identical, including the Gemini-1.x message. Python's Url context tool cannot be used in Gemini 1.x. wording was deliberately not adopted — it would break a pre-existing assertion for no benefit.
  • _is_managed_agent is not ported. adk-js has no equivalent of LlmRequest._is_managed_agent (no isManagedAgent anywhere in core/src), so that disjunct of Python's condition is out of scope.

Deliberately out of scope (not silently dropped): core/src/utils/model_name.ts is untouched, so adk-js's isGemini2OrAbove still lacks Python's EAP-model handling; google_search_tool.ts:54 has the same missing bypass and is, after this PR, the last built-in tool whose model-id gate still ignores the flag; vertex_rag_retrieval_tool.ts has no model gate at all. Each is a separate change. Of the three, only the model_name.ts EAP gap has an open PR on this fork (#372); the google_search_tool.ts bypass is queued but unimplemented, and is not folded in here to keep this diff to the two call sites the task names.

Collision check (required before implementation). gh pr list --state open --limit 200 plus a per-PR file scan for built_in_code_executor|url_context_tool|utils/model_name returned three adjacent-but-distinct PRs, none of which implements this bypass:

Not stacked on any of them: neither the source hunks nor the new test cases depend on their changes, and branching from main keeps this reviewable on its own. Textual conflicts with #372/#413 in the test files are possible at merge time and are trivial (adjacent it blocks).

Note on as never (revised after complexity review). The two new url-context cases pass toolContext: {} as never, matching the six pre-existing cases in the same file and the analogous cases in enterprise_web_search_tool_test.ts:126 / google_maps_grounding_tool_test.ts:99. An earlier revision built a real Context -> InvocationContext -> LlmAgent/createSession/PluginManager graph to avoid the cast; that was removed. UrlContextTool.processLlmRequest destructures {llmRequest} only, so the graph populated a parameter the code under test never reads while coupling a model-id gating test to three unrelated constructors. The root cause is upstream — ToolProcessLlmRequest.toolContext (core/src/tools/base_tool.ts:26) is declared required but is unused by every built-in tool's override — and fixing that signature is a separate change, not something to work around with a stub graph in one file. This PR therefore adds no any, @ts-expect-error or eslint-disable, and no new kind of cast.

Testing Plan

Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.

Four new cases; no existing test case was modified, skipped, or deleted.

core/test/code_executors/built_in_code_executor_test.ts

  1. adds the tool for internal-model-v1 when the check is disabled;
  2. still throws ... not supported for model undefined when the model is unset and the check is disabled (pins the retained llmRequest.model && guard).

core/test/tools/url_context_tool_test.ts 3. adds urlContext for internal-model-v1 when the check is disabled (that id fails both predicates, so one case covers both guards); 4. still rejects gemini-1.5-pro with requires Gemini 2 or above when the check is disabled.

npx vitest run --project unit:core \
  core/test/code_executors/built_in_code_executor_test.ts \
  core/test/tools/url_context_tool_test.ts
# Test Files 2 passed (2) | Tests 15 passed (15)

One test-fixture change beyond the four new cases, called out deliberately. Both suites now pin the flag off with vi.stubEnv(MODEL_ID_CHECK_ENV_VAR, undefined) in beforeEach (vi.unstubAllEnvs() in afterEach), matching existing usage in core/test/telemetry/setup_test.ts and dev/test/server/adk_api_server_test.ts. The save-and-restore idiom used by the sibling suites restores the ambient value, which is not sufficient here: because the source now honours the flag, a developer with ADK_DISABLE_GEMINI_MODEL_ID_CHECK=true exported in their shell would see two pre-existing cases fail (throw error if model is invalid, throws for unsupported (non-Gemini) model) — those assertions are only valid with the flag off. Measured before the fixture change:

ADK_DISABLE_GEMINI_MODEL_ID_CHECK=true npx vitest run --project unit:core <both files>
# Tests  2 failed | 13 passed (15)

and after it, green in both directions:

npx vitest run ...                                  # 15 passed
ADK_DISABLE_GEMINI_MODEL_ID_CHECK=true npx vitest run ...  # 15 passed

No existing assertion was weakened: the two cases above still assert the same throw, they are now simply hermetic with respect to the environment variable this PR makes them sensitive to.

Proving the tests can fail. Every new test was run against mutated source and observed to FAIL:

Mutation Failing test Message
drop || modelCheckDisabled (executor) case 1 expected [Function] to not throw an error but 'Error: Gemini code execution tool is …' was thrown
drop llmRequest.model && (executor) case 2 expected [Function] to throw an error
drop && !bypassModelCheck from both url-context guards case 3 URL context tool is not supported for model internal-model-v1
drop && !bypassModelCheck from the second guard only case 3 URL context tool requires Gemini 2 or above, but got internal-model-v1
drop && !isGemini1Model(...) from bypassModelCheck case 4 promise resolved "undefined" instead of rejecting

The 3rd and 4th rows are why case 3 alone is sufficient to cover both guards. Note case 2 does carry mutation signal: with the flag off the pre-existing model is not provided case passes either way (because isGemini2OrAbove(undefined) is false), so case 2 is the only test that pins the model guard once the bypass is on.

Coverage of the two changed source files (v8, both files scoped):

File                     | % Stmts | % Branch | % Funcs | % Lines | Uncovered
built_in_code_executor.ts|      80 |      100 |      75 |      80 | 29-37
url_context_tool.ts      |   94.44 |      100 |   66.66 |   94.44 | 32-33

100% branch coverage; every line added by this PR is covered. The uncovered lines are pre-existing code this PR does not touch: the isBuiltInCodeExecutor type guard (29-37) and UrlContextTool.runAsync (32-33).

Static checks (on the pushed commit):

npx eslint <4 changed files>      # exit 0
npx prettier --check <4 changed files>  # All matched files use Prettier code style!
npm run ts:check                  # 281 errors, identical to the pre-change baseline
                                  # (measured via git stash); none in the 4 changed files
npm run build                     # success

npm run ts:check is red on main for 41 unrelated files (281 errors); this PR neither adds to nor fixes that count, verified by running it on a stashed tree.

Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.

No integration fixture is warranted: this is a conditional change in existing code paths with no I/O, no network and no cross-component wiring, and the unit tests drive the real BuiltInCodeExecutor / UrlContextTool objects (no mocks, no stubbed model-name helpers) through the public processLlmRequest API. To reproduce the original bug and confirm the fix by hand:

npm ci && npm run build

# 1. Reproduce on main: both calls throw for a custom model id even with the flag set.
git checkout main -- core/src/code_executors/built_in_code_executor.ts \
                     core/src/tools/url_context_tool.ts && npm run build
ADK_DISABLE_GEMINI_MODEL_ID_CHECK=true node -e "
const {BuiltInCodeExecutor, URL_CONTEXT} = require('./core/dist/cjs/index.js');
const req = () => ({model: 'internal-model-v1', contents: [], toolsDict: {}, liveConnectConfig: {}});
try { new BuiltInCodeExecutor().processLlmRequest(req()); } catch (e) { console.error('executor:', e.message); }
URL_CONTEXT.processLlmRequest({llmRequest: req(), toolContext: {}}).catch(e => console.error('url_context:', e.message));
"
# executor: Gemini code execution tool is not supported for model internal-model-v1
# url_context: URL context tool is not supported for model internal-model-v1

# 2. Restore the fix (git checkout HEAD -- <the two files> && npm run build): the same
#    script prints
#      executor tools: [{"codeExecution":{}}]
#      url_context tools: [{"urlContext":{}}]
#    and gemini-1.5-pro is still rejected with "requires Gemini 2 or above".

# 3. The suites are green with the flag set in the ambient environment:
ADK_DISABLE_GEMINI_MODEL_ID_CHECK=true npx vitest run --project unit:core \
  core/test/code_executors/built_in_code_executor_test.ts \
  core/test/tools/url_context_tool_test.ts

Checklist

[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.

CI

All checks green on run-tests for ubuntu-latest, macos-latest and windows-latest (commit 2ac42f8c).

macos-latest failed twice before passing, both times with the same unrelated error: Test timed out in 40000ms in tests/integration/app_loader/app_loader_test.ts:82 (windows-latest was then cancelled by matrix fail-fast, not failed; ubuntu-latest passed every time). Logged rather than papered over, with the diagnosis:

  • That suite references nothing this PR touches — no BuiltInCodeExecutor, UrlContextTool, URL_CONTEXT, urlContext or codeExecution anywhere under tests/integration/app_loader/.
  • It shells out to npm install per fixture at test time, so its runtime is registry/network-bound rather than code-bound; a cold macOS runner exceeding the 40s hook budget is the expected failure mode. Running it locally fails at the same npm install step for an unrelated environment reason, confirming the install is what the test spends its time on.
  • The identical commit passes on re-run.

Not fixed here, and deliberately not queued as new work: #405 (align integration install hooks on the project-wide hook timeout) and #407 (pin the integration-fixture install mode) already modify this exact file for this exact problem.

Amaad Martin added 4 commits August 1, 2026 10:36
The model-id escape hatch is honoured by vertex_ai_search_tool,
enterprise_web_search_tool and google_maps_grounding_tool, but the built-in
code executor still hard-throws for any non-Gemini-2+ model id. Consult
isGeminiModelIdCheckDisabled() in the gate so a user who opts out of the
naming convention gets a consistent bypass, matching adk-python's
built_in_code_executor.process_llm_request.

The llmRequest.model guard is retained: a request that names no model still
throws rather than attaching a code-execution tool.
Both model guards now respect the escape hatch: a non-Gemini id such as
internal-model-v1 fails isGeminiModel and isGemini2OrAbove, so bypassing only
the first would leave the second throwing and the flag still broken.

The bypass excludes Gemini 1.x, which does not support url_context at all.
adk-js folds Python's dedicated Gemini-1.x rejection into the !isGemini2OrAbove
guard, so a bare disjunction would let gemini-1.5-pro through with the flag set
- something adk-python's url_context_tool rejects before the bypass is even
considered. google_maps_grounding_tool encodes the same shape.
Save-and-restore of the original value restores the ambient value, so a
developer with ADK_DISABLE_GEMINI_MODEL_ID_CHECK exported in their shell would
see the two pre-existing 'throws for an unsupported model' cases fail - the
source now honours the flag, so those assertions are only valid with it off.

Stub the variable to undefined before each case (vi.unstubAllEnvs restores the
ambient value at teardown) so both suites are green whether or not the flag is
set in the environment.
UrlContextTool.processLlmRequest destructures {llmRequest} only, so the
Context -> InvocationContext -> LlmAgent/createSession/PluginManager graph
populated a parameter the code under test never reads, while coupling a
model-id gating test to three unrelated constructors.

Pass toolContext: {} as never like the six pre-existing cases in this file and
the analogous cases in the enterprise_web_search and google_maps_grounding
suites. The cast is inert here and its removal is being handled file-wide as a
separate cleanup.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant