Skip to content

Test: pin the CFC model gate semantics in the runner (adk-python parity) - #716

Open
AmaadMartin wants to merge 2 commits into
mainfrom
fix/cfc-model-gate-semantics
Open

Test: pin the CFC model gate semantics in the runner (adk-python parity)#716
AmaadMartin wants to merge 2 commits into
mainfrom
fix/cfc-model-gate-semantics

Conversation

@AmaadMartin

@AmaadMartin AmaadMartin commented Aug 6, 2026

Copy link
Copy Markdown
Owner

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

  1. Link to an existing issue (if applicable):
    N/A

  2. Or, if no issue exists, describe the change:

Problem: The supportCfc model gate disagrees across the two SDKs, and neither SDK tests it. adk-js runner.ts calls isGemini2OrAbove, so it accepts gemini-3.0-pro and a Vertex publisher path. adk-python runners.py calls startswith('gemini-2') on the raw id, so it rejects both. An untested gate can be "fixed" on the wrong side.

Solution: Pin the adk-js semantics with tests. CFC is supported if, and only if, the canonical model id names a Gemini model of major version 2 or above, after the Vertex publisher path is stripped. adk-python is the side that needs correcting: its gate must admit what its own BuiltInCodeExecutor admits, and that executor uses is_gemini_eap_or_2_or_above. adk-js behaviour is unchanged; this is one new test file plus a one-line comment.

Scope notes:

Testing Plan

Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.

New file core/test/runner/runner_cfc_test.ts, 4 cases. Each drives a real Runner over an InMemorySessionService with runConfig: {supportCfc: true} and asserts on the gate's own observable, agent.codeExecutor.

case model id result
accept gemini-2.5-flash executor installed
accept gemini-3.0-pro executor installed
accept projects/.../publishers/google/models/gemini-2.5-flash executor installed
reject claude-3-5-sonnet throws, no executor

The accept cases stop downstream at CFC is not yet supported in callLlmAsync, because the CFC call path is not implemented (llm_agent.ts). The tests assert that stop; they do not assert a successful run.

Proof the tests can fail. The only source change is a comment, so mutation testing is the only evidence these tests carry signal. I ran both mutations against the new file.

Mutation 1, the adk-python literal. Replace !isGemini2OrAbove(modelName) with !modelName.startsWith('gemini-2'). 2 of 4 fail:

× accepts the next major and installs a code executor
  → expected 'CFC is not supported for model: gemini-3.0-pro in agent: cfc_agent' to be undefined
× accepts the Vertex publisher path form and installs a code executor
  → expected 'CFC is not supported for model: projects/test-project/locations/
     us-central1/publishers/google/models/gemini-2.5-flash in agent: cfc_agent'
     to be undefined

Mutation 2, remove the gate. Replace the condition with false. 1 of 4 fails:

× rejects a non-Gemini model and installs no code executor
  → expected 'Gemini code execution tool is not supported for model claude-3-5-sonnet'
     to be 'CFC is not supported for model: claude-3-5-sonnet in agent: cfc_agent'

Mutation 2 also shows why the gate should track the executor: with the gate gone, BuiltInCodeExecutor rejects the same model one layer later.

Both mutations were reverted. Coverage: the source change is a comment, so it adds no executable line. The tests cover both branches of the gate.

Manual End-to-End (E2E) Tests:
None. adk-js does not implement the CFC call path, so there is no end-to-end CFC run to exercise. The mutation runs above are the substitute.

Local validation on the pushed commit:

npx vitest run --project unit:core core/test/runner/runner_cfc_test.ts   # 4 passed
npx vitest run --project unit:core core/test/runner/                     # 56 passed
npx eslint core/src/runner/runner.ts core/test/runner/runner_cfc_test.ts # clean
npx prettier --check core/src/runner/runner.ts core/test/runner/runner_cfc_test.ts  # clean
npm run build                                                            # ok

npm run ts:check reports pre-existing errors in the core test tree (open PR #248 fixes them). It reports none for the two files in this PR.

Checklist

[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.

Amaad Martin added 2 commits August 6, 2026 04:13
The runner gates supportCfc on isGemini2OrAbove, so it accepts Gemini major
2 and above and unwraps a Vertex publisher path. adk-python uses a literal
startsWith('gemini-2') on the raw id, so it rejects gemini-3.0-pro and the
path form. Neither runtime has a test on this gate.

Add core/test/runner/runner_cfc_test.ts to pin the adk-js semantics, plus a
comment above the gate explaining why it uses the shared predicate. No
production behaviour changes.
The removed lines restated isGemini2OrAbove's own contract, claimed a
provider-wrapper strip that extractModelName does not do, and described a
startsWith('gemini-2') implementation this file never had.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant