Test: pin the CFC model gate's rejection of unversioned Gemini Early Access ids - #910
Open
AmaadMartin wants to merge 1 commit into
Open
Test: pin the CFC model gate's rejection of unversioned Gemini Early Access ids#910AmaadMartin wants to merge 1 commit into
AmaadMartin wants to merge 1 commit into
Conversation
AmaadMartin
force-pushed
the
feat/pin-cfc-model-gate-eap-rejection
branch
from
August 11, 2026 05:25
86ef9f6 to
eeb74f6
Compare
…i Early Access ids The gate in Runner.runAsync admits only a Gemini id with a parseable major version of 2 or above. Unversioned Early Access ids such as gemini-flash-early-exp fall outside that rule, and no test held them there. The new suite pins the six gate outcomes. Nothing changes at runtime.
AmaadMartin
force-pushed
the
feat/pin-cfc-model-gate-eap-rejection
branch
from
August 11, 2026 05:46
eeb74f6 to
4c93f6b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
Link to an existing issue (if applicable):
N/A
Or, if no issue exists, describe the change:
Problem: The Compositional Function Calling (CFC) gate in
Runner.runAsyncrejects an unversioned Gemini Early Access id such asgemini-flash-early-exp, and no test pins that. A contributor could widenisGemini2OrAbove, or swap the gate forisGeminiModel, and admit an unvalidated model family without a single test turning red.Solution: A new suite pins the six outcomes of the gate. The diff is test-only, so there is no behaviour change and no source file is touched.
Cross-language note:
adk-pythonrejects the same two ids at its own gate (Runner._new_invocation_context), so the pinned cases agree in both runtimes. Thegemini-3.xand Vertex publisher-path cases diverge between the runtimes; they are out of scope here and are tracked as a separate task.Collision check:
gh pr list --repo AmaadMartin/adk-js --state openplus searches forcfc,model gateandgemini early access. Open PR #716 adds the same test file but asserts nothing about Early Access ids. This PR supersedes #716, so #716 should be closed. I did not stack on it, because the stacked PR would have to revert part of its base: PR #716 assertsgemini-3.0-proand the Vertex publisher path, which are the out-of-scope cases named above. PRs #372, #615 and #618 touchcore/src/utils/model_name.ts; this change does not.Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.
The 292
tscerrors are a pre-existing baseline on this branch point. I measured the tree with and without the new file and the count and the file list are the same.Note on the base. The first CI run failed on
ubuntu-latestwith seventests/integration/workflowssample failures. Those are unrelated to this change: the fork'smaincarried the identity-preamble prompt change without the follow-up that repairs the recorded workflow samples, so every pull request on that base failed the same way. I fast-forwarded the fork'smainto the upstream commit that carries the repair and rebased onto it.Coverage: the suite executes every statement and every branch of the gate block in
core/src/runner/runner.ts. The change adds no source line, so there is no new source code to cover.Proof that each test can fail. I mutated the gate once per test, re-ran the suite, then restored the gate.
core/src/runner/runner.tsisGeminiModelinstead ofisGemini2OrAbovethrowBuiltInCodeExecutorisBuiltInCodeExecutorguardrunConfig.supportCfc &&conditionsupportCfc: falsecaseThe first mutation is the one that matters, and its output is the argument for the decision:
Widening the gate does not make an Early Access id work. It moves the failure later, into
BuiltInCodeExecutor.processLlmRequest, which gates onisGemini2OrAboveitself, and the message gets worse.Review note. An earlier revision also added a three-line comment above the gate, claiming the narrow rule keeps the run off the unimplemented CFC path. That claim is wrong: a versioned id reaches the same
CFC is not yet supported in callLlmAsyncthrow, as the accept-case test asserts. I removed the comment, so the diff is now test-only.Manual End-to-End (E2E) Tests:
No E2E test applies. A CFC run cannot complete in
adk-js, becauseLlmAgent.callLlmAsyncrefuses the path unconditionally. The vitest command above is the acceptance check. To reproduce by hand:Checklist
[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.