Fix: await the conformance run and exit non-zero when a conformance test fails - #839
Open
AmaadMartin wants to merge 3 commits into
Open
Fix: await the conformance run and exit non-zero when a conformance test fails#839AmaadMartin wants to merge 3 commits into
AmaadMartin wants to merge 3 commits into
Conversation
added 3 commits
August 8, 2026 19:25
The CLI needs one fact from a conformance run: did anything fail? The function accumulated the counts and threw them away, so no caller could tell a clean run from a failed one.
The action dropped the runIntegrationTests promise, so the command exited 0 after a run in which every test failed, and a thrown error surfaced as an unhandled rejection. It now awaits the run, reports an error through the CLI logger, and sets process.exitCode.
Sibling suites in this directory are named <module>_test.ts.
7 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
N/A
Problem:
adk integration conformancealways exits 0. The action callsrunIntegrationTestswithoutawait, andrunIntegrationTestsreturns nothing, so no CI job can gate on the suite. A run that throws also escapes as an unhandled rejection instead of thelogger.errorline every other command incli.tsprints.Solution:
runIntegrationTestsnow returns the number of failed tests, and the action awaits it. The action setsprocess.exitCode = 1when a test failed or the run threw. I chose a bare count over a{passed, skipped, failed}object because the CLI reads exactly one fact; the other two fields would have no reader. I choseprocess.exitCodeoverprocess.exit(1)because the run writes its whole summary immediately before, andprocess.exit()discards stdout writes still pending on a pipe — the CI case this fix exists to serve.Behaviour change: a caller that relied on this command exiting 0 after failures was relying on the bug.
Collision check:
gh pr list --repo AmaadMartin/adk-js --state open --limit 200, thengh pr diff --name-onlyon every adjacent PR. Three PRs touch the same files, none implements this fix:run_integration_tests.tsand addsdev/test/integration/run_integration_tests_test.ts. My two source edits sit outside the block Fix: report the reason a conformance test failed #745 rewrites. My new tests use that same conventional filename, so whichever PR lands second should merge its cases into the one file.cli.tsandcli_test.tsfor logging precedence and option arity.I branched from
mainrather than stacking, because the three overlaps are mutually exclusive bases and a stacked base gets no CI here.Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.
npm run ts:checkreports 287 pre-existing errors, all undercore/test/**andtests/integration/**. None is indev/, and my change adds none. Several open PRs (#736-#743) are fixing that backlog.Proving each test can fail. I ran every new test against a mutated source and recorded the failure:
awaitin the actionexpected true to be falseawaitin the actionexpected +0 to be 1awaitin the actionexpected "error" to be called with argumentsif (failedCount > 0) {...}expected +0 to be 1failedCount > 0->failedCount >= 0expected 1 to be +0catchblockpromise rejected "Error: boom" instead of resolvingagents_dirastestsDir,forceRunAll: falseexpected "spy" to be called with argumentsreturn failedTests.length->return 0expected +0 to be 2;expected +0 to be 1return failedTests.length->+ 1expected 1 to be +0Integration tests: none. The command's only integration surface is a corpus of conformance YAML, and this repository ships none. A fixture tree is out of proportion to a five-line exit-code fix.
Manual End-to-End (E2E) Tests:
I built the CLI and ran it against synthetic corpora outside the repository.
<agents>holds oneroot_agent.yaml.Checklist
[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.