Test: fail CI when a vitest project is not run by any npm script - #343
Open
AmaadMartin wants to merge 2 commits into
Open
Test: fail CI when a vitest project is not run by any npm script#343AmaadMartin wants to merge 2 commits into
AmaadMartin wants to merge 2 commits into
Conversation
added 2 commits
July 30, 2026 15:21
The root npm scripts pick vitest projects with hand-written --project flags, so a project declared in vitest.config.ts runs only once some script names it. Nothing kept the two lists in sync, and the drift was silent: unit:integrations was declared but selected by no script, so its tests never ran and its version assertion rotted unnoticed. Add a meta test under the existing integration project (already selected by test:coverage, which validation.yaml runs on all three OSes) asserting set equality in both directions, so an orphaned project and a stale --project flag each fail the build naming the offender.
Addresses simplicity-audit feedback on the first commit: read the root manifest with a JSON import instead of readFileSync + JSON.parse + a type assertion (matching integrations/test/version_test.ts), drop two single-use constants, and replace the hand-rolled flag tokenizer with node:util parseArgs. parseArgs also types a valueless `--project` as boolean rather than silently yielding undefined from an out-of-range index, so that case is now rejected explicitly instead of leaking a non-string into the comparison.
This was referenced Jul 30, 2026
Open
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
Closes: #issue_number
Related: #issue_number
Problem: The root
package.jsontest scripts select vitest projects with hand-written--projectflags, and nothing keeps that hand-written list in sync with the projects declared invitest.config.ts. When they drift, the failure is silent — the affected tests simply stop running and nobody is told.This is not hypothetical.
unit:integrationswas declared invitest.config.tsand named by no script, sointegrations/test/**never ran in CI. Its only test assertedexpect(version).toBe('1.3.0')whileintegrations/src/version.tshad been bumped to1.4.0by release-please. That test was red and invisible..github/workflows/validation.yamlrunsnpm run test:coverage, which could not see it.The rename half of the same drift is equally silent: renaming a project in
vitest.config.tswithout updating the scripts leaves a--projectflag that matches nothing, and vitest is happy to run zero files.Solution: One meta test,
tests/integration/repo_config/vitest_projects_test.ts, asserting set equality between the two lists in both directions:vitest.config.tsis named by some root npm script;--projectflag in a root npm script names a declared project.Each assertion computes the offending names and compares against
[], so the failure diff prints the drifted name (expected [ 'unit:integrations' ] to deeply equal []) alongside a message stating the remedy. That failure message is the point of the guard, so the two directions are deliberately not collapsed into one boolean assertion.Design notes:
tests/integration/**, which is theintegrationproject — already selected bytest,test:integrationandtest:coverage, andtest:coverageis whatvalidation.yamlruns onubuntu-latest,windows-latestandmacos-latest. Ascripts/check_vitest_projects.mjs+ workflow step was the larger option, not the smaller one: plainnodecannot importvitest.config.ts, so it would have to regex-parse TypeScript.vitest.config.tsdirectly, so it sees the realtest.projectsarray rather than a regex over source. (vitest.config.tsuses__dirname, which does not exist in native ESM; vitest's module runner injects it into transformed modules. This was verified empirically before the assertions were written.)test.namefrom raises an error naming the index, because a shape it silently skipped would be exactly the drift it exists to catch. Same for a config with notest.projectsand for a script that passes--projectwith no value.cross-languagebytest:cross-language, whichcross-language-integration.ymlruns), so an allowlist would be dead code today and a place to hide drift tomorrow. If a future project genuinely cannot be run by a script, the fix is a one-line script.node:utilparseArgs, not substring matching —String.includes('unit:core')would false-positive on a hypotheticalunit:core:slow. Both--project nameand--project=nameare read.strict: falseis required because the non-test scripts carry flags this guard does not model. NoteparseArgstypes a valueless--projectasboolean; that case is rejected explicitly rather than cast away.Known limitations (deliberate scope boundaries, not oversights):
--project unit:integrationsfrom onlytest:coveragewhiletest/test:unitkeep it would still pass. Asserting that workflows invoke each test script is a strictly larger change; queued as a separate follow-up.package.json; per-workspace scripts (e.g.integrations/package.json's"test": "vitest") are out of scope.--project "unit:*". This guard models--projectvalues literally, so the two are mutually exclusive as written: whichever merges second needs reconciling. They are alternative solutions to the same root problem, so reviewers should pick one rather than merge both. Glob support here would be a few lines, but was left out as speculative while Fix: select vitest unit projects by wildcard so new unit suites cannot be silently excluded #311 is unmerged.Collision check (before implementation):
gh pr list --repo AmaadMartin/adk-js --state open --limit 100plus agh pr diff --name-onlysweep of every open PR forvitest.config.ts/ rootpackage.json/repo_confighits. Found #236 and #245 as near-duplicate implementations of the prerequisite edits (script flags + version test) and #311/#237/#261 as adjacent. No open PR ships the guard. Since #236 is the older of the duplicate pair and is already the base of #311, this PR stacks on it rather than re-landing those edits.Note on
integrations/test/version_test.ts(in the parent PR, #236, not this diff): it rewrites an existing test rather than adding one, which the contributor guidelines normally forbid. The exception applies here — the old assertion pinned a stale'1.3.0'literal that was already wrong and could never fail because the project was never run. Bumping the literal to'1.4.0'would just re-arm the same trap at the next release, so it now reads the expected value fromintegrations/package.json, which release-please bumps in the same commit assrc/version.ts.Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.
Every test was proven able to fail. Each was run against mutated code and confirmed to FAIL, so none of them is a green light with no signal:
package.jsonto its state onmain, where no script namesunit:integrations— the exact bugexpected [ 'unit:integrations' ] to deeply equal []e2e→e2e2invitest.config.tsexpected [ 'e2e2' ] to deeply equal [](unreferenced) andexpected [ 'e2e' ] to deeply equal [](dangling flag)test.projectsguardexpected [Function] to throw error matching /declares no test.projects/ but got 'Cannot read properties of undefined…'expected [Function] to throw an error--project=namespellingexpected [ 'unit:core' ] to deeply equal [ 'unit:core', 'e2e' ]--projectinstead of throwingexpected [Function] to throw an errorMutation 1 is worth calling out: removing the flag from only
test:coveragedoes not trip the guard, becausetestandtest:unitstill name the project. The honest reproduction ismain's actual state, where no script names it — that is what was run.Coverage. This change ships no production code, and
vitest.config.ts's coverageincludeis limited tocore/src,dev/src,integrations/src, none of which are touched — so new-line coverage of shipped source is vacuous. The new code is the test, and all six of its cases are exercised and individually proven falsifiable above. Runningunit:integrationscan only raise the measured floor (integrations/src/version.tsbecomes covered), so the existing thresholds (86/87/88/86) need no edit.CI status: absent. This is a stacked PR whose base is
fix/run-unit-integrations-vitest-project, notmain;validation.yamltriggers onpull_request: branches: [main], so no test job runs for this PR. Validated locally on the exact pushed commit instead:Honest reporting on two repo-wide commands:
npm run ts:checkis already red onmain(308 errors, the subject of a separate open PR). Verified this change adds none: the count is 308 both with and without the new file, and no error line referencesrepo_config.npm run test:coveragedoes not pass in this sandbox, with 26 failing tests across 24 files. All are pre-existing environment failures — fixturenpm installtimeouts, ambient-env and shell-dependent cases, several with their own open PRs. Verified pre-existing rather than assumed: the new file passed inside that run (✓ |integration| tests/integration/repo_config/vitest_projects_test.ts), and re-running the failing integration files with the new file physically removed from the tree reproduced the failures unchanged.Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.
Checklist
[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.