Skip to content

Test: fail CI when a vitest project is run by no root package.json script - #543

Open
AmaadMartin wants to merge 4 commits into
mainfrom
feat/vitest-projects-script-guard
Open

Test: fail CI when a vitest project is run by no root package.json script#543
AmaadMartin wants to merge 4 commits into
mainfrom
feat/vitest-projects-script-guard

Conversation

@AmaadMartin

@AmaadMartin AmaadMartin commented Aug 2, 2026

Copy link
Copy Markdown
Owner

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

  1. Link to an existing issue (if applicable):
    No existing issue is linked — the defect was found by inspection of the repo's own test wiring.

  2. Or, if no issue exists, describe the change:

Problem: vitest.config.ts declares a list of test projects, but nothing runs a project unless a root package.json script names it with --project <name>. That coupling is invisible: adding a project to vitest.config.ts looks like it wires tests into CI, but the tests only run if someone also edits a script.

The repo already contains exactly that defect. vitest.config.ts declares six projects; unit:integrations is referenced by no root script:

project referenced by a root script (before this PR)?
unit:core yes (test, test:unit, test:coverage)
unit:dev yes (test, test:unit, test:coverage)
unit:integrations no — referenced nowhere
integration yes (test, test:integration, test:coverage)
e2e yes (test, test:e2e, test:coverage)
cross-language yes (test:cross-language)

unit:integrations arrived with the integrations package and was never added to a script, so integrations/test/**/*_test.ts has never executed — not locally via npm test, and not in CI (.github/workflows/validation.yaml runs npm run test:coverage). The consequence was already live: integrations/test/version_test.ts asserted expect(version).toBe('1.3.0') while integrations/src/version.ts exports '1.5.0'. That test has been failing-if-run since the version bumps and nobody noticed, which is precisely the failure mode a guard should prevent. Running it confirms this rather than assuming it:

× version > should match the version declared in package.json
  → expected '1.5.0' to be '1.3.0' // Object.is equality

Solution: three commits, one per concern, plus a fourth that simplifies how the two new tests read package.json (see Review revisions at the end).

  1. integrations/test/version_test.ts — de-stale the assertion (41533b6d). This modifies an existing test rather than adding one beside it, because the existing assertion encodes wrong behaviour: the literal is stale, not merely inconvenient. It is done in its own commit, as the repo guideline on editing existing tests requires. The literal is replaced with the invariant the test is really protecting — that integrations/src/version.ts and integrations/package.json stay in sync. Rationale for deriving instead of bumping '1.3.0' to '1.5.0': release-please bumps integrations/package.json and (via its extra-files entry) integrations/src/version.ts, but never the test, so any literal goes stale again at the next release and turns the automated release PR red. The derived form cannot drift, and it upgrades a tautology into a real check that the two release-please targets stay in sync. The assertion that still pins the old behaviour is the same one, expect(version).toBe(...) — only the expected value's source changed, from a hardcoded literal to the package manifest.

  2. package.json — wire unit:integrations in (261c18b8). Added to the three scripts that enumerate unit projects (test, test:unit, test:coverage). test:integration, test:e2e and test:cross-language are left untouched; they are deliberately single-project entry points.

  3. tests/integration/repo_config/vitest_projects_test.ts — the guard (85b3528f). It reads the project names out of vitest.config.ts and the scripts out of the root package.json, and asserts every declared project is selected by at least one script. Notable design points:

    • Two assertions, not one. The "every project is run" assertion would pass vacuously on an empty project list, so a separate assertion fails when the list is empty. They are deliberately not merged.
    • Whole-token matching, not substring matching, so --project unit:core-extra cannot "satisfy" unit:core. Both --project <name> and --project=<name> spellings are accepted, because vitest accepts both.
    • An unreadable project entry throws, naming its index, rather than being skipped. A silently skipped project is exactly the drift the guard exists to catch, so a future maintainer who adopts vitest's {label, color} name form is told to extend the guard instead of quietly losing coverage of it.
    • All offenders are reported at once (expect(unreferenced).toEqual([])) rather than aborting on the first.
    • No config change was needed to pick the file up: the integration project already includes tests/integration/**/*_test.ts, and CI runs it via test:coverage --project integration.

Scope notes:

  • The other half of the chain (script → workflow) — asserting that a root test script is actually invoked by a workflow — is deliberately not in this PR; it is a separate change. Nothing under .github/workflows/** is touched here.
  • vitest.config.ts is not modified: the guard reads it, it does not reshape it. Coverage thresholds are untouched — integrations/src/** was already in coverage.include and effectively uncovered, so running unit:integrations can only raise the measured numbers against thresholds that are minima.
  • No new dependencies; package-lock.json is untouched. node:fs, node:path, node:url and vitest only.
  • No any, no @ts-expect-error/@ts-ignore, no eslint-disable, no coverage suppressions anywhere in the diff.

Collision check (performed before implementation). gh pr list --repo AmaadMartin/adk-js --state open --limit 1000 was swept for adjacent work. This change overlaps two open, unmerged PRs on the same fork: #343 (fix/vitest-project-drift-guard) proposes the same guard file, and #236 (fix/run-unit-integrations-vitest-project, which #343 is stacked on) proposes the same package.json wiring and the same version_test.ts fix. Both are still open and unmerged, so the defect is live on main. Two further PRs are adjacent and should be reconciled by whoever lands this: #311 proposes an alternative fix for the same root cause (wildcard project selection instead of enumerated --project flags), which would make part of the script wiring here moot; #479 proposes a different assertion for the same version test. #418 implements the script → workflow half and is complementary, not conflicting. This PR is branched from main and is self-contained, so it lands regardless of which of the overlapping PRs is chosen; if #236 or #343 lands first, the corresponding commit here becomes a no-op and should be dropped rather than merged twice.

Testing Plan

Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.

Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.

The deliverable is itself test code, so the verification below is what proves it works. coverage.include covers only core/src, dev/src and integrations/src, so neither touched test file is instrumented; the guard's own error paths are therefore covered by direct tests of its helpers rather than by a coverage number.

Targeted runs (all green, on the exact pushed commit):

$ npx vitest run --project integration tests/integration/repo_config/vitest_projects_test.ts
 ✓ |integration| tests/integration/repo_config/vitest_projects_test.ts (13 tests) 11ms
 Test Files  1 passed (1)
      Tests  13 passed (13)

$ npx vitest run --project unit:integrations
 ✓ |unit:integrations| integrations/test/version_test.ts (1 test) 6ms
 Test Files  1 passed (1)
      Tests  1 passed (1)

Quality gates:

$ npx prettier --check tests/integration/repo_config/vitest_projects_test.ts integrations/test/version_test.ts package.json
All matched files use Prettier code style!

$ npx eslint tests/integration/repo_config/vitest_projects_test.ts integrations/test/version_test.ts
(clean, exit 0)

$ bash scripts/check_license.sh
✅ All files have the correct license header.

$ npm run build
(exit 0)

$ npm run ts:check
281 pre-existing errors, 0 in the touched files.

Disclosure on ts:check: this clone reports 281 error TS lines, all in pre-existing core/test/** and tests/integration/** files ([BASE_AGENT_SIGNATURE_SYMBOL] mismatches from @google/adk resolving to built dist/types). Stashing this branch's changes and re-running tsc --noEmit yields the same 281 — this change adds none, and grep for the touched paths in the output returns nothing.

Proof each test can fail. Every new assertion was run against mutated code and observed to fail:

# Mutation Result
1 Removed --project unit:integrations from all three root scripts (i.e. reverted commit 2) vitest projects > are each run by at least one root package.json script FAILED: expected [ 'unit:integrations' ] to deeply equal []
2 Added a seventh project named bogus to vitest.config.ts referenced by no script Same test FAILED: expected [ 'bogus' ] to deeply equal []
3 Replaced the first assertion's input with an empty project list vitest projects > are declared in the root config FAILED: expected [] to not have a length of +0
4 Changed integrations/src/version.ts to export '1.4.0', drifting it from package.json version > should match the version declared in package.json FAILED: expected '1.4.0' to be '1.5.0'
5 Made runsProject use substring matching instead of whole-token matching runsProject > does not match a name that merely extends the requested one FAILED: expected true to be false
6 Made declaredProjectNames invent a placeholder name instead of throwing All four declaredProjectNames > rejects … tests FAILED: expected [Function] to throw an error

Every file was restored after each probe and git status was confirmed clean apart from the three intended files.

One honest caveat on mutation 3. Setting projects: [] directly in vitest.config.ts does not reach the guard at all: vitest itself refuses to start with Error: No projects were found … The projects definition: []. That is still a hard, non-silent CI failure, but it means the empty-list assertion is proven by mutating its input (row 3 above) plus the direct declaredProjectNames([]) and declaredProjectNames(undefined) tests, not by emptying the real config.

Manual End-to-End (E2E) Tests:

To see the guard do its job, from a clean checkout of this branch:

  1. npm install && npm run build
  2. Open vitest.config.ts and add a project the scripts do not name, e.g. a seventh entry {test: {name: 'bogus', environment: 'node', include: ['tests/bogus/**/*_test.ts']}}.
  3. npx vitest run --project integration tests/integration/repo_config/vitest_projects_test.ts — it fails and prints bogus in the diff, with the message pointing at the fix.
  4. Add --project bogus to any root script and re-run — it passes.
  5. Revert both edits.

To confirm the previously-orphaned project now runs through the normal entry point: npm run test:unit includes |unit:integrations| in its output, which it did not before this PR.

Checklist

[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.

Review revisions

Round 2 (complexity): both new tests hand-rolled fileURLToPath + path.resolve + readFileSync + JSON.parse to load a package.json that vitest and tsc already load natively. Replaced with a JSON import attribute, which is the established idiom here — 23 existing call sites, e.g. tests/integration/skills/loader/agent_test.ts:14 — and needs no config change under nodenext resolution (4f4194fb, −26 lines):

import packageJson from '../package.json' with {type: 'json'};          // version_test.ts
import rootPackage from '../../../package.json' with {type: 'json'};    // the guard

That removed three node: imports from each file, both path constants, and the single-caller readRootScripts wrapper. It also deleted the two hand-written annotations that existed only to re-type JSON.parse's any return ({version: string} and {scripts?: Record<string, string>}) — the import infers real types instead, which in turn made the scripts ?? {} fallback provably dead.

Because the read mechanism changed, every probe in the table above was re-run against the revised code and still fails as recorded. Probe 4 was strengthened while re-running it: rather than restoring the old '1.3.0' literal (an assertion that no longer exists), it now mutates integrations/src/version.ts to export '1.4.0', which directly exercises the invariant the rewritten test protects — that version.ts and package.json stay in sync. All 13 + 1 tests pass on the revised commit, and npm run ts:check still adds zero errors (281 pre-existing, none in the touched files).

CI note on the revision commit. run-tests (windows-latest) failed once on 4f4194fb with core/test/code_executors/unsafe_local_code_executor_test.ts > UnsafeLocalCodeExecutor > should execute shell code and return stdout — Test timed out in 5000ms. That test spawns a real shell (echo "Hello, Shell!") against vitest's default 5s budget and is in unit:core, which this PR does not touch; the same test passed on windows-latest on the previous commit of this branch, whose only delta is the JSON import in two test files. Re-running the job passed in 9m14s. All four test jobs (run-tests, ubuntu-latest, macos-latest, windows-latest) plus check-license are green on 4f4194fb.

Amaad Martin added 4 commits August 2, 2026 13:55
integrations/test/version_test.ts asserted toBe('1.3.0') while
integrations/src/version.ts exports '1.5.0', so the test has been
failing-if-run since the release bumps. It escaped notice because the
unit:integrations vitest project it lives in is not run by any root
script.

An existing test is modified here rather than added alongside because
the assertion encodes wrong behaviour: the literal is stale, not merely
inconvenient. release-please bumps integrations/package.json and, via
its extra-files entry, integrations/src/version.ts, but never the test,
so any literal goes stale again at the next release and turns the
automated release PR red. Deriving the expected value from
package.json cannot drift and turns a tautology into a real check that
the two release-please targets stay in sync.
vitest.config.ts has declared a unit:integrations project since the
integrations package landed, but no root script ever passed
--project unit:integrations, so integrations/test/**/*_test.ts has
never executed -- not locally via npm test, and not in CI, which runs
npm run test:coverage.

Add the flag to the three scripts that enumerate unit projects.
test:integration, test:e2e and test:cross-language stay untouched;
they are deliberately single-project entry points.
…ript

Nothing runs a vitest project unless a root script names it with
--project <name>, and that coupling is invisible: adding a project to
vitest.config.ts looks like it wires tests into CI, but the tests only
run if someone also edits a script. unit:integrations drifted exactly
this way and went unrun for its whole life.

The guard reads the project names out of vitest.config.ts and the
scripts out of package.json and asserts every declared project is
selected by at least one script. A separate assertion fails when the
project list is empty so the check can never pass vacuously, and an
entry whose name cannot be read throws with its index rather than
being skipped -- a silently skipped project is the drift being
guarded against.

Flags are matched as whole tokens, so --project unit:core-extra does
not satisfy unit:core.
Both tests hand-rolled fileURLToPath + path.resolve + readFileSync +
JSON.parse to read a package.json that vitest and tsc already load
natively. The repo already uses the import-attribute form in 23 places
(e.g. tests/integration/skills/loader/agent_test.ts), and nodenext
resolution enables it with no config change.

Drops the three node: imports from each file, the two path constants,
and the single-caller readRootScripts wrapper. It also removes the two
hand-written annotations that existed only to re-type JSON.parse's any
return: the import infers real types, which in turn makes the
scripts ?? {} fallback provably dead.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant