Skip to content

Chore: derive the vitest coverage include list from the npm workspaces - #362

Open
AmaadMartin wants to merge 1 commit into
mainfrom
fix/coverage-include-from-workspaces
Open

Chore: derive the vitest coverage include list from the npm workspaces#362
AmaadMartin wants to merge 1 commit into
mainfrom
fix/coverage-include-from-workspaces

Conversation

@AmaadMartin

@AmaadMartin AmaadMartin commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

  1. Link to an existing issue (if applicable):
    Closes: #issue_number
    Related: #issue_number
  2. Or, if no issue exists, describe the change:

Problem: vitest.config.ts hardcoded the coverage denominator as one glob per workspace:

include: [
  'core/src/**/*.ts',
  'dev/src/**/*.ts',
  'integrations/src/**/*.ts',
],

That list duplicates the root package.json workspaces array with nothing keeping the two in sync. If a workspace is added and this list is not extended by hand, the workspace's sources are simply absent from the denominator.

The failure is silent by construction. coverage.thresholds are percentages of whatever is in the denominator, so dropping a workspace removes it from the numerator and the denominator together: the reported percentage stays healthy, npm run test:coverage in .github/workflows/validation.yaml stays green, and nothing in the output hints that a whole package is unmeasured. (Contrast the opposite mistake — sweeping extra files in — which fails loudly by pushing the percentage below the threshold.)

I reproduced this end to end. With a synthetic 4th workspace (plugins/, containing one uncalled function) declared in package.json, comparing the key set of coverage/coverage-final.json:

config state denominator plugins/src/thing.ts measured?
hardcoded list (before) 221 files no — silently absent
derived from workspaces (after) 222 files yes

This is not hypothetical drift. Commit a65d05ff ("feat(integrations): create new top-level integrations package", #449) added the integrations workspace and did two of the three required updates — it extended coverage.include and added the unit:integrations vitest project — but never added --project unit:integrations to the test* scripts. That script omission is a separate concern and deliberately not touched here; it is cited only as evidence that hand-maintained parallel lists in this file do drift.

Solution: derive the include list from the workspaces declaration, so the denominator is a function of the declared workspace set and there is no second list to remember:

const {workspaces} = JSON.parse(
  readFileSync(path.resolve(__dirname, 'package.json'), 'utf8'),
) as {workspaces: string[]};

const coverageInclude = workspaces.map(
  (workspace) => `${workspace}/src/**/*.ts`,
);

Plus one tripwire test that pins the config's resolved coverage.include to that derivation, so re-hardcoding the list fails CI.

Design notes:

  • The thresholds are deliberately unchanged (statements 86 / branches 87 / functions 88 / lines 86), and so is the comment above them. This change is required to leave the denominator alone, and the measurements below show it does. The re-justification asked for is evidence they are still correct, not new numbers.
  • No coverage.exclude was added. In vitest, setting coverage.exclude replaces the default exclude array rather than merging with it, so adding one entry would silently re-enable node_modules, dist, dot-directories, *.d.ts and config files as coverage candidates. The derived include is already exact.
  • No try/catch or fallback list around the read. If the root package.json were unreadable or malformed this throws at config load and vitest refuses to start, which is the correct loud behaviour; a fallback would reintroduce exactly the silent hardcoded list this change removes.
  • import path from 'path' was left alone rather than churned to node:path; only the new import uses the node: prefix.
  • The test asserts against the default export's resolved coverage.include, not against an exported copy of the derivation. Exporting coverageInclude and asserting on that would let someone re-hardcode include: while leaving the export in place and still pass; reading the real config value closes that hole and keeps the config's only export the default one.
  • Placed at tests/integration/repo_config/, matching the existing convention — every test under tests/integration/ lives in a topic subdirectory, and there are no top-level *_test.ts files there.

Collision check (required before starting): I ran gh pr list --state open --limit 100 and inspected every plausibly adjacent PR. No open PR lands this change. Three touch the same file or area and are semantically disjoint:

Testing Plan

Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.

Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.

Added tests/integration/repo_config/coverage_config_test.ts, a tripwire asserting the config's resolved coverage.include equals the derivation from workspaces:

$ npx vitest run --project integration coverage_config_test
 ✓ |integration| tests/integration/repo_config/coverage_config_test.ts (1 test) 4ms
 Test Files  1 passed (1)
      Tests  1 passed (1)

The new config lines carry no coverage obligation — vitest's default **/{...,vitest,...}.config.* exclude keeps vitest.config.ts out of the denominator — so the test exists for regression value, not for a percentage.

Proof the test can fail. Two mutations, both run against the test:

Mutation A — narrow the derivation to workspaces.slice(0, 2).map(...):

AssertionError: expected [ 'core/src/**/*.ts', …(1) ] to deeply equal [ 'core/src/**/*.ts', …(2) ]
  [
    "core/src/**/*.ts",
    "dev/src/**/*.ts",
-   "integrations/src/**/*.ts",
  ]

Mutation B — restore the hardcoded literal and declare a 4th workspace, i.e. reproduce the exact drift this change prevents:

AssertionError: expected [ 'core/src/**/*.ts', …(2) ] to deeply equal [ 'core/src/**/*.ts', …(3) ]
  [
    "core/src/**/*.ts",
    "dev/src/**/*.ts",
    "integrations/src/**/*.ts",
-   "plugins/src/**/*.ts",
  ]

Manual End-to-End (E2E) Tests:

The core verification is that the denominator is unchanged for the repository as it stands. Reproduce with:

npm install && npm run build
CI=true npm run test:coverage -- --coverage.reportOnFailure

1. Deterministic denominator check. Compare the sorted key set of coverage/coverage-final.json (the exact file set forming the denominator) between the hardcoded and derived configs, over a single cheap coverage run:

rm -rf coverage
npx vitest run --project integration coverage_config_test --coverage --coverage.reporter=json
node -e "console.log(Object.keys(require('./coverage/coverage-final.json')).length)"

Result: 221 files both ways, diff of the sorted key sets is empty — identical file sets. (221 = 191 core/src + 27 dev/src + 3 integrations/src.)

2. Full per-file coverage table. npm run test:coverage before and after, on the same machine and environment:

statements branches functions lines
threshold 86 87 88 86
main (before) 90.1 88.95 90.88 90.1
this branch (after) 90.05 88.92 90.79 90.05
this branch, repeat run 90.1 88.95 90.88 90.1

The per-file row list is byte-identical before and after — no row appears or disappears, which is the denominator claim. All four values clear 86 / 87 / 88 / 86 with ~2-4 points of headroom on every run, which is why the thresholds are left untouched.

The small percentage wobble on 10 rows is numerator flake, not a denominator change, and is demonstrated as such: running the coverage twice on this same commit produces the same wobble on the same rows, and the second run reproduces the main baseline exactly (90.1 / 88.95 / 90.88 / 90.1). The moving rows — dev/src/utils/agent_loader.ts, dev/src/utils/file_utils.ts, core/src/tools/base_tool.ts — are precisely those exercised by the integration suites whose beforeAll runs npm install in a fixture (agent_dirname_test, app_loader_test, build_setup_test, skills/script_js). Those suites hit hook timeouts in my sandbox, identically before and after; they are a pre-existing environment limitation, not a regression from this change.

2b. CI coverage gate, measured on the real matrix. npm run test:coverage ran green on all three CI operating systems against this branch, i.e. the derived denominator, with the thresholds untouched:

All files statements branches functions lines
threshold 86 87 88 86
ubuntu-latest 90.12 88.93 90.88 90.12
macos-latest 90.12 88.76 90.87 90.12
windows-latest 90.17 89.03 90.88 90.17

The tightest margin is branches on macOS at 88.76 against a threshold of 87 — the same leg that is tightest on main today. The new tripwire test also passed on all three (2-6ms). This is the threshold re-justification: the values are still correct, so they are left alone.

3. Bug reproduction / fix confirmation. With a synthetic plugins workspace declared in package.json: the hardcoded config yields 221 files with plugins/src/thing.ts absent and no warning; the derived config yields 222 files with plugins/src/thing.ts present at 0%, so the omission becomes loud. The synthetic workspace was removed afterwards and is not part of this diff.

4. Full gate sweep, on the exact commit pushed:

npm run ts:check      # 308 pre-existing errors on main, 308 on this branch -- zero new, none in the changed files
npm run lint          # pass
npm run format:check  # pass ("All matched files use Prettier code style!")
bash scripts/check_license.sh  # pass
npm run docs:check    # pass

(tsc --noEmit type-checks the whole tree including tests and does not pass on main today; I verified the count is unchanged at 308 by running it on a stashed tree, and that no error references vitest.config.ts or the new test.)

No suppressions were added: git diff main -U0 | grep -E '@ts-expect-error|@ts-ignore|eslint-disable|v8 ignore|as any|as never' returns nothing.

Checklist

[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.

…kspaces

The coverage denominator was a hand-maintained glob per workspace that
duplicated the root package.json `workspaces` array with nothing keeping the
two in sync. A workspace missing from that list drops out of both the
numerator and the denominator, so the reported percentage stays healthy and
the CI coverage gate stays green while an entire package goes unmeasured.

Derive the include list from `workspaces` instead, and add a tripwire test
that pins the vitest config's resolved `coverage.include` to that derivation
so re-hardcoding the list fails CI.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant