Skip to content

Test: pin the integration-fixture install mode (and reject the fixture-lockfile proposal) - #407

Open
AmaadMartin wants to merge 6 commits into
fix/integration-hook-timeout-single-sourcefrom
feat/pin-integration-fixture-installs
Open

Test: pin the integration-fixture install mode (and reject the fixture-lockfile proposal)#407
AmaadMartin wants to merge 6 commits into
fix/integration-hook-timeout-single-sourcefrom
feat/pin-integration-fixture-installs

Conversation

@AmaadMartin

@AmaadMartin AmaadMartin commented Jul 31, 2026

Copy link
Copy Markdown
Owner

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

  1. Link to an existing issue (if applicable):
    Closes: #issue_number
    Related: #issue_number
  2. Or, if no issue exists, describe the change:

Problem: Integration-fixture installs are non-deterministic in two independent ways.

The install mode is unpinned. The 14 fixtures depend on the in-repo packages via "@google/adk": "file:../../../../core" and "@google/adk-devtools": "file:../../../../dev", installed with a bare npm install. Whether that produces two symlinks or a packed copy of the entire transitive graph is decided by npm's install-links setting, which this repo never pins — and whose default flipped from true to false at npm 9→10. The repo also pins no Node/npm version: no .nvmrc, no .npmrc, no engines field, no packageManager field, and actions/setup-node@v6 is used with no node-version input, so CI runs on whatever the runner image ships.

Measured on this branch (js_esm fixture, Node 22.22.2 / npm 9.2.0, warm cache, root installed first):

npm default here (install-links=true) --install-links=false
fixture install 64.4 s 0.616 s
generated package-lock.json 312,841 B, 632 entries 3,739 B, 5 entries
node_modules top level 424 directories 2 symlinks
link: true on the workspace entries absent present on both

The installed versions are unpinned. CI installed the repo root with npm install, which re-resolves and silently accepts a manifest that has drifted from the committed package-lock.json. Since core's 29 and dev's 19 runtime dependencies resolve through the repo-root node_modules (verified: from core/, winston, express and zod all resolve to <repo>/node_modules/...), a drifted root tree is silently handed to every fixture.

Solution: three changes, all test/CI plumbing. No core/, dev/ or integrations/ source is touched, so there is no public API or runtime behavior change.

  1. Pin the install mode. All 5 literal fixture-install call sites (14 runtime invocations) go through one shared installFixtureDeps in tests/integration/test_case_utils.ts passing --install-links=false --no-audit --no-fund, plus a shared cleanupFixtureDeps teardown. A new regression test asserts the fixture really did get links.
  2. Pin the versions. .github/workflows/validation.yaml:32 and .github/workflows/cross-language-integration.yml:26 switch from npm install to npm ci, so the root node_modules is reified strictly from the committed lockfile. Change Can you implement parallel tool in adk-js calling for this issue: https://github.com/google/adk-js/issues/233. Ignore the PR that's already been made for it. Staging #1 guarantees the fixtures resolve through that tree; only this guarantees the tree matches what is committed.
  3. Record the decision. tests/**/package-lock.json stays ignored and stays deleted in afterAll, with the rationale at the ignore line.

Why not commit the fixture lockfiles / switch the fixtures to npm ci? That was the original proposal, and the measurements above reject it:

  1. There is no transitive re-resolution or version drift at the fixture level to fix. In link mode the install creates two symlinks and installs nothing else.
  2. npm ci is not faster at the fixture level. The whole speed difference comes from install-links, not from ci vs install.
  3. A committed fixture lockfile would pin nothing. The 5-entry lockfile is the two link stubs plus two entries that merely restate ranges already declared in core/package.json and dev/package.jsonno resolved versions, no integrity hashes.
  4. It would make CI depend on which npm the contributor used: cross-major npm ci hard-fails with EUSAGE — 14 files × every dependency bump × every npm-major mismatch, for zero determinism.

The determinism the proposal wanted is real, but it lives at the repo root, which is what change #2 enforces.

Behavioral change for contributors: a PR that edits any workspace package.json without regenerating the root package-lock.json now fails at the CI install step with a precise EUSAGE message instead of silently re-resolving. That is the intended effect, called out here so reviewers are not surprised. Contributors on npm 9 will also see the integration suites get dramatically faster.

Collision check (gh pr list --repo AmaadMartin/adk-js --state open --limit 100, then gh pr diff --name-only on every plausibly adjacent PR):

Testing Plan

Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.

Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.

vitest.config.ts restricts coverage collection to core/src, dev/src and integrations/src; nothing here is inside that set, so no coverage threshold applies and none were touched. The real check is an outcome assertion on the install, not a mock of npm's argv.

New regression testshould link the workspace packages rather than pack them, added inside the existing describe.each so it runs for all six build_setup fixtures. It parses <fixture>/package-lock.json (typed with a narrow local interface FixtureLockfile; no any, no suppressions anywhere in this diff) and asserts link === true for both workspace entries. It asserts on the lockfile rather than lstat().isSymbolicLink() because npm creates junctions rather than symlinks on Windows and the matrix includes windows-latest, whereas link: true is stable across npm majors and every OS.

Proof the new test can fail (required). Mutation: in installFixtureDeps, --install-links=false--install-links=true, then npx vitest run --project integration tests/integration/build_setup -t "should link the workspace packages". Result — all 6 fail:

 FAIL  |integration| .../build_setup_test.ts > Build setup > js_commonjs > should link the workspace packages rather than pack them
 ... (all six fixtures)
AssertionError: expected undefined to be true // Object.is equality
- Expected: true
+ Received: undefined
 ❯ tests/integration/build_setup/build_setup_test.ts:68:63
      Tests  6 failed | 24 skipped (30)
   Duration  428.77s

That run also quantifies the win: 420.85 s for the 6 packed installs alone, against 93.80 s for the entire linked suite including all 26 assertions. The mutation was reverted before committing.

Proof change #2 does something (the EUSAGE guard). Added left-pad to core/package.json without regenerating the root lockfile, then ran both install commands against the identical drifted state:

$ npm install --dry-run --no-audit --no-fund     # the OLD CI step
exit=0
added 75 packages in 2s                          # silently re-resolves

$ npm ci --no-audit --no-fund                    # the NEW CI step
exit=1
npm ERR! code EUSAGE
npm ERR! `npm ci` can only install packages when your package.json and
npm ERR! package-lock.json ... are in sync.
npm ERR! Missing: left-pad@1.3.0 from lock file

Reverted afterwards; git status --porcelain core/package.json package-lock.json is empty and npm ci is green again. Against the committed lockfile at this HEAD, npm ci succeeds in 16.1 s, so change #2 will not redden CI on merge.

Error paths. installFixtureDeps awaits execAsync with no try/catch, so a failed fixture install rejects out of beforeAll unchanged; there is no npm ci -> npm install fallback anywhere, since a fallback that rescues a stale lockfile would make the enforcement a no-op. cleanupFixtureDeps was exercised on the error path by the mutation run above: it ran in afterAll despite six failing tests and left git status --porcelain tests/ clean.

One failure contract per teardown. fs.rm(force: true) already swallows ENOENT, so the .catch(() => {}) wrappers the old teardowns carried were dead for their only plausible error and merely hid the rest. They are gone from every hook this PR touches -- the shared helper, the dist/ removal in build_setup_test.ts, and the three generated-file removals in the skills suite -- so a hook no longer swallows rm failures on one line and rejects on them the next. The consequence is deliberate: an EBUSY/EPERM (plausible on the windows-latest leg) now fails the afterAll instead of silently leaving junk for the next run, and cleanupFixtureDeps' doc says so rather than claiming "best-effort".

Targeted suites re-run on the exact committed tree (per the repo guideline, not the full npm test):

$ npx vitest run --project integration tests/integration/build_setup
  Test Files 1 passed (1) | Tests 26 passed | 4 skipped (30) | 99.71s
$ npx vitest run --project integration tests/integration/app_loader
  Test Files 1 passed (1) | Tests 6 passed (6) | 45.24s
$ npx vitest run --project integration tests/integration/agent_loader
  Test Files 1 passed (1) | Tests 3 passed (3) | 24.37s
$ npx vitest run --project integration tests/integration/skills/script_js
  Test Files 1 passed (1) | Tests 1 passed (1) | 13.33s
$ git status --porcelain        # after all four -> empty

Empty git status after the runs confirms none of the 14 fixture directories leaks a node_modules or package-lock.json.

CI status: absent, not green — validated locally instead. This is a stacked PR whose base is fix/integration-hook-timeout-single-source, and .github/workflows/validation.yaml triggers on pull_request: branches: [main], so run-tests will not fire for this base. On the exact pushed commit:

$ npm run build   # ok
$ npm run lint    # ok  (eslint "**/*.ts", clean)
$ npm ci          # ok, 14.9s  (root lockfile in sync at HEAD)

tsc --noEmit reports no error in any file this PR touches. (The repo-wide ts:check is not currently green — pre-existing failures in core/test/** and two unrelated tests/integration/** files, which PRs #326/#370 address.)

Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.

# 1. See the two install modes diverge on your own npm.
npm ci                       # root, so the fixtures have something to resolve against
cd tests/integration/build_setup/js_esm

rm -rf node_modules package-lock.json
time npm install --install-links=true  --no-audit --no-fund
wc -c package-lock.json; ls node_modules | wc -l      # large lockfile, hundreds of dirs

rm -rf node_modules package-lock.json
time npm install --install-links=false --no-audit --no-fund
wc -c package-lock.json; ls -A node_modules | wc -l   # ~3.7 KB, 2 symlinks
node -e "const p=require('./package-lock.json').packages;
         console.log(p['node_modules/@google/adk'].link)"   # true

rm -rf node_modules package-lock.json; cd -

# 2. Confirm core's deps come from the ROOT lockfile, not a per-fixture resolve.
node -e "console.log(require.resolve('winston/package.json',{paths:['./core']}))"

# 3. Confirm the new CI install step catches manifest drift.
#    Add any dependency to core/package.json, do NOT run npm install, then:
npm ci        # -> EUSAGE, "Missing: <dep> from lock file"
#    Revert core/package.json afterwards.

Checklist

[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.

Amaad Martin added 6 commits July 31, 2026 11:16
… pinned

Integration fixtures depend on the in-repo core and dev packages via
file: specifiers, so npm's install-links setting decides whether the
fixture gets two symlinks or a packed copy of their whole transitive
graph. That setting is unpinned today and its default flipped between
npm majors, so the same suite installs in 0.6s on one contributor's
machine and 64s on another's -- and only the link mode resolves core's
and dev's dependencies through the repo-root node_modules that the
committed root package-lock.json pins.

Add one install helper that passes the flag explicitly, and one teardown
helper, so all five fixture-install call sites share the same semantics.
Replaces the five bare `npm install` call sites and their hand-rolled
node_modules/lockfile teardowns. The build_setup suite keeps its dist/
removal inline and the skills suite keeps its generated-artifact
removals, since neither is part of dependency installation.

Adds a regression test, one per build_setup fixture, asserting the
lockfile records link: true for both workspace packages. It asserts on
the lockfile rather than lstat().isSymbolicLink() because npm creates
junctions on Windows, which the CI matrix includes.
…k-budget rationale

The tests/**/package-lock.json ignore was added incidentally and has no
recorded rationale, which invites re-litigation. In link mode a fixture
lockfile is five entries restating ranges already declared in
core/package.json and dev/package.json, with no resolved versions and no
integrity hashes, so committing one would pin nothing while costing 14
files of churn per dependency bump.

The integration hook budget's justification cited a ~70s network-bound
install. Pinning --install-links=false makes the install a sub-second
link operation with no network access, so the remaining cost in those
hooks is the ts_* fixtures' npm run build. Corrects the prose; the 120s
value is unchanged.
Pinning --install-links=false guarantees the fixtures RESOLVE THROUGH the
repo-root node_modules; it does not guarantee that tree matches the
committed package-lock.json. npm install re-resolves and silently accepts
a manifest that has drifted from the lockfile, so the fixtures can be
handed a transitive graph the lockfile never recorded. npm ci reifies
strictly from the lockfile and fails loudly instead.

Verified both halves locally by adding a dependency to core/package.json
without regenerating the root lockfile: npm install exits 0 and re-resolves
("added 75 packages"), while npm ci exits 1 with

  npm ERR! code EUSAGE
  npm ERR! Missing: left-pad@1.3.0 from lock file

Against the committed lockfile at this HEAD, npm ci succeeds in 16.1s, so
this does not redden CI on merge.
cleanupFixtureDeps' doc claimed "best-effort", but fs.rm(force: true)
only swallows ENOENT -- an EBUSY or EPERM rejects and fails the afterAll.
That is the intended behavior (a fixture that cannot be cleaned pollutes
the next run), so correct the comment rather than re-add a catch-all.

Drop the .catch(() => {}) still wrapping the dist/ removal two lines
below, so both halves of the same hook report failures the same way.
…ardown contract

The rationale for pinning --install-links=false was written out four
times with three different sets of numbers. Keep the full explanation at
installFixtureDeps, which is the only place that actually sets the flag,
and reduce the other three to a pointer: the .gitignore comment to one
line, the vitest.config.ts hook-budget note to one sentence with no
duplicated figures, and the regression test's comment to just the
non-obvious part (why the lockfile rather than lstat).

Also drop the three now-dead .catch(() => {}) on the skills teardown's
generated-file removals. fs.rm(force: true) already swallows ENOENT,
which is the only error they plausibly guarded -- the same reasoning
cleanupFixtureDeps and the dist/ removal rely on. That afterAll was
swallowing rm failures on three lines and then calling a helper
documented to reject on them.

Document --no-audit --no-fund, which the doc comment above the command
did not previously mention.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant