Skip to content

Test: Resolve integration fixtures from the workspace root instead of installing node_modules per fixture - #299

Open
AmaadMartin wants to merge 2 commits into
mainfrom
feat/reuse-workspace-node-modules-in-fixtures
Open

Test: Resolve integration fixtures from the workspace root instead of installing node_modules per fixture#299
AmaadMartin wants to merge 2 commits into
mainfrom
feat/reuse-workspace-node-modules-in-fixtures

Conversation

@AmaadMartin

@AmaadMartin AmaadMartin commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

  1. Link to an existing issue (if applicable):
    Closes: #issue_number
    Related: #issue_number
  2. Or, if no issue exists, describe the change:

Problem: The app_loader, agent_loader and skills/script_js integration fixtures each run their own npm install in a beforeAll hook. That is 8 installs of the same ~600-package dependency graph per integration run, into 8 throwaway node_modules trees, on ubuntu-latest / windows-latest / macos-latest, on every push and PR. Nothing under test needs it: npm install at the repository root is already a hard precondition of the test job (.github/workflows/validation.yaml:31-41), and Node resolves @google/adk / @google/genai by walking up to that root.

On a 2-4 vCPU runner those 8 concurrent installs are pure contention. Locally in this sandbox they do not even complete: a single fixture install measured 65.84 s / 606 packages, so all three suites blow their beforeAll timeout and are red on main.

Solution: Delete the 8 installs and let the fixtures resolve from the workspace-root node_modules that the root install already populates. Each fixture's start script now invokes the built dev CLI directly.

Why node <path>/cli_entrypoint.js and not adk run or npx @google/adk-devtools run (please read — this corrects a wrong assumption)

The design this PR was written against assumed npm install populates node_modules/.bin/adk (from dev/package.json's bin), so the fixtures could just say adk run agent.ts and let npm's PATH walk find it. That assumption is false, and I verified it three ways before changing the approach:

  1. In this repo, after a clean npm install on a fresh checkout, node_modules/.bin/adk does not exist. It only appears if you run npm install again after npm run build.
  2. Minimal reproduction outside this repo (workspaces root + one workspace declaring bin), run under npm 9.2.0, 10.9.9 and 11.18.0 — all three skip the workspace bin link when the bin target file does not exist at install time.
  3. CI runs npm install (line 32) → npm run build (line 38) → tests (line 41), so dev/dist/esm/cli_entrypoint.js never exists during the install. node_modules/.bin/adk is therefore never created in CI.

Confirmed empirically in the fixture with the bin link removed:

> dirname-test@1.0.0 start
> adk run agent.ts
sh: line 1: adk: command not found

npx @google/adk-devtools run … (the current script) is worse, not better: npm resolves a package name against the nearest package root — the fixture — so once the fixture has no node_modules it goes to the registry and silently tests the published devtools instead of this working tree. A green build proving nothing is worse than a red one.

node ../../../../dev/dist/esm/cli_entrypoint.js run <file> has neither problem: it cannot fall back to the registry, and it fails loudly if the workspace was not built. Verified working with no fixture node_modules and no .bin/adk:

$ cd tests/integration/agent_loader/__dirname && npm run start
> node ../../../../dev/dist/esm/cli_entrypoint.js run agent.ts
Running agent dirname_agent, type exit to exit.
[user]: [dirname_agent]: I'm stubby model response!

What changed

  • 7 fixture manifestsscripts.start switched to the built CLI; dead devDependencies (file:../../../../dev, file:../../../../core) removed. discovery/package.json only loses its devDependencies (it has no scripts).
  • Every fixture package.json stays on disk. getTypeFromPackageJson (dev/src/utils/agent_loader.ts:591-614) walks to the nearest manifest, so deleting one would let the walk reach the repo root's "type": "module" and flip app_ts/app_js/app_default/__dirname/__filename/discovery from CJS to ESM — the CJS-vs-ESM split is exactly what these suites exist to test. No tests/integration/package.json was added for the same reason: 9 agent files under tests/integration/ have no sibling manifest and currently resolve ESM by walking to the root.
  • 3 test files — the install hooks become a one-line precondition assert; the node_modules / package-lock.json teardown goes away. No assertion changed. script_js keeps its three generated-artifact removals; app_loader keeps loader.disposeAll().
  • tests/integration/workspace_cli.ts (new, 40 lines, 1 exported function) — asserts the built CLI exists. It earns its place: getResponse (tests/integration/test_case_utils.ts:376) only subscribes to stdout, so a missing build surfaces as expected '' to contain "I'm stubby model response!" with node's MODULE_NOT_FOUND swallowed. With the guard you get Missing …/dev/dist/esm/cli_entrypoint.js. Run npm install && npm run build at the repository root first.
  • dev/src drive-by fix (required, not optional) — see below.
  • tests/integration/build_setup/**is deliberately untouched. Itsfile:installs arenpm pack+ extract, so each fixture gets a real published-layout copy ofcorehonouring thefiles allowlist — which is the property that suite exists to prove. It keeps all 6 installs.

Trade-off, stated explicitly: the 7 affected fixtures no longer demonstrate a self-contained consumer install; @google/adk now resolves through a workspace symlink into core/. Node still enforces core/package.json's exports map, so import '@google/adk' behaves identically; what those fixtures no longer exercise is the files allowlist. That coverage lives in build_setup, which keeps its real installs.

Required dev/src fix: getProjectNodeModulesDir did not walk up

adk run defaults to compile: true, bundle: true and writes the bundle to a temp directory outside the repo. esbuild's external list (sqlite3, onnxruntime-node, pg-native, jiti, …) is not inlined, so at runtime Node must resolve those from the temp dir — which only works because linkProjectNodeModules symlinks a node_modules into it.

getProjectNodeModulesDir found the nearest package.json and then checked only for a node_modules sibling of that file. These fixtures keep their package.json but lose their node_modules, so it returned undefined, no symlink was created, and any external reached at runtime would be MODULE_NOT_FOUND. That is a regression this PR would otherwise introduce, and it is the same latent bug for any real user running adk run inside an npm/pnpm workspace with hoisted dependencies.

It now walks up for the nearest ancestor node_modules, mirroring Node's own resolution. Strictly additive: for a standalone project with its own install the nearest package.json and the nearest node_modules are the same directory, so the result is identical; it never removes a link that used to be created.

tryToFindFileRecursively and the new tryToFindFolderRecursively now share one bounded walk that differs only in the existence predicate, rather than the second being a copy-paste of the first. tryToFindFileRecursively keeps its exact signature, contract and error string — it still has a live caller at dev/src/cli/deploy/deploy_utils.ts:157.

Collision check against open PRs on this fork (required, recorded here)

gh pr list --repo AmaadMartin/adk-js --state open --limit 100 + gh pr diff --name-only on every plausibly adjacent PR. Four overlap; none lands this change:

PR What it does Relationship
#218 fix/flaky-install-bound-integration-suites Adds tests/integration/fixture_project.ts (installFixtureProject) and routes all four suites through it; keeps every install, adds cache flags + CI tweaks. Touches the same 3 files, opposite direction: it stabilises the installs, this removes them.
#276 feat/trim-integration-test-npm-installs Stacked on #218. Removes the discovery install only (1 of 8) and deletes discovery/package.json, accepting the CJS→ESM flip. Genuine partial overlap on the discovery hunk.
#257, #237, #260, #256, #247, #235 Timeout retuning / vitest-project + CI-job splits. Adjacent, no semantic overlap.

I branched from main rather than stacking, for three reasons: (a) #276 deletes discovery/package.json, which contradicts the CJS/ESM invariant above — stacking would inherit a design decision this change is specifically arguing against; (b) both bases are unmerged fork branches and stacking would drag #218's workflow/go_server.ts/test_server_test.ts hunks into this diff; (c) the changes are semantically disjoint — this PR deletes the very installs the fixture_project.ts helper exists to wrap. If #276 lands first, the discovery hunk here becomes a trivial conflict to resolve in its favour or mine; there is no logical dependency in either direction.

Testing Plan

Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.

All runs below use node v22.22.2 / npm 9.2.0, with git clean -xdf tests/integration before each run (a stale fixture node_modules silently satisfies resolution and invalidates the measurement) and node_modules/.bin/adk deleted, to reproduce the CI state exactly.

Measured before/after — CI, Run tests and check code coverage step

The number this task actually owes. Baseline = the three most recent validation runs on branches that touch none of these suites.

OS this PR baseline runs
ubuntu-latest 252 s 248 / 243 / 248 s
macos-latest 340 s 247 / 181 / 221 s
windows-latest 398 s 400 / 367 / 368 s

The step total does not improve, and I am not going to claim it does. On ubuntu and windows it is flat; the macos sample is higher but that runner swings 181 → 247 s across baseline runs alone, so one sample says nothing.

Per test file, ubuntu-latest (the low-noise runner, same four runs):

file this PR baseline runs
app_loader/app_loader_test.ts 42.8 s 48.8 / 45.5 / 48.7 s
agent_loader/agent_dirname_test.ts 12.7 s 18.3 / 18.3 / 18.4 s
skills/script_js/agent_test.ts 4.9 s 6.5 / 6.1 / 6.2 s
build_setup/build_setup_test.ts (unchanged control) 93.0 s 86.9 / 85.9 / 86.4 s

The three changed files get 11-31% faster and the untouched control gets ~7% slower — because it now competes for a machine the deleted installs used to occupy. Read together, those two rows say the same thing: the integration step is bounded by build_setup_test.ts's six sequential installs, which this change deliberately does not touch, so freeing 8 installs elsewhere converts to slack rather than to wall clock. The win here is 8 fewer concurrent npm install processes and 8 fewer node_modules trees written and deleted per run per OS, not a faster step.

CI flakiness: 3 attempts were needed, and this change plausibly contributes (please read)

Attempt 1: macos failed on AgentLoader discovery … should discover apps vs agents at 40848 ms against its 40000 ms limit. Attempt 2: windows failed on tests/integration/a2a/basic/a2a_agent_test.ts with listen EACCES: permission denied ::1:49881 (a random ephemeral port landing in a Windows reserved range). Attempt 3: windows failed on run_skill_script_tool_test.ts > successfully executes a real PowerShell skill script at 5022 ms. Attempt 4: all three OSes green.

None of those three is in a file this PR touches, each already has a dedicated open PR, and all four of the suites in scope here passed on all three OSes in every attempt. The discovery test in particular is pre-existingly marginal: 22.3 s and 31.3 s on baseline macos, and 28.4 s on ubuntu in the same run that timed out on macos — statistically identical to the 25.7-28.6 s baseline, so this change did not make that test slower.

But the honest read is not "unrelated": removing 8 install hooks means more test files execute concurrently, which raises contention and therefore the flake rate of tests that are already sitting near a hard timeout or racing for a port. I have deliberately not retuned TEST_EXECUTION_TIMEOUT or split the vitest projects here — that is separate work with several PRs already open against it, and folding it in would muddy the measurement above.

Local measurements (node v22.22.2 / npm 9.2.0, git clean -xdf tests/integration before every run — a stale fixture node_modules silently satisfies resolution and invalidates the measurement — and node_modules/.bin/adk deleted to reproduce CI exactly). This sandbox proxies npm through a slow gateway, so a single fixture install costs 65.84 s / 606 packages and every install-bound suite times out on main; the numbers are therefore an upper bound on the benefit, not a CI prediction.

npx vitest run --project integration before (main) after
run 1 / 2 / 3 170.55 / 171.46 / 149.12 s 70.40 / 62.34 / 68.85 s
median 170.55 s (4 files failed) 68.85 s (1 file failed)
file, run in isolation before after
app_loader/app_loader_test.ts 167.02 s — FAIL, Hook timed out in 40000ms, 6 skipped 36.09 s — 6 passed
agent_loader/agent_dirname_test.ts 126.87 s — FAIL, hook timeout, 3 skipped 18.85 s — 3 passed
skills/script_js/agent_test.ts 66.99 s — FAIL, Hook timed out in 60000ms, 1 skipped 11.41 s — 1 passed
build_setup/build_setup_test.ts (unchanged control) FAIL, Hook timed out in 10000ms FAIL, 68.32 s — byte-identical file, unchanged behaviour

Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.

  • npx vitest run --project unit:dev dev/test/utils/file_utils_test.ts dev/test/utils/agent_loader_test.ts50 passed
    • 4 new tryToFindFolderRecursively cases: found in the source folder; found several levels up; skips a same-named plain file and keeps walking (the branch that distinguishes it from the file variant); throws once maxIterations is exhausted.
    • 2 new AgentFile.load() cases: links an ancestor node_modules into the compiled output dir (asserts the real fs.readlink target and that the walk starts from the agent's own directory); skips the link entirely when no ancestor has one.
    • dev/test/utils/agent_loader_test.ts's explicit vi.mock factory was updated to export tryToFindFolderRecursively (the suite fails to load otherwise). No test was deleted, skipped, .only'd or weakened.
  • npx vitest run --project integration tests/integration/workspace_cli_test.ts2 passed

Every new test was proven to fail against the unfixed code. Mutations run and their exact failure messages:

mutation test that went red message
getProjectNodeModulesDir reverted to nearest-dir-only (no upward walk) links an ancestor node_modules into the compiled output directory ENOENT: no such file or directory, readlink '/tmp/agent-loader-output-testxFqI9l/node_modules'
try/catch removed from getProjectNodeModulesDir skips the node_modules link when no ancestor has one Error: No node_modules found in /nowhere
tryToFindFolderRecursively given isFileExists instead of isFolderExists all 3 positive folder-walk cases promise rejected "Error: No node_modules found in /a/b/c or…" instead of resolving
exhausted walk returns a path instead of throwing both …throws when … not found within maxIterations cases promise resolved "'/a/node_modules'" instead of rejecting
assertWorkspaceAdkCliAvailable made a no-op rejects naming the missing CLI path… promise resolved "undefined" instead of rejecting
DEV_CLI_PATH pointed at dist/cjs both workspace_cli cases promise rejected "Error: Missing /tmp/workspace-cli-…" / expected [Function] to throw error including 'Missing …'

Coverage. dev/src/utils/file_utils.ts: every line of the new code is covered (the only uncovered lines, 36-51, 58-61, 96-99, are pre-existing console.error paths in createFolder/removeFolder/listFiles/saveToFile). getProjectNodeModulesDir is fully covered on both branches (try 37 hits, catch 1 hit). I deliberately did not add coverage for the pre-existing linkProjectNodeModules branches at 632-633 (link already exists) and 642-645 (EEXIST tolerance), or for its win32 'junction' argument: none of them is new code, and covering the platform branch needs a process.platform global mutation that buys no signal about this change.

Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.

git clean -xdf tests/integration
npm install && npm run build
rm -f node_modules/.bin/adk          # reproduce the CI state exactly (see the note above)

# 1. Full integration project.
npx vitest run --project integration
#    -> 35/36 files pass. Only build_setup_test.ts fails, identically to main,
#       on its own npm install hook timeout in this sandbox.

# 2. Prove resolution really comes from the workspace root, not leftovers.
cd tests/integration/agent_loader/__dirname && ls   # agent.ts  model_response.json  package.json
npm run start                                        # -> [dirname_agent]: I'm stubby model response!

# 3. Prove the failure mode is loud (negative check).
mv dev/dist/esm/cli_entrypoint.js /tmp/ && npx vitest run --project integration \
  tests/integration/agent_loader/agent_dirname_test.ts

Step 3 output — fails in ~7 s naming the path, not on a 40 s timeout, and above all not via a silent registry download:

Error: Missing /…/adk-js/dev/dist/esm/cli_entrypoint.js. Run `npm install && npm run build` at the repository root first.
 ❯ assertWorkspaceAdkCliAvailable tests/integration/workspace_cli.ts:35:11

Postconditions verified. git grep -n "npm install" -- tests/integration returns only build_setup_test.ts:29 plus the two documentation/message strings in the new helper. git grep -l adk-devtools -- tests/integration returns only build_setup/** and a pre-existing in-process import {AdkApiServer} from '@google/adk-devtools' in adk_web/webui_test.ts. git status is clean after a full integration run, with no node_modules or package-lock.json left under app_loader/, agent_loader/ or skills/.

Known risk, verified rather than assumed. The 6 fixture agents that import {createModelContent} from '@google/genai' now resolve 1.52.0 from the workspace root instead of 2.9.0 from their own tree (the lockfile hoists 1.52.0 for @google-cloud/vertexai@^1.45.0 and pushes core's ^2.9.0 down to core/node_modules). Confirmed by resolving from a fixture directory: it lands on the root copy. All three suites pass on it, and tsc already type-checked these files against the root 1.52.0 typings, so the type-check and the runtime now agree where they previously disagreed. core's own import still resolves to core/node_modules/@google/genai@2.9.0. Deduping @google/genai is separate, already-tracked work and is not absorbed here.

Repo-wide gates. CI on the pushed commit eacb087: run-tests (ubuntu-latest) ✅, run-tests (macos-latest) ✅, run-tests (windows-latest) ✅, cross-language ✅, check-license ✅. Locally on the same commit: npm run build ✅, npm run lint ✅, npm run format:check ✅. npm run ts:check reports 308 errors before and after this change — all pre-existing in core/test/**; this PR adds none, and none of my files appear in the output.

No suppressions. git diff main -U0 | grep -E '@ts-expect-error|@ts-ignore|eslint-disable|as any|: any|v8 ignore|istanbul ignore' returns nothing. No CHANGELOG.md, no version bumps, no lockfile churn, no new dependency, no CI workflow edits.

Checklist

[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.

Amaad Martin added 2 commits July 30, 2026 00:10
…estor

getProjectNodeModulesDir only looked for a node_modules sitting next to the
nearest package.json, so a project in an npm/pnpm workspace - where deps are
hoisted to the workspace root - got no node_modules symlink in the esbuild
output directory and every entry on the bundler's external list became
unresolvable at runtime.

Walk up for the nearest ancestor node_modules instead, mirroring Node's own
resolution. tryToFindFileRecursively and the new tryToFindFolderRecursively now
share one bounded walk that differs only in the existence predicate.
…f installing them

The app_loader, agent_loader and skills/script_js fixtures each ran their own
npm install, materialising the same ~600-package dependency graph into eight
throwaway node_modules trees per integration run, on three CI runner OSes.
Nothing under test needs that: the workspace-root install is already a
precondition of the test job, and Node resolves @google/adk and @google/genai
by walking up to it.

The fixtures now invoke the built dev CLI directly. `npx @google/adk-devtools`
resolves the *package* against the fixture's own tree and would silently fetch
the published devtools from the registry once the fixture has no node_modules -
a green build proving nothing. `adk` on PATH is not an option either: npm skips
linking a workspace bin whose target does not exist yet, and CI installs before
it builds, so node_modules/.bin/adk never exists there.

Each fixture keeps its package.json: getTypeFromPackageJson walks to the
nearest one, so removing it would flip the CJS fixtures to ESM via the root
manifest and erase the very distinction these suites test.

build_setup is deliberately untouched. Its file: installs produce real
published-layout copies of core, which is what that suite exists to verify.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant