Skip to content

Test: fail webui_test fast when the repo has not been built (stacked on #359) - #559

Open
AmaadMartin wants to merge 1 commit into
fix/webui-test-named-hook-timeoutsfrom
fix/webui-test-build-precondition-guard
Open

Test: fail webui_test fast when the repo has not been built (stacked on #359)#559
AmaadMartin wants to merge 1 commit into
fix/webui-test-named-hook-timeoutsfrom
fix/webui-test-build-precondition-guard

Conversation

@AmaadMartin

Copy link
Copy Markdown
Owner

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

  1. Link to an existing issue (if applicable):
    N/A — no public issue.
  2. Or, if no issue exists, describe the change:
    Problem: tests/integration/adk_web/webui_test.ts is the only test in the
    repo that is inherently a built-artifact test: it asserts that the compiled
    ADK CLI and the packaged adk-web bundle serve the dev UI over HTTP. When the
    tree is not (fully) built, it fails with messages that describe the symptom and
    never name the cause. Both failure modes were reproduced locally on this branch:
  • Using ADK API server variantAdkApiServer mounts
    express.static(path.join(__dirname, '../../browser'))
    (dev/src/server/adk_api_server.ts), i.e. dev/dist/browser, which
    ensureBrowserAssets() in dev/build.js populates by downloading the
    google/adk-web v1.0.0 adk-web-browser.zip release asset. With that
    directory absent, GET /dev-ui/ falls through to Express's default 404
    handler and the suite reports AssertionError: expected 404 to be 200.
  • Run from ADK CLI variantAdkTsApiServer spawns
    node <repo>/dev/dist/esm/cli_entrypoint.js web …. With that file absent,
    node exits 1 and BaseTestServer.startProcess rejects with
    Error: CLI exited prematurely with code 1.

Note this is not a 60 s spawn-timeout burn, as one might assume: spawn
succeeds and node exits immediately, so the rejection is prompt (measured
7.57 s wall for the whole vitest run, dominated by transform/collect). The DX
problem is the message, not the duration.

The dist/esm present + dist/browser missing state is not exotic:
dev/build.js main() runs esbuild first and ensureBrowserAssets() last, so
an offline machine or a failed release fetch leaves exactly that state.

Solution: a precondition guard inside that one test file. An outer
beforeAll checks the two artifacts with fs.existsSync and, if either is
missing, throws once naming the build command and the missing absolute path(s):

webui_test.ts exercises built output; run `npm run build` first. Missing: <abs path>[, <abs path>]

beforeAll on the outer describe (rather than module scope) runs once, ahead
of both nested describe.each suites' own beforeAll hooks, so it
short-circuits both variants before the 20 s server-start hook and is reported
as a suite failure rather than a file-collection error. On a built repo the
guard costs two existsSync stats and behaviour is unchanged.

Scope is deliberately one file, 24 added lines. Three things were considered
and not done:

  • vitest.config.ts was not touched. The integration project aliases only
    @google/adk and @google/adk-integrations; @google/adk-devtools resolves
    through the workspace symlink to dev/dist/esm/index.js. Adding an alias to
    dev/src would relocate __dirname inside adk_api_server.ts and move the
    express.static root to the nonexistent dev/browser, breaking the
    Using ADK API server variant outright.
  • No generalized guard in tests/integration/test_api_server.ts. A shared
    guard would also cover the tests/integration/a2a/** suites and
    tests/cross_language/a2a/go_ts/; that is tracked as separate work, and
    pre-empting it here would create a conflicting diff.
  • No new module for the helper. An 8-line assertion used by one file is not
    a helper cluster, and extracting it would break the one-file scope.

Accepted limitation (stated plainly): when dev/dist is absent entirely,
the static ESM import of @google/adk-devtools at the top of the file fails to
resolve before any module body or hook executes, so the guard cannot fire. That
case already produces a self-describing error naming dev/dist/esm/index.js, so
no machinery was added to chase it. The guard covers the partial-build states,
which are the ones that produce misleading messages. Converting the import to a
dynamic await import() to beat this would change the describe.each argument
shape for near-zero benefit.

CI impact: none. .github/workflows/validation.yaml runs npm run build
before npm run test:coverage, so the guard never fires on GitHub Actions. This
is a pure local-developer-experience fix. Coverage thresholds are likewise
untouched: the coverage include globs are core/src/**, dev/src/**,
integrations/src/**, so test files are not instrumented and no coverage number
exists for this code.

Collision check. Ran
gh pr list --repo AmaadMartin/adk-js --state open --limit 1000 (457 open PRs)
and filtered for webui / dev-ui / build-guard work. One PR overlaps:
#359 Fix: name the webui_test hook timeouts and give the teardown an explicit budget, which edits the same file and inserts its constants at the same
anchor (between the __dirname shim and the outer describe). It does not
implement this guard, so this is an overlap, not a duplicate — this PR is
therefore stacked on fix/webui-test-named-hook-timeouts rather than branched
from main
, and the guard is written on top of #359's
SERVER_START_TIMEOUT / SERVER_STOP_TIMEOUT constants. Merge #359 first. No
other open PR touches tests/integration/adk_web/.

Testing Plan

Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.

No new test file was added: the guard is test-side logic in an integration
suite that is itself the regression surface, and it lives outside the coverage
include globs, so no instrumented coverage number exists for it (relocating it
into dev/src/** to become coverable would ship test scaffolding in a published
package). It is verified instead by exercising every branch of it directly —
both missing-artifact paths and the all-present path — and by control runs
proving the messages it replaces.

Positive run on a fully built tree (npm install && npm run build):

$ npx vitest run --project integration tests/integration/adk_web/webui_test.ts
 ✓ |integration| tests/integration/adk_web/webui_test.ts (2 tests) 2732ms
 Test Files  1 passed (1)
      Tests  2 passed (2)

Proof the guard can fail (mutation / negative runs). Each artifact was moved
away in turn and the suite re-run. Both branches of
REQUIRED_BUILD_ARTIFACTS.filter(...) and the missing.length > 0 throw are
exercised:

$ mv dev/dist/browser /tmp/adk-browser-bak
$ npx vitest run --project integration tests/integration/adk_web/webui_test.ts
 FAIL  |integration| tests/integration/adk_web/webui_test.ts > WebUI Integration Test
Error: webui_test.ts exercises built output; run `npm run build` first. Missing: <repo>/dev/dist/browser/index.html
 ❯ assertRepoIsBuilt tests/integration/adk_web/webui_test.ts:46:11
 Test Files  1 failed (1)
      Tests  2 skipped (2)
   Duration  7.01s (… tests 6ms …)
$ mv dev/dist/esm/cli_entrypoint.js /tmp/cli_entrypoint.js.bak
$ npx vitest run --project integration tests/integration/adk_web/webui_test.ts
 FAIL  |integration| tests/integration/adk_web/webui_test.ts > WebUI Integration Test
Error: webui_test.ts exercises built output; run `npm run build` first. Missing: <repo>/dev/dist/esm/cli_entrypoint.js
 ❯ assertRepoIsBuilt tests/integration/adk_web/webui_test.ts:46:11
 Test Files  1 failed (1)
      Tests  2 skipped (2)
   Duration  7.12s (… tests 4ms …)

Note tests 6ms / tests 4ms: the guard short-circuits before any server
starts. The remaining wall time is vitest transform/collect.

Control runs — the same two broken states with the guard reverted
(git stash of this diff), confirming the guard genuinely replaces the
misleading output and that these tests are not passing for free:

# dev/dist/browser missing, guard reverted
 FAIL  … > 'Run from ADK CLI' > should load the WebUI correctly …
 FAIL  … > 'Using ADK API server' > should load the WebUI correctly …
AssertionError: expected 404 to be 200 // Object.is equality
      Tests  2 failed (2)

# dev/dist/esm/cli_entrypoint.js missing, guard reverted
 FAIL  … > 'Run from ADK CLI'
Error: CLI exited prematurely with code 1
      Tests  1 passed | 1 skipped (2)

Artifacts were restored after each case and the suite re-run green
(2 passed); git status shows only the single intended file modified.

Local validation of the exact pushed commit (this PR is stacked, so
validation.yamlpull_request: branches: [main] — does not trigger and
no CI test job runs on it; it will run once rebased onto main after #359
merges):

$ npm run build      # OK, incl. adk-web v1.0.0 asset fetch into dev/dist/browser
$ npm run lint       # exit 0
$ npm run format:check   # "All matched files use Prettier code style!"
$ npx vitest run --project integration tests/integration/adk_web/webui_test.ts   # 2 passed

npm run ts:check is already red on main in this fork (302 pre-existing
errors, e.g. BASE_AGENT_SIGNATURE_SYMBOL mismatches in core/test/a2a/); it
reports 308 on this branch's base both with and without this diff, and zero
of them come from webui_test.ts. This change introduces no new type errors.

Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.

npm install && npm run build

# 1. Built tree: unchanged, 2 passed.
npx vitest run --project integration tests/integration/adk_web/webui_test.ts

# 2. Simulate a failed adk-web asset download (the realistic partial build).
mv dev/dist/browser /tmp/adk-browser-bak
npx vitest run --project integration tests/integration/adk_web/webui_test.ts
#   -> "webui_test.ts exercises built output; run `npm run build` first.
#       Missing: …/dev/dist/browser/index.html"   (not "expected 404 to be 200")
mv /tmp/adk-browser-bak dev/dist/browser

# 3. Simulate a missing CLI entrypoint.
mv dev/dist/esm/cli_entrypoint.js /tmp/cli_entrypoint.js.bak
npx vitest run --project integration tests/integration/adk_web/webui_test.ts
#   -> same sentence, Missing: …/dev/dist/esm/cli_entrypoint.js
#      (not "CLI exited prematurely with code 1")
mv /tmp/cli_entrypoint.js.bak dev/dist/esm/cli_entrypoint.js

Checklist

[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.

The suite serves built output over HTTP: the CLI variant spawns
dev/dist/esm/cli_entrypoint.js and the in-process variant mounts
dev/dist/browser as the /dev-ui static root. On a partially built tree
the failures describe the symptom, not the cause - 'expected 404 to be
200' when the adk-web bundle is absent, 'CLI exited prematurely with
code 1' when the entrypoint is.

Check both artifacts in an outer beforeAll and throw once, naming the
missing paths and the build command.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant