Skip to content

Fix: scrub ADK environment variables from unit test runs - #302

Open
AmaadMartin wants to merge 2 commits into
mainfrom
fix/hermetic-unit-test-env
Open

Fix: scrub ADK environment variables from unit test runs#302
AmaadMartin wants to merge 2 commits into
mainfrom
fix/hermetic-unit-test-env

Conversation

@AmaadMartin

@AmaadMartin AmaadMartin commented Jul 30, 2026

Copy link
Copy Markdown
Owner

Please ensure you have read the contribution guide before creating a pull request.

Link to Issue or Description of Change

  1. Link to an existing issue (if applicable):
    N/A — no existing issue.

  2. Or, if no issue exists, describe the change:

Problem: a unit test must produce the same result on a contributor's laptop and in CI, and today it does not. The three unit projects run in a process that inherits the developer's shell, and a lot of ADK production code reads configuration straight from process.env. A contributor who has GOOGLE_CLOUD_PROJECT, DATABASE_URL, GEMINI_API_KEY or an OpenTelemetry endpoint exported — the normal state for anyone who also uses ADK — is running a different suite from the one CI runs. GitHub Actions runners export none of these, which is exactly why this class of bug is invisible in CI.

Two reproductions on main, both green in CI:

# 1. providers get installed from the ambient endpoint, so three
#    not.toHaveBeenCalled() assertions blow up
OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 \
  npx vitest run --project unit:core core/test/telemetry/setup_test.ts   # 4 failed / 2 passed

# 2. every `web` / `api-server` case throws
#    "Unsupported session service URI: bogus://nope"
DATABASE_URL=bogus://nope \
  npx vitest run --project unit:dev dev/test/cli/cli_test.ts             # 11 failed / 13 passed

Solution: a test-only backstop. tests/unit_setup.ts is wired as setupFiles into the three unit:* projects; it deletes the ADK-relevant variables from the worker's environment before the test file is imported, and writes them back in an afterAll. The policy is a unit test that needs a value sets it explicitly.

Two exported constants, because one of the two rules cannot be a literal list:

  • SCRUBBED_ENV_VARS — the 13 names that core/src, dev/src or integrations/src read directly.
  • SCRUBBED_ENV_PREFIXES = ['ADK_']core/src/features/feature_registry.ts composes ADK_ENABLE_${featureName} / ADK_DISABLE_${featureName} from an enum that grows, so the ADK family cannot be enumerated. Every ADK-owned variable is ADK_-prefixed, so the prefix is the accurate invariant. No speculative extra prefixes (GOOGLE_, OTEL_) were added.

Design points worth calling out:

  • Module scope, not beforeEach. This is load-bearing. The scrub has to land before the test file's module body runs, because several test files capture process.env at module or describe scope and restore it later. More concretely, a top-level beforeEach would run after core/test/models/apigee_llm_test.ts's beforeAll, which sets APIGEE_PROXY_URL, GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION and relies on them — it would delete the values that suite needs. Mutation 2 below demonstrates exactly that breakage, so the design is pinned by a test rather than by a comment.
  • setupFiles, not globalSetup. globalSetup runs in the Vitest main process and cannot touch a worker's process.env; it is also run-wide, so scrubbing there would hit e2e, where ambient credentials are how a developer supplies real ones. tests/global_setup.ts is untouched.
  • Only the unit:* blocks are wired. integration, e2e and cross-language are unchanged and keep the ambient environment. grep -n setupFiles vitest.config.ts shows exactly three occurrences of UNIT_SETUP_FILE, all inside unit:* blocks.
  • A plain delete, not vi.stubEnv. A stub is undone by any test calling vi.unstubAllEnvs() (or by the unstubEnvs: true option under review in Fix: make env-var stubbing hermetic across the test suite (vitest unstubEnvs + ambient-env coverage) #281), which would resurrect the ambient value. A delete is not.
  • GOOGLE_APPLICATION_CREDENTIALS is deliberately not scrubbed. An earlier revision of this branch included it. It fails the invariant that every listed name has a reader — grep -r GOOGLE_APPLICATION_CREDENTIALS core/src dev/src integrations/src returns nothing — and deleting it would not make auth hermetic anyway, since Application Default Credentials also resolve via the gcloud well-known file and the metadata server. A unit test that reaches real auth is broken for a different reason.
  • No production code changes, no dependency added.

Known remaining gap (not fixable here, stated rather than hidden): dev/test/cli/cli_test.ts imports dev/src/cli/cli.ts, whose module scope calls dotenv.config({quiet: true}). That import happens after setup files have run, so a root .env would repopulate scrubbed variables for that one file — and because dotenv does not override already-set variables, the scrub makes a root .env more likely to apply there than it is today. There is no .env in the repo (it is gitignored) and no other unit test imports a module that calls dotenv.config(). A separate in-flight change removes that call; this one cannot.

Collision check. gh pr list --repo AmaadMartin/adk-js --state open --limit 1000 (a --limit 100 scan silently truncates against the ~380 open PRs on this fork), then gh pr diff --name-only on every plausibly adjacent PR:

I branched from main rather than stacking: the overlapping PRs have no single correct base, and a stacked base would suppress CI (the workflow triggers on pull_request: branches: [main]).

Rebased onto current main. The previous head of this branch conflicted with main in vitest.config.ts (main added INTEGRATION_HOOK_TIMEOUT_MS / INTEGRATION_TEST_TIMEOUT_MS in the same place this change adds UNIT_SETUP_FILE), and GitHub could not build a merge ref, so no CI ran at all on that commit. Rebasing resolves it by keeping both constant blocks; everything reported below was re-measured on the rebased tree.

Testing Plan

Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.

Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.

Three near-identical test files — core/test/unit_setup_test.ts, dev/test/unit_setup_test.ts, integrations/test/unit_setup_test.ts. The duplication is intentional: setupFiles is configured per Vitest project, so each project needs its own test to prove its own wiring. They are deliberately not factored into a shared helper; the indirection would obscure what is being proven. The core one adds two cases beyond the wiring assertions: that the scrub does not empty the environment, and that a value the test file sets itself survives — the assertion that pins the module-scope design.

Proof each test can fail. Four mutations, each run against the real suite:

# Mutation Result
1 delete setupFiles from the unit:core block, dirty shell core/test/unit_setup_test.ts still passes 4/4 — see the honest note below. core/test/telemetry/setup_test.ts, which does not import the setup module, fails 4 / 2
2 module-scope scrub → top-level beforeEach a value set by the test file › survives the scrub FAILS (expected undefined to be 'explicit-project') and ApigeeLlm LLMRegistry integration › ApigeeLlm is registered by default FAILS (Proxy URL must be provided via the constructor or APIGEE_PROXY_URL environment variable.) — exactly the breakage the module-scope design exists to avoid
3 SCRUBBED_ENV_PREFIXES = [] tracing_test.ts under ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false FAILS 3 / 3
4 drop DATABASE_URL and OTEL_EXPORTER_OTLP_ENDPOINT from the list reproduction 1 FAILS 4 / 2 and reproduction 2 FAILS 11 / 13 — i.e. the exact regression this revision fixes

Honest note on mutation 1. The design expected the unit_setup_test.ts assertions to fail when the wiring is removed. They do not, and the reason is structural: those tests import tests/unit_setup.js to get the constants, and that import evaluates the module, which performs the scrub. So the file scrubs its own environment even with setupFiles deleted (setup 0ms in the run output confirms the setup file did not run). The wiring is therefore genuinely pinned by mutation 1's second half, plus 3 and 4 — all of which use test files that do not import the setup module — while the three unit_setup_test.ts files pin the scrub policy. Flagging this rather than reporting a mutation result I did not observe.

Coverage. coverage.include is core/src/**, dev/src/**, integrations/src/**; everything added here lives under tests/ and */test/, so it contributes no lines to the report and cannot move the thresholds (unchanged at statements 86 / branches 87 / functions 88 / lines 86). Every branch of the scrub policy is asserted behaviourally instead: the list rule, the prefix rule, the narrowness of the scrub, and "explicit set wins".

The afterAll restore is verified by probe, not by assertion. Its only observable moment is after the last hook of a test file has run, and hooks are 'stack'-ordered, so a test-file afterAll always runs before it — no in-suite assertion can see it. Instead I temporarily registered a probe hook ahead of the restore (so reverse ordering put it after) that recorded the environment it saw, and ran dev/test/unit_setup_test.ts with GOOGLE_CLOUD_PROJECT=ambient-proj ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false exported. It recorded GOOGLE_CLOUD_PROJECT=ambient-proj ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false, confirming both a listed variable and a prefix-matched one are written back. The probe was removed and is not part of this PR.

Manual End-to-End (E2E) Tests:

The deliverable is suite-wide hermeticity, so the verification is the whole unit suite under both environments. After npm install:

# clean baseline
npx vitest run --project unit:core --project unit:dev --project unit:integrations

# poisoned shell
GOOGLE_CLOUD_PROJECT=bogus-project GOOGLE_CLOUD_LOCATION=bogus-location \
GOOGLE_GENAI_USE_VERTEXAI=true GEMINI_API_KEY=bogus-key \
DATABASE_URL=bogus://nope OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 \
ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false ADK_ENABLE_PROGRESSIVE_SSE_STREAMING=true \
  npx vitest run --project unit:core --project unit:dev --project unit:integrations

Both report Test Files 2 failed | 184 passed (186) and Tests 4 failed | 2579 passed (2583) — identical counts and an identical set of failing tests, which is the postcondition this change exists to establish.

Those 4 failures are pre-existing and unrelated: a stale expected version string in integrations/test/version_test.ts, and three unsafe_local_code_executor_test.ts output-file cases. Verified by checking out main with this change absent and reproducing them (Tests 4 failed | 15 passed (19)). They are the subject of #479 and #355 / #516.

Both original reproductions now pass:

OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 \
  npx vitest run --project unit:core core/test/telemetry/setup_test.ts   # 6 passed (was 4 failed / 2 passed)
DATABASE_URL=bogus://nope \
  npx vitest run --project unit:dev dev/test/cli/cli_test.ts             # 24 passed (was 11 failed / 13 passed)
ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false \
  npx vitest run --project unit:core core/test/telemetry/tracing_test.ts # 6 passed (was 3 failed / 3 passed)

Other gates, run on the pushed commit:

  • bash scripts/check_license.shAll files have the correct license header.
  • npm run lint — clean.
  • npm run format:checkAll matched files use Prettier code style!
  • npx tsc --noEmitFound 301 errors in 49 files, identical to main, all pre-existing core/dist/types vs core/src identity mismatches from a stale build; grepping the output for unit_setup or vitest.config returns nothing, so this diff adds none.

No any, no @ts-expect-error, no eslint-disable and no coverage-ignore pragma appears anywhere in this diff.

Checklist

[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.

Amaad Martin added 2 commits August 2, 2026 21:08
The three unit projects run in a process that inherits the developer's
shell, and a lot of ADK production code reads configuration straight
from process.env. A contributor with GOOGLE_CLOUD_PROJECT, DATABASE_URL
or an OpenTelemetry endpoint exported runs a different suite from the
one CI runs, because Actions runners export none of these:

  OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4318 \
    npx vitest run --project unit:core core/test/telemetry/setup_test.ts
  # 4 failed / 2 passed -- maybeSetOtelProviders() installs providers
  # from the ambient endpoint

  DATABASE_URL=bogus://nope \
    npx vitest run --project unit:dev dev/test/cli/cli_test.ts
  # 9 failed / 12 passed -- "Unsupported session service URI"

Wire tests/unit_setup.ts into the three unit:* projects. It deletes the
13 variables that core/src, dev/src or integrations/src read directly,
plus the ADK_ prefix family, then writes them back in an afterAll. The
prefix rule cannot be a literal list: feature_registry.ts composes
ADK_ENABLE_${featureName} from an enum that grows.

The scrub runs at module scope, not in a beforeEach. It has to land
before the test file's module body, and a top-level beforeEach would run
after apigee_llm_test.ts's beforeAll and delete the three variables that
suite sets for itself.

GOOGLE_APPLICATION_CREDENTIALS is deliberately absent: no ADK source
reads it, and deleting it would not make auth hermetic anyway, since
Application Default Credentials also resolve via the gcloud well-known
file and the metadata server.

integration, e2e and cross-language keep the ambient environment, which
is how a developer supplies real credentials to them.
One test file per unit project, because setupFiles is configured per
Vitest project and each project's wiring needs its own proof. The
duplication is deliberate; a shared helper would obscure what is being
proven.

The core file adds the two cases that pin the design rather than the
config: that the scrub does not empty the environment, and that a value
the test file sets for itself survives it. The second fails if anyone
converts the module-scope scrub into a beforeEach.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant