Fix: make env-var stubbing hermetic across the test suite (vitest unstubEnvs + ambient-env coverage) - #281
Open
AmaadMartin wants to merge 3 commits into
Open
Fix: make env-var stubbing hermetic across the test suite (vitest unstubEnvs + ambient-env coverage)#281AmaadMartin wants to merge 3 commits into
AmaadMartin wants to merge 3 commits into
Conversation
added 3 commits
July 29, 2026 19:43
vi.stubEnv leaks into every later test unless unstubEnvs is enabled, so a stub set by one test silently changes the environment the next one reads. process.env is process-global, so the blast radius is every test file sharing the worker, not just the file that stubbed. Vitest projects inherit nothing from the root-level config, so both flags have to live in each project's own test block. The two env_stub_hermeticity_test.ts probes pin that placement: they are deliberately order-dependent and fail if the flags are set at the root or dropped from a project.
vertex_ai_utils_test replaced process.env wholesale with a shallow copy, which swaps out Node's env object and cannot interoperate with vitest's automatic unstub. It now uses vi.stubEnv and neutralises the two variables it reads. setup_test and vertex_ai_session_service_test assert negative cases while leaving the ambient OTEL endpoint and express-mode variables in place, so they fail on a machine that exports them; both now stub those to undefined.
Nothing pinned dev/src/cli/cli.ts's DATABASE_URL fallback or dev/src/utils/telemetry_utils.ts's four OTEL_EXPORTER_OTLP_* reads, so a developer or CI runner exporting either silently got a different session service or a different telemetry branch and the suite stayed green. Both suites stub the variables they read, including to undefined for the not-set cases, so they produce the same result on a clean machine and a polluted one.
This was referenced Jul 30, 2026
Open
This was referenced Aug 7, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Please ensure you have read the contribution guide before creating a pull request.
Link to Issue or Description of Change
Closes: #issue_number
Related: #issue_number
Problem: Test results depend on which environment variables happen to be exported on the machine running them, and on which test ran before them.
vi.stubEnvleaks between tests. Vitest'sunstubEnvsdefaults tofalse, so a value set withvi.stubEnvstays applied toprocess.envfor every subsequent test until someone callsvi.unstubAllEnvs()by hand.core/test/code_executors/agent_engine_sandbox_code_executor_test.tsstubsGOOGLE_CLOUD_PROJECT=''mid-file and never unstubs. It does not fail today, but the next test added after it that expectsGOOGLE_CLOUD_PROJECTto be'test-project'will pass or fail purely based on its position in the file.process.envis process-global, so the blast radius is every file sharing the worker.core/test/utils/vertex_ai_utils_test.tshand-rolledprocess.envsave/restore, replacing the whole object with a shallow copy inbeforeEachand restoring the reference inafterEach. Reassigningprocess.envswaps out Node's special env object for a plain one, and cannot interoperate withvi.unstubAllEnvs(), which writes restored values onto whateverprocess.envpoints at when it runs.dev/src/cli/cli.ts:62'sDATABASE_URLfallback ordev/src/utils/telemetry_utils.ts:129-134's fourOTEL_EXPORTER_OTLP_*reads. A developer or CI runner exportingDATABASE_URLsilently gets aDatabaseSessionServiceinstead of anInMemorySessionServicefromadk web/api_server/run; one exporting anyOTEL_EXPORTER_OTLP_*silently switchessetupTelemetry()onto its env-driven branch. The suite stays green either way.Solution: Make the harness guarantee env isolation instead of relying on every author remembering an
afterEach. Zero production-code changes — the diff isvitest.config.tsplus files undercore/test/anddev/test/.vitest.config.ts: addunstubEnvs: trueandunstubGlobals: trueto each of the six inline projects. They are deliberately not at the root: Vitest's Test Projects guide states "None of the configuration options are inherited from the root-level config file", and inline project entries default toextends: false, so a root placement is a silent no-op.extends: truewas rejected because it would also pull the rootpoolOptions,globalSetupandcoverageinto every project. This is verified empirically by mutation M4 below.core/test/utils/{env_stub_hermeticity_test.ts},dev/test/utils/{env_stub_hermeticity_test.ts}(new): two ordered probe tests per project that pin the flags. Two near-identical files is intentional — the non-inheritance above means each project needs its own probe.core/test/utils/vertex_ai_utils_test.ts: migrated offprocess.envmutation tovi.stubEnv, with abeforeEachthat neutralisesGOOGLE_GENAI_USE_VERTEXAIandGOOGLE_API_KEY. No manualvi.unstubAllEnvs()— that is now the config's job, and its absence is part of what proves the config works. All eight existing cases are unchanged in intent.core/test/telemetry/setup_test.ts: replaced the now-redundantvi.unstubAllEnvs()with explicit neutralisation of all fourOTEL_EXPORTER_OTLP_*variables. The old call cleared stubs; the new loop neutralises ambient values, which is the actual defect.core/test/sessions/vertex_ai_session_service_test.ts: same class of defect, found by running the suite with a polluted environment (see "Manual E2E Tests"). Express mode resolves a key from the ambient env, which stops the constructor throwing, so'throws an error if no client and no project/location provided'fails on a machine exportingGOOGLE_GENAI_USE_VERTEXAI+GOOGLE_API_KEY. Now stubs both toundefined.dev/test/cli/cli_test.ts,dev/test/utils/telemetry_utils_test.ts(new): cover the two previously untested ambient reads, stubbing the variables explicitly (including toundefinedfor the "not set" cases).Collision check (run before implementing, per contribution hygiene):
gh pr list --state open --limit 100plus agh pr diff --name-onlyscan of all 100 open PRs. No open PR addsunstubEnvs/unstubGlobalsor migrates env stubbing, so this does not duplicate live work. Files that merely overlap, none of which implements this change:vitest.config.tsin #261 (hoists the duplicatedaliasmap), #237 (adds anintegration:slowproject) and #247 (addsserver.deps.external);core/test/utils/vertex_ai_utils_test.tsin #227, #265 and #268 (all adding new cases for other features);dev/test/cli/cli_create_test.tshermeticity in #259 and #203. Because three mutually-conflicting siblings touchvitest.config.ts, this branches frommainrather than stacking on any one of them; the conflicts are textual and trivial in every direction.dev/test/cli/cli_create.tsand its test are deliberately not touched — #259/#203 own them.Intentionally not done: only
unit:coreandunit:devget probe files, though all six projects get the flags. The other four projects contain novi.stubEnv/vi.stubGlobalcall at all (verified by grep), so a probe there would pin nothing anyone can break. Note also that the probes rely on default sequential ordering: they would pass vacuously under-tfiltering orsequence.shuffle.Testing Plan
Please describe the tests that you ran to verify your changes. This is required for all PRs that are not small documentation or typo fixes.
Unit Tests:
[x] I have added or updated unit tests for my change.
[x] All unit tests pass locally.
npx vitest run --project unit:core --project unit:dev— 176 files / 2429 tests pass, up from 2411 on the base commit (+18 new). The single failure,dev/test/cli/cli_create_test.ts > should handle Vertex AI selection with gcloud defaults, is pre-existing on the base commit and reproduces identically there; it is the ambient-GOOGLE_CLOUD_PROJECTdefect owned by the concurrentcli_create_testtask, and this machine exports that variable.Also clean on the exact pushed commit:
npm run build,npm run lint,npm run format:check.tsc --noEmitreports 308 errors before and after, byte-identical once line numbers are normalised — all pre-existing; this branch adds none.Coverage.
dev/src/utils/telemetry_utils.tsreaches 100% branch coverage and all four functions this PR targets (otelEnvVarsEnabled,setupTelemetry,setupGcpTelemetryExperimental,setupTelemetryFromEnvExperimental, lines 128-190) are fully covered. File-level statements read 51.07% because the file also containshrTimeToNanoseconds,ApiServerSpanExporterandInMemoryExporter(lines 26-125) — pre-existing untested code that this PR does not touch and deliberately does not pad tests for.Proof that each new test can fail. Every mutation below was applied, run, and reverted; the working tree is clean and this PR contains no
srcchanges.unstubEnvs: truefrom theunit:coreprojectcore/.../env_stub_hermeticity_test.ts > does not inherit stubs from the previous testFAILS —AssertionError: expected 'stubbed' to be undefinedunstubEnvs: truefrom theunit:devprojectdev/.../env_stub_hermeticity_test.tssame test FAILS —AssertionError: expected 'stubbed' to be undefinedunstubGlobals: truefrom theunit:coreprojectAssertionError: expected true to be false // Object.is equalitytestblock instead of the projectsAssertionError: expected 'stubbed' to be undefined. This is the empirical proof that a root placement is a no-opcli.ts:62: drop|| process.env.DATABASE_URLexpected InMemorySessionService{…} to be an instance of DatabaseSessionService(web, api_server, run)cli.ts:62:process.env.DATABASE_URL || options[...]should prefer --session_service_uri over DATABASE_URLFAILS —expected DatabaseSessionService{…} to be an instance of InMemorySessionServicetelemetry_utils.ts: deleteOTEL_EXPORTER_OTLP_METRICS_ENDPOINTfromendpointVarsshould add no hooks of its own when OTEL_EXPORTER_OTLP_METRICS_ENDPOINT is setFAILS —expected "spy" to be called with arguments: [ [] ]telemetry_utils.ts: invert theotelToCloud/otelEnvVarsEnabled()precedenceshould prefer the GCP branch over the env branchFAILS —expected "spy" to be called with arguments: [ { enableTracing: true, …(2) } ]setup_test.tsOTEL neutralisation, withOTEL_EXPORTER_OTLP_ENDPOINTexportedexpected "setGlobalTracerProvider" to not be called at all, but actually been called 1 timesvertex_ai_session_service_test.tsstubs, withGOOGLE_GENAI_USE_VERTEXAI+GOOGLE_API_KEYexportedthrows an error if no client and no project/location providedFAILS —expected [Function] to throw error including 'Either (Project ID and Location) or a…' but got 'Authentication is not set up. Please …'One negative result, reported rather than hidden: reverting the
vertex_ai_utils_test.tsmigration under the same polluted environment does not fail — the old file happened to be ambient-safe because each casedeleted the variable it read. That change is therefore a migration off theprocess.env-reassignment anti-pattern (which cannot interoperate with the now-global unstub), not a defect fix, and its proof is only that all eight cases still pass. It is not claimed as a bug fix.Manual End-to-End (E2E) Tests:
Please provide instructions on how to manually test your changes, including any necessary setup or configuration.
The point of the change is that a polluted environment no longer changes the result. From the repo root:
Both report
Test Files 1 failed | 176 passed (177)/Tests 1 failed | 2429 passed (2430)— identical, with the same single pre-existingcli_create_test.tsfailure described above.On the base commit, run 2 fails two extra suites that run 1 passes (
core/test/telemetry/setup_test.ts, 4 cases, andcore/test/sessions/vertex_ai_session_service_test.ts, 1 case) — that divergence is the defect this PR removes.--project integration --project e2ewas also exercised, but it is not a usable signal on a machine without live credentials: three consecutive runs of the same tree produced 22, 24 and 38 failures with 50, 46 and 24 skips, i.e. it is nondeterministic here. What can be stated deterministically is that no file undertests/orintegrations/test/callsvi.stubEnvorvi.stubGlobalat all (grep-verified), sovi.unstubAllEnvs()/vi.unstubAllGlobals()operate on an empty stub map there and are provable no-ops. Theprocess.envreads in those trees are top-leveldescribe.skipIf/it.skipIfguards evaluated at collection time, before any unstub runs.Flipping the flags on cannot break an existing stub: the automatic unstub runs in
onBeforeTryTask, which@vitest/runnerinvokes before a suite'sbeforeEachhooks, and all eleven files callingvi.stubEnv/vi.stubGlobalinstall their stubs inbeforeEach(or in the test body) — none inbeforeAll. That ordering is also whycore/test/code_executors/agent_engine_sandbox_code_executor_test.tskeeps working with no change.Checklist
[x] I have read the CONTRIBUTING.md document.
[x] I have performed a self-review of my own code.
[x] I have commented my code, particularly in hard-to-understand areas.
[x] I have added tests that prove my fix is effective or that my feature works.
[x] New and existing unit tests pass locally with my changes.