Add env vars that are excluded from the process cache key

[jasonwbarnett]

Closes #23645.

Environment variable values are part of a process's cache key, which is right by default. But some values unavoidably vary per run and cannot affect what a process produces, and today passing one means every run computes a key nothing else can match.

The case that motivated this: buildkite-test-collector is a [pytest11] plugin that auto-loads in every pytest process and reads BUILDKITE_ANALYTICS_TOKEN and BUILDKITE_BUILD_ID straight from os.environ. The only way to get those into the sandbox is [test].extra_env_vars, which puts roughly 25 per-job values (job id, build id, commit, agent id, instance id, step id, …) into every test process's key. No test result is ever reused across builds, and with remote_cache_write = true each run also writes entries under keys that can never be read back. Those values label a report; they cannot change whether a test passes. Pants had no way to say so.

Approach

Process.uncached_env is a second env map that is excluded from the Command proto, and so from the action_digest that keys both the local and remote caches. It is merged into env by bounded::CommandRunner, immediately alongside the existing execution_slot_variable injection — which already works exactly this way, since context.rs wraps bounded::CommandRunner in both remote_cache::CommandRunner and cache::CommandRunner. The TODO above that injection ("they might currently be applied above a cache") describes the property this change relies on deliberately.

Details worth calling out for review:

• The field is #[derivative(PartialEq = "ignore", Hash = "ignore")], so two processes differing only in uncached_env are the same graph node. Without that they would dedupe to separate nodes that both resolve to the same cache key — harmless but wasteful.
• Because it is a separate field from env, exclusion from the Command proto is automatic rather than a filter that a future edit could forget.
• env wins a name collision. An explicitly cache-keyed value is the more specific request, and honouring it keeps the key honest.
• InteractiveProcess.from_process folds the values into env. An interactive process is never cached, so the distinction does not apply, and --debug should behave like a normal run. I only found this because the run_pytest test helper asserts the two paths agree — a good check.

User-facing surface is [test].uncached_env_vars plus an uncached_env_vars field on test targets, threaded through PexProcess/VenvPexProcess and pytest_runner. The field is part of TestMetadata, so targets wanting different values partition into separate batches — a batch runs as one process, so they have to.

Two limits, documented rather than designed around

1. On a cache hit the process does not run, so these values are not observed at all. This cannot be used to make something happen on every run. For result reporting that is a real behaviour change: a cached test contributes no upload. Arguably more honest — the test did not run — but it is a change, and the option help says so.
2. Under remote execution the Command proto is the only channel to the worker, so values excluded from it cannot be delivered. remote::CommandRunner::run warns and drops them. I chose warn over error because the contract is that these values cannot affect the result, so the execution is still correct.

The obvious risk is misuse — a behaviour-affecting variable in the new field yields silently wrong cached results. I have tried to make the option help and the Process docs blunt about that rather than reassuring. If you would rather the footgun were harder to reach (an allowlist, a different name, plugin-API only with no test surface), I am happy to rework it.

Tests

Rust, in process_execution/src/tests.rs:

• process_equality_ignores_uncached_env — identity and hash ignore the field, while the same names/values in env still separate two processes.
• uncached_env_is_absent_from_the_action_digest — adding an uncached_env var, or changing its value, leaves both the command and action digests byte-identical; the same names/values in env change both. The second half is what stops this test passing vacuously if a future change starts folding the field into the request.

Python, in pytest_runner_integration_test.py:

• test_uncached_env_vars — values from both the option and the target field reach the test process, with and without an inline value, and a name in both lists resolves to the cached one.
• Two cases added to the existing test_partition parametrisation: targets differing in uncached_env_vars split into separate batches, and ordering does not affect partitioning.

Ran locally: cargo test -p process_execution (74 passed), cargo clippy -p process_execution -p remote clean, cargo fmt --all, and pants test over pytest_runner_integration_test.py, core/goals/test_test.py and engine/process_test.py (all green), plus pants fmt lint on the touched Python.

Two suites reported failures in my environment that are not related to this change, for the record: three pytest_runner_integration_test cases skip for want of a CPython 3.10 on the box, and two engine/process_test.py cases error at fixture setup with Failed to begin watching the filesystem: Too many open files — this machine's fs.inotify.max_user_instances is 128 and the module builds 33 RuleRunners. Both pass when run in isolation.

Not verified locally: anything requiring a remote-execution server, so the drop-and-warn path in remote::CommandRunner is reasoned-through rather than exercised. Worth a close look in review.

[jasonwbarnett]

Force-pushed: the first version of this branch was accidentally based on a ~200-commit-old main from a stale local checkout, which put the release note in docs/notes/2.28.x.md and targeted the pre-reorganisation src/rust/engine/process_execution/ paths. Rebuilt on current main (2.34.0.dev3): note moved to docs/notes/2.34.x.md, Rust edits moved to src/rust/process_execution/, and get_filtered_environment reworked for the call-by-name rule API it now uses. Re-verified from scratch on the new base — no change to the approach.

[jasonwbarnett]

Verified end-to-end against the real repo this was motivated by, via PANTS_SOURCE (a ~64k-file monorepo pinned to Pants 2.33, with a custom plugin). It boots and loads the plugin fine; the only friction was that repo's [pex-cli].version pin predating 2.34's floor, worked around with --pex-cli-use-unsupported-version=warning.

The option. Same target, BUILDKITE* removed from [test].extra_env_vars and five per-job names declared as uncached_env_vars instead:

Run	BUILDKITE_ANALYTICS_TOKEN, _BUILD_ID, _JOB_ID, _COMMIT	Result
1	fake-token-run-1, build-AAAA, job-1111, commitAAA	ran; buildkite-test-collector reached 401 Unauthorized from analytics-api.buildkite.com, i.e. the token arrived and the upload was attempted
2	all five values changed	cached locally for all 6 test targets
3 (control)	same changed values, but via extra_env_vars	ran — all 6 re-executed

The target field. A probe test asserting os.getenv("VERIFY_TARGET_FIELD") == "value-one", with uncached_env_vars=["VERIFY_TARGET_FIELD"] on the target:

• VERIFY_TARGET_FIELD=value-one → ran, passed (so the field delivers the value).
• VERIFY_TARGET_FIELD=value-two-CHANGED → cached locally, passed.
• Same changed value moved to extra_env_vars → ran, and failed with AssertionError: got 'value-two-CHANGED'.

That last pair is the part I found most convincing: the control's failure message proves the new value genuinely reaches the process, so the cache hit in the uncached_env_vars case is a real hit rather than a no-op. Adding the field to a BUILD file also did not invalidate the target's existing cache entry, which is the behaviour you'd want.

One wrinkle worth knowing about, and an argument for the docs being explicit: under --test-output=all, a cached run replays the captured stdout, so the collector's 401 lines appear again on a cache hit. I initially misread that as a live upload. The run was cached locally for all six targets — nothing executed, nothing uploaded. Anyone adopting this for result reporting will see those replayed lines in CI logs and could easily draw the wrong conclusion about coverage.