Add pre-release E2E CI: full-path release gate with layered per-leg judging - #1303

overlay2 fixed the docker-host eviction (200G plateau vs the 1792Gi limit, pod
alive well past the ~4min eviction point) and the stall fix kept the sglang legs
alive through a 5min source build. Run 1.0.1a0.dev202608280519+ci then exposed the
next layer: most legs never launched `optimize` at all.

`claude --print` is one turn, and the agent sometimes ends it with a progress note
instead of finishing the job. Live transcripts:

  baremetal-vllm-3h  setup: "Base tooling installed. Waiting on the vLLM ROCm wheel
                            install."          (never printed `setup complete: ...`)
                     demo:  "Good progress -- torch is in ... Waiting on the monitor."
  baremetal-vllm-12h demo:  "Install step started. Waiting for it to complete."

Everything the agent leaves running is a child of that turn, so it dies with it:
both legs were left with no claude and no optimize process, and the stall check
correctly reaped them 600s later. This is why the previous run's vllm legs passed
and this one's did not -- there the demo turn happened to run 16min and got as far
as `setsid nohup optimize`; here it returned after 2-4min. Non-deterministic agent
behaviour, so the harness has to converge it rather than hope.

Setup turn: the prompt's contract is a literal `setup complete: <mode>/<backend>`
line, so grep for it and re-drive the prompt (LEG_TURN_ATTEMPTS, default 3) until
the agent reports it, failing the leg only after that.

Demo turn: re-drive off the EXISTING stall signal instead of probing "is it
launched" right after the turn. The demo skill's launcher runs install.sh before
backgrounding optimize, so state.json can legitimately be 10+ minutes out and a
short-grace probe would double-launch; whereas "no state.json and nothing written
under the leg root for 600s" already means nothing is running. On that signal, ask
the agent to finish the job (LEG_DEMO_REDRIVES, default 2) before giving up.

Split the wait loop's clocks so the re-drive cannot extend the pod deadline:
start_ts still backs deadline_s and is never reassigned, while the new grace_ts
backs the stall window and restarts after each re-driven turn. The invariant
bootstrap deadline < SaFE pod timeout < poll GLOBAL_TIMEOUT_S is unchanged.

Both demo prompts now state that the turn is single and non-interactive, that
ending it on "install started" / "waiting on the monitor" kills the work, that
optimize must be detached with setsid nohup, and that the run must be confirmed
live (nested run dir + state.json + live PID) before the turn ends.

Tests: leg_run_started is exercised against a real session tree (a stray
state.json at the session top must not count as a launch), the setup marker is
cross-checked against all four setup prompts so the grep and the prompts cannot
drift, and the deadline clock is pinned against being reset by a re-drive.

Co-authored-by: Cursor <cursoragent@cursor.com>

… drop preempt

Three things, all from run 1.0.1a0.dev202608280605+ci.

1. Setup was budgeted in TURNS, and that was my regression. `claude --print` is
"print response and exit", and a framework install (vLLM ROCm wheel into an
isolated venv, or SGLang compiled from source for gfx950) runs 10-30min -- far
longer than one turn holds. So every agent backgrounds the install, polls its
log, and eventually answers with a progress note, which ends the turn with setup
incomplete. A cap of 3 turns amounted to ~4min of wall clock and killed two legs
whose installs were provably still going:

  baremetal-vllm-3h    died 3.5min in, mid "Phase 2 (isolated vLLM install) is underway"
  baremetal-sglang-12h died 5.7min in, mid "Still waiting on the kernel build"

Budget it in time instead: LEG_SETUP_DEADLINE_S (45m) bounds the whole step, and
the existing liveness signal decides failure -- a leg dies only once nothing has
been written under the leg root for LEG_SETUP_STALL_S (10m), i.e. nothing is
installing. LEG_SETUP_MAX_TURNS stays purely as a token-spend backstop.

2. Follow-up turns now RESUME the leg's conversation instead of re-feeding the
prompt as a fresh turn. Re-feeding threw away everything the agent already knew,
leaving it to rediscover which install it had launched and which log it was
watching. `--session-id` opens a stable per-leg UUID (sha1 of leg+CI_VERSION,
shaped into a v4 UUID -- distinct per leg, which matters on the docker host where
four agents share one pod) and `--resume` continues it with a one-line nudge. The
demo turn resumes the same session too, exactly as a human would keep one chat.
Resume failure degrades to a standalone turn rather than failing the leg.

That the detached install survives a turn boundary is verified: setup_vllm.log
grew 71KB -> 77KB across one, and baremetal-vllm-12h converged to
`setup complete: baremetal/vllm` on turn 3 and went on to launch optimize.

3. Remove the `preempt` job and pre-release-e2e-reap.sh. It could never work. A
newer run cannot tear down an older one: while the concurrency group is held,
GitHub keeps the new run at run-level `pending` with an EMPTY jobs array (run
33145644120 sat pending 20min with zero jobs), so no job of it exists to reclaim
anything. And it ran on a GitHub-hosted runner while SAFE_API_BASE is an
in-network NodePort, so every run logged `[preempt] could not list workloads;
skipping reclaim` after a 30s curl timeout, having stopped nothing -- three runs
checked, all identical. Keeping it only made the gate look protected.

Teardown runs the other way round instead: poll fails fast on the first FAIL
(POLL_FAIL_FAST, default on). It is a release gate -- the first FAIL already
blocks the release, so waiting the rest out buys nothing while costing a 12h
leg's GPUs and the only self-hosted runner, which is what left the next fix's run
stuck at `pending` until the old run was cancelled by hand. Remaining legs are
recorded as not judged rather than FAIL. Poll also sleeps in POLL_SLEEP_SLICE_S
slices so a cancel lands in seconds instead of at the end of a full interval.
Leaked pods stay covered by the dispatch-side reap, which -- unlike preempt --
runs in-network and only executes once this run holds the runner, i.e. once the
previous run is provably done.

Tests: setup budget must not be turn-based, follow-ups must resume, the session
uuid is checked for shape/stability/uniqueness, all agent invocations must go
through the single teeing helper, and the orchestration guard pins that no job
touching SaFE runs on a GitHub-hosted runner and that fail-fast is armed on every
FAIL path.

Co-authored-by: Cursor <cursoragent@cursor.com>

…ory to 512Gi

Fail-fast still marks the gate FAIL and releases the runner, but workloads
that were still optimizing are left alive for post-mortem. Dispatch reap on
the next run remains the backstop. Baremetal legs now request 512Gi memory
and ephemeral storage after sglang-12h OOM at 128Gi.

Co-authored-by: Cursor <cursoragent@cursor.com>

…vior

Reformat test_pre_release_gate_orchestration and test_pre_release_stall_liveness
so ruff format --check passes in CI.

Co-authored-by: Cursor <cursoragent@cursor.com>

VERSION_TAG now hashes CI_VERSION with GITHUB_RUN_ID so dispatch reap can
stop pods from a superseded push even when the wheel version is unchanged.
Poll detects a newer queued workflow run for the same PR branch, releases
the runner without stopping workloads, and leaves teardown to the successor.

Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

Align poll gate with optimize CLI exit 0: final.json must carry a terminal
stop_reason in the success set (time_exhausted, target_reached, etc.).
TARGET_GAIN still flows to the demo skill at 100 but no longer gates PASS.

Co-authored-by: Cursor <cursoragent@cursor.com>

Set nested container caps to 256g/512g for 3h/12h legs (64g shm) to avoid
OOM on 14B-FP8 docker paths, and bump the privileged docker-host CPU request
to 196 for four parallel agent/setup processes atop 4x32 container caps.

Co-authored-by: Cursor <cursoragent@cursor.com>

Judge PASS/FAIL from state.json stop_reason instead of final.json, and
complete bootstrap when optimize exits cleanly even if final.json is late.
Sync 3h to framework-only (no --no-framework-agent, 0.90 framework pct),
align 12h prompts with the 14b skill, and raise demo re-drive budget to 5.

Co-authored-by: Cursor <cursoragent@cursor.com>

Mirror the 3h skill launch requirements: spell out phase budget flags and
forbid --no-framework-agent / --no-kernel. Keep target-gain at 50 for users.

Co-authored-by: Cursor <cursoragent@cursor.com>

Queue the privileged docker host first so dockerd startup and image pulls
overlap with baremetal scheduling instead of starting after four 1-GPU pods.

Co-authored-by: Cursor <cursoragent@cursor.com>

Read root-only state.json via sudo on the baremetal runner, publish readable
permissions from bootstrap, continue per-leg polling after gate FAIL, and use
distinct report icons for SKIP vs FAIL.

Co-authored-by: Cursor <cursoragent@cursor.com>

Document that legs with an empty stop_reason rely on GLOBAL_TIMEOUT_S
rather than an NFS stall check.

Co-authored-by: Cursor <cursoragent@cursor.com>

Normalize quote style so ruff format --check passes in CI.

Co-authored-by: Cursor <cursoragent@cursor.com>

Co-authored-by: Cursor <cursoragent@cursor.com>

…ale artifacts

A reused CI_VERSION (reuse_ci_version, or a job re-run) puts a run on the paths a
finished run already wrote, and nothing proved which run an artifact belonged to.
Verdicts are recorded once and the poll loop breaks as soon as nothing is pending,
so one stale read on the first tick declared the whole gate PASS before any pod had
booted; bootstrap likewise saw an old "setup complete" in the appended agent log and
an old state.json as its own.

Dispatch now hands its VERSION_TAG to the pods and to the poll, bootstrap stamps it
into .session_dir, and the poll ignores a pin carrying any other tag. The wait loop
only considers a state.json newer than the leg start, and the agent transcript is
rotated instead of appended -- both keep the previous run's files for post-mortem.

The pod hard-timeout counted only the demo wait, leaving it 15m BELOW what bootstrap
can spend once the 45m setup budget is included: a leg using its full setup budget
was killed by SaFE mid-wait, losing the clean return and the logs. Size both pod
deadlines off setup + demo wait + margin, raise the poll global timeout above them,
and pin the whole ladder in a test so it cannot drift again.

Co-authored-by: Cursor <cursoragent@cursor.com>

…erstating the gate

Three claims the code did not back, and one asymmetry that cost a leg.

The first setup turn was called bare while every later one is wrapped, so under
set -e a single transient CLI error (a 429, a dropped connection) killed a
multi-hour leg outright -- the failure mode the retry loop exists to absorb.
Guard it like the others and let the stall check and the setup deadline decide.

MAX_BOOT_FAILS was declared and documented as a tolerance, but nothing ever read
it: the boot-failure count lives in the report journal, not state.json. Drop the
knob rather than leave a criterion the gate does not enforce.

Three comments promised the API key never reaches NFS. It does -- the leg's .env
sits beside the workspace the agent reads it from, and the EXIT trap that scrubs
it cannot run on a SIGKILL. Say that, so the next reader knows it still needs
hardening instead of trusting a guarantee that was never true.

Declare hyperloom-pre-e2e-baremetal in actionlint.yaml, which already claimed to
cover this workflow; undeclared, it fails pre-commit on unrelated work.

Co-authored-by: Cursor <cursoragent@cursor.com>

…ace -k

The demo prompts stated that the 100% target is not the gate but never why it is
set out of reach: to stop the loop converging on the skill's own target so the
full phase sequence runs. Without that a reader takes it for a performance goal.

The scripts have always honoured SAFE_CACERT ahead of skip-verify, matching
ci-e2e and forge-e2e, but the workflow wired no variable into it -- so the only
reachable mode was SAFE_INSECURE=1, sending an admin token over an unverified
connection with no way to opt out. Inject PRE_E2E_SAFE_CACERT and document it.
Unset, behaviour is unchanged.

Co-authored-by: Cursor <cursoragent@cursor.com>

pyproject.toml has to stay in on.pull_request.paths, or a PR that only bumps the
version would never start the workflow. The scope decision then read "no version
bump" as "the CI's own logic changed" -- but a plain dependency or tooling edit
satisfies that same path filter, so it took the scripts-only branch and spent
four 3h legs on a change with no bearing on the release path.

Decide on the CI paths instead. The file list was already being computed for the
log line; use it, and run nothing when it comes back empty. This makes the
run=false output reachable for the first time: resolve still reports, while build
and run skip on their existing conditions.

The classification had no test despite being described as one -- add it, running
the real decide step over a scratch git repo across five scenarios: version bump,
CI-path change, dependency-only edit, dependency edit alongside a CI change, and
manual dispatch.

Co-authored-by: Cursor <cursoragent@cursor.com>

…ng for 14h

workload_phase sent curl's stderr to /dev/null and swallowed its exit status, so
jq read an empty body and produced an empty phase. Empty is not one of the
terminal phases, so every leg stayed PENDING and the poll held 8 GPUs and the only
self-hosted runner until GLOBAL_TIMEOUT_S -- with nothing in the log to say the API
had refused the connection, rejected the token, or gone away.

It now returns __APIERR__ with the HTTP status and keeps curl's diagnosis, which
the loop reports once per tick rather than once per leg. After API_FAIL_ABORT
consecutive polls in which EVERY query failed (~20min by default) it stops waiting
and fails the unjudged legs with that reason. Only a total outage counts: one
workload disappearing must not end a run whose other legs are progressing, and
state.json still comes off NFS, so a leg that exits cleanly is judged either way.
What an unreachable API costs is telling a zombie leg from a slow one, which is
the only thing the remaining wait was for.

Co-authored-by: Cursor <cursoragent@cursor.com>

…p the map

DOCKER_GPU_MAP travelled to the host pod alongside DOCKER_LEGS carrying a
leg->index mapping. Being a second copy of an ordering the list already has, the
only thing it could contribute was disagreeing with it: `jq -r '.[$l]'` answers
the string "null" for a missing key, arithmetic reads that as 0, and two legs bind
renderD128 and the same card. Both lists were built in one loop, so nothing in
this repo could trigger it -- but the map bought no safety for that risk either.

The host now numbers DOCKER_LEGS as it walks it, which is the assignment. Dispatch
numbers the same list for its summary line and cannot drift, because it is the
same list. A leg name that is not in it yields empty rather than 0, and run_leg's
existing GPU_INDEX guard still rejects that.

Assignment is now positional rather than fixed per leg name, so a subset run packs
onto the low cards instead of leaving gaps; each leg's card is logged on launch.

Co-authored-by: Cursor <cursoragent@cursor.com>

GH_STATUS_SHA came only from the pull_request event, so on workflow_dispatch it was
empty and gh_status_on / gh_report_on returned early without logging: no per-leg
pre-release-e2e/<leg> statuses, no sticky report, nothing saying why. Manual
dispatch is the entry point that carries reuse_ci_version and a leg subset, so the
run most likely to happen at release time was the one with no visibility.

Fall back to github.sha, the dispatched ref's real commit. The pull_request path is
unchanged -- statuses still land on the PR head, not the ephemeral merge commit --
and a populated SHA also revives the poll's commit->PR lookup, which was dead code
in exactly the case it was written for.

Co-authored-by: Cursor <cursoragent@cursor.com>

reuse_ci_version skips the build job that publishes the wheel, the bootstrap script
and the prompts, and nothing on the CI side looked at whether they were there. A
wrong version string or a cleaned-up NFS dir was therefore discovered only inside
the pods, and only after each had been scheduled, pulled a ROCm image, apt-installed
jq and npm-installed the Claude CLI -- the wheel check sits behind all of that in
bootstrap's entry. All eight legs then reported the same "state.json missing
(workload phase=Failed)", which names none of it, and SaFE deletes a failed leg's
pod before its stdout can be read.

Stat the four things the pods need before dispatch and fail with the list of what
is absent. It costs nothing and no GPU is claimed.

Also name the provenance on a reused version: skipping build means the pods run the
bootstrap and prompts staged by THAT build, not this branch's. Reusing a wheel to
retest a script change would otherwise pass silently on the old script -- a quiet
wrong answer rather than a loud failure. The step now prints the commit and run id
the staged copies came from.

Co-authored-by: Cursor <cursoragent@cursor.com>