Add pre-release E2E CI: full-path release gate with layered per-leg judging - #1303
Open
lishuoshuo-amd wants to merge 40 commits into
Open
Add pre-release E2E CI: full-path release gate with layered per-leg judging#1303lishuoshuo-amd wants to merge 40 commits into
lishuoshuo-amd wants to merge 40 commits into
Conversation
…udging.
Build the CI wheel, stage it plus the in-pod bootstrap/prompts to NFS, then
run 8 demo legs (baremetal/docker x vLLM/SGLang x 3h/12h) as SaFE Authoring
workloads through the Claude CLI + setup/demo skills -- the same path a user
takes. target_gain=100% is a hard release gate; each leg reports PASS/FAIL on
its own terminal (3h legs surface first) via per-leg commit statuses.
Triggers: a push to main that bumps pyproject's version runs all 8 legs; a
push that only changes this CI's scripts/prompts/workflow (version unchanged)
runs the 4 fast 3h legs to validate the logic change; manual dispatch runs a
chosen subset (default all 8). All environment values are ${{ secrets.* }} /
${{ vars.* }} references -- none are hard-coded.
CI E2E report — ✅ Succeeded
|
The poll step now upserts ONE sticky comment (matched by an HTML marker, same
pattern as ci-e2e-dispatch.sh) and PATCHes it in place as each leg finishes, so
a single comment updates incrementally (design point C). Since this CI runs on
push to main (no PR number), it resolves the PR the triggering commit was merged
from via GET /commits/{sha}/pulls, falling back to a commit comment when none is
found. Widens the workflow token to pull-requests/issues/contents: write.
Dispatch now attaches a pod hard-deadline per duration -- 3h legs get 3+1h
(14400s), 12h legs get 12+1h (46800s). The shared privileged docker host runs a
mix of 3h and 12h nested legs, so its deadline is the MAX over its legs (else a
3h deadline would kill a still-running 12h leg). SaFE terminates the pod at the
deadline; the poll then judges that leg FAIL. The deadline field name defaults
to activeDeadlineSeconds (k8s convention) and is TODO-flagged for the owner to
confirm against the real SaFE Authoring API; DEADLINE_FIELD="" omits it.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The bootstrap script only reads these as plain text for `claude --print`, so the extension was never load-bearing. Markdown fits better -- they are natural-language instructions, and .md renders headings/lists/code blocks and matches the repo's SKILL.md style. Content is semantically unchanged (same flags, overrides, and hard-constraint wording -- this is an automated release gate); only the formatting was reorganized into sections. Updates the two bootstrap references and the build job's cp glob from *.txt to *.md. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Confirmed against the Primus-SaFE codebase that the create-workload body embeds WorkloadSpec inline, whose top-level integer-seconds `timeout` field is enforced by WorkloadTTLController for all workload kinds (incl. Authoring), counted from dispatch time. Replaces the placeholder `activeDeadlineSeconds` default (a k8s-Job convention that is not the Authoring create-API field). DEADLINE_FIELD stays overridable; "" falls back to workspace maxRuntime / poll-side GLOBAL_TIMEOUT_S. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…inD)
Verified on a real SaFE privileged Authoring pod (MI355X x8, rocm/pytorch
base image) that the base image ships no docker/dockerd, and fixed the
nested-container path against what actually works:
bootstrap-pre-release.sh
* add ensure_dockerd(): apt-get install docker.io, then start a pod-local
dockerd detached via setsid with --storage-driver=vfs (no systemd in the
pod; overlay-on-overlay fails). Called before fanning out docker legs.
docker-run-hyperloom.sh
* GPU index -> renderD is stride 8, not +1: RD=128+GPU_INDEX*8 (verified
GPU i == renderD(128+8i) via /sys/class/drm + rocm-smi --showbus).
* --group-add video FAILS (pod /etc/group has no video/render names); use
numeric device-node GIDs from `stat -c %g` instead.
* isolate a single card via /dev/kfd + one renderD node only (drop the
cardN device; card numbering is not guaranteed aligned to GPU order),
plus --security-opt seccomp=unconfined and HIP_VISIBLE_DEVICES=0.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Policy change: after judging (and on job cancel), STOP each SaFE workload
instead of DELETE. Stop frees the GPUs immediately -- so a 3h leg that
finished early does not idle-hold its card until the `timeout` deadline --
while keeping the workload record and pod filesystem for post-hoc inspection.
SaFE exposes no start/restart, so Stopped records are cleaned up manually.
Verified 2026-08-27: POST /api/v1/workloads/{id}/stop exists and returns 200.
* poll: add stop_workloads() at the end, dedup by workloadId (the 4 docker
legs share one host workload).
* workflow: cancel step switched from DELETE to POST .../stop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Rename every ${{ secrets.* }} / ${{ vars.* }} reference to carry a PRE_E2E_
prefix so this release-gate CI's config never collides with the existing
per-PR CI (ci-e2e.yml) secrets/variables in the same repo. GITHUB_TOKEN is
left as-is (built-in, cannot be renamed).
Only the reference names change; the in-workflow env: keys handed to the
dispatch/poll scripts (SAFE_API_BASE, NFS_ROOT, ...) are unchanged, so no
script edits are needed.
Owner action: create the repo Secrets/Variables under the new PRE_E2E_ names.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run the release gate on the OPEN PR (before merge), not after merge. A push-to-main trigger is deliberately removed so a bad release is blocked while the PR is open and merging does not re-run it. resolve.decide now classifies against the PR base branch instead of HEAD~1: version bump vs base -> FULL (8 legs); CI-logic-only change -> SCRIPTS-ONLY (4 fast 3h legs). Uses BASE_SHA/BASE_REF from the event; checkout fetch-depth raised to 0 so the base commit is present for the diff/version compare. poll: PR number now comes straight from the event (PR_NUMBER) with the commit->PR reverse lookup kept as a workflow_dispatch fallback; per-leg commit statuses target the PR head sha (github.sha is the merge commit). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the release-gate jobs off the shared Hyperloom-e2e-ci runner (which the per-PR smoke test ci-e2e.yml also uses, causing build to queue behind it) onto their own SaFE-hosted AutoscalingRunnerSet `hyperloom-pre-e2e-ci`. All three jobs (resolve/build/run) now target it, so the pre-release gate no longer contends with the per-PR CI for runner slots. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…-hfjjz The bare install name hyperloom-pre-e2e-ci did not pick up jobs (queued >2min); the SaFE AutoscalingRunnerSet's listener pod is hyperloom-pre-e2e-ci-hfjjz-*, so the runner scale set name carries the -hfjjz suffix. Point runs-on at it. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Switch runs-on from the SaFE ARC scale-set (hyperloom-pre-e2e-ci-hfjjz) to a standard self-hosted runner on crsuse2-m2m-061 (label hyperloom-pre-e2e-baremetal). The SaFE runner-proxy image predates the USER_APIKEY auth support (PR #662, 2026-07-10), so it never sends the injected platform key and every runner start 401s. A direct GitHub runner bypasses runner-proxy entirely. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Two bare-metal fixes: - dispatch: report create_workload failures to stderr, not stdout. The call runs inside wid=$(create_workload ...) command substitution, so a stdout message was captured into $wid and never reached the CI log -- the job failed with exit 1 and no visible reason. stderr surfaces the real HTTP status + SaFE API body. - build: add an idempotent step to mkdir+chown the CI NFS root. The runner runs as 'ubuntu', not in the group owning /shared_nfs, so it cannot create the root. Self-heals across runner reinstall / NFS remount. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SaFE derives the k8s object name from displayName. CI_VERSION contains '+'
(1.0.0.dev...+ci), which is not a legal RFC 1123 subdomain char, so every
create_workload 422'd ('metadata.name Invalid value'). Fold illegal chars to
'-', lowercase, collapse/trim dashes before POSTing. The poll step keys off
the DISPATCH_MAP (leg -> workloadId), not the name, so this is safe.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
SaFE's vworkload admission webhook enforces a stricter rule than RFC 1123: 1-44 chars, must start with an alphabetic char, end alphanumeric, lowercase [a-z0-9-]. The old 'e2e-<CI_VERSION>-<leg>' name embedded the full version (1.0.0.dev...+ci) -> 47 chars AND a leading-digit/'.'/'+' after the prefix. Build the name as 'e2e-<leg>-<6-hex CI_VERSION hash>' so the leg stays intact and never collides across runs, and harden the sanitizer (strip leading non-alpha, cap 44, re-trim trailing dash). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…nd, resources)
Six fixes surfaced by the first live run (CI_VERSION 1.0.0.dev202608271014+ci):
1. 401 subscription-key: AMD's APIM gateway (llm-api.amd.com) rejects the bearer
key alone -- it needs an Ocp-Apim-Subscription-Key header. Thread a new
ANTHROPIC_CUSTOM_HEADERS through workflow -> dispatch env -> bootstrap .env; the
CLI expands ${ANTHROPIC_API_KEY} so one secret covers bearer + subscription.
2. kind Authoring -> PyTorchJob: the Authoring mutating webhook overwrites
EntryPoints to `sleep infinity`, so our bootstrap never auto-ran. PyTorchJob is
not in that mutate switch and honors the submitted entrypoint; privileged/8-GPU/
useWorkspaceStorage/timeout are all kind-agnostic. E2E_DOCKER_HOST now travels in
the workload env (a `VAR=1;` command prefix wouldn't export into the entrypoint).
3. root permissions: `claude --print` is fail-closed and dies with no approver ->
add --dangerously-skip-permissions; it refuses under root unless IS_SANDBOX=1
(SWSPLAT-42390), matching Hyperloom's own kernel-agent. Also cd into the leg root
so claude finds the workspace .env instead of blocking at '/'.
4. minimal base image: install Node/npm (claude CLI) and jq up front -- the SaFE
rocm/pytorch image ships neither.
5. host resources: raise the privileged docker host to mem 2048Gi / ephemeral
1792Gi (ref sglang-kimik3-2). The first run was EVICTED at ephemeral 200Gi:
vfs storage x 8 ROCm images has no layer dedup.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…repo var needed)
The Ocp-Apim-Subscription-Key header value is just the API key (${ANTHROPIC_API_KEY},
expanded in-pod by Hyperloom's parse_custom_headers), carrying no new secret -- the
key already reaches the pod via ANTHROPIC_API_KEY_B64. So instead of requiring a
PRE_E2E_ANTHROPIC_CUSTOM_HEADERS repo variable (a "forgot to set it -> 401" trap),
bootstrap now defaults ANTHROPIC_CUSTOM_HEADERS to
`Ocp-Apim-Subscription-Key: ${ANTHROPIC_API_KEY}` whenever a gateway base URL is set,
matching Hyperloom's .env.template and hyperloom-setup SKILL.md exactly. An externally
supplied header still overrides; direct api.anthropic.com emits no header.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The release-gate legs hold up to 8 GPUs for as long as 14h and block the release, so they must not be starved behind dev workloads. SaFE's scheduler orders the queue by Spec.Priority (int; High=2/Med=1/Low=0 per Primus-SaFE constant.go; the webhook clamps to [0,2]). Add `priority: $prio` to the create-workload body, default 2, overridable via PRIORITY. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… rerun) Make "push a fix -> the stale run is torn down and the fresh code reruns from scratch" automatic: * concurrency.cancel-in-progress: true, group keyed per-PR. GitHub now cancels the in-flight run on a newer commit to the same PR; the run's `if: cancelled()` step stops its SaFE workloads (freeing GPUs) before the new run dispatches. Previously (cancel-in-progress: false) a new push queued behind a run built from superseded code, and stale legs kept their GPUs until they timed out. * dispatch writes DISPATCH_MAP INCREMENTALLY (record_dispatch appends after every successful create, seeded with an empty map up front) instead of once at the end. With cancel-in-progress a cancel can land mid-dispatch; the end-only write would leave already-created workloads absent from the map and leak their GPUs. Now the cleanup step always sees every workload created so far. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ded) claude --print is a single non-interactive turn: the demo skill backgrounds `optimize` (setsid nohup) and returns immediately, so run.sh exited 0 and SaFE marked a false "Succeeded" after ~6min for a 3h/12h benchmark that never wrote reports/final.json. run_leg now blocks after the demo turn until the backgrounded optimize reaches a terminal state: it globs the newest nested session dir under $session that holds a state.json (make_session_dir creates $session/<model>/<UTC_ts>-<rand8>/ and re-pins INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR only inside the CLI process, so the glob is authoritative), re-points the poll's .session_dir pin at it, and polls until reports/final.json exists (or state.json has a terminal stop_reason + a short grace). Returns 0 iff final.json exists (PASS/FAIL by gain stays the poll's job); returns 1 on startup grace (optimize never launched), hard deadline (hours*3600+3600), or a terminal stop_reason with no report -> SaFE marks Failed, not a false Succeeded. Fixing the .session_dir pin also fixes a second latent bug: the poll looked for $session/reports/final.json (parent), but the report lives in the nested run dir. dispatch: widen SaFE pod timeout (DEADLINE_3H_S 14400->16200, DEADLINE_12H_S 46800->48600) so the pod hard-timeout stays strictly greater than bootstrap's own in-pod wait deadline; otherwise SaFE could pre-empt the pod mid-wait and lose the clean return-1 path. Ordering per leg: bootstrap deadline < SaFE pod timeout < poll GLOBAL_TIMEOUT_S (50400s). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…eak) The concurrency.cancel-in-progress knob only cancels the GitHub JOB; it does NOT reliably stop the SaFE PyTorchJob pods a superseded/failed run already created. The `if: cancelled()` cleanup only gets a short grace window, and a job that FAILS (not cancels) after dispatch skipped cleanup entirely. Verified: 3 `Running` e2e workloads (incl. an 8-GPU docker host) leaked from a dead run and idle-held their cards, starving the next run. Correct order is the reverse of "cancel job -> hope the pod stops": a NEW run now STOPS every stale e2e-* workload (this workspace, non-terminal phase, not this run's own VERSION_TAG) up front via GET /workloads + POST /stop, frees the GPUs, then dispatches. The old run's poll then sees phase=Stopped and judges those legs FAIL -- the GitHub job ends as a CONSEQUENCE of stopping the pod, not the other way round. reap is resilient to an unreachable API (skips, never aborts under set -e). Also broaden the in-run cleanup step from `if: cancelled()` to `if: cancelled() || failure()` so a job that fails after dispatch stops its own workloads too (idempotent; poll's stop_workloads handles the success path; the next run's reap is the final backstop). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ase-e2e run Temporary, for testing the pre-release E2E gate end-to-end on all 8 legs. The PR's resolve job classifies a version bump vs main as a FULL run (all 8 legs) instead of the scripts-only 4-leg scope. 1.0.1a0 is a legal PEP 440 alpha so the CI_VERSION (1.0.1a0.dev<ts>+ci) and wheel build stay valid. REVERT to 1.0.0 before merge. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Docker legs hung waiting for a final.json that never came: the nested container mounted only $ROOT and $NFS_ROOT (the CI subdir), never the model dir (/shared_nfs/models). So HYPERLOOM_MODEL_PATH resolved to a path absent inside the container and optimize could not boot the server. The demo skill already requires this (examples/*/SKILL.md): "If ... a pre-downloaded model directory is outside the workspace, add matching -v host_path:host_path mounts." docker-run-hyperloom.sh was missing it. Mount the model's PARENT dir (minimal exposure); skip the extra -v when the model already lives under $ROOT/$NFS_ROOT to avoid a duplicate mount. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…) + pin image tag to the skill Docker-host pod was EVICTED live (2026-08-28): "ephemeral local storage usage exceeds the total limit of containers 1792Gi". Root cause: the pod-local dockerd uses --storage-driver=vfs (no layer dedup), and its default data-root /var/lib/docker lives in the container rootfs, which counts against the pod's ephemeralStorage quota. 4 legs pulling+running big ROCm images multiply under vfs and blow past 1792Gi. Fix: this is a privileged host pod, so it sees the node's own disks directly. Point the vfs data-root at /shared-data (node-local ~28T NVMe xfs, a real host filesystem, NOT the container rootfs overlay -> not counted toward ephemeralStorage). Reap stale e2e-docker data-roots from older runs first (/shared-data is host-persistent, so vfs copies would otherwise accumulate and fill the node NVMe). Also (obs1): stop the docker setup agent from freelancing the image tag. The prompts now extract the EXACT tag from the demo skill's SKILL.md via grep and forbid any version bump / substitution. bootstrap injects HYPERLOOM_SKILL_PATH (the leg's demo skill file at $root/.claude/skills/<demo>/SKILL.md, where pip --target materializes the [tool.setuptools.data-files] example skills) so the grep has a concrete path. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Run 1.0.1a0.dev202608280354+ci failed 6 of 8 legs on two independent causes,
both now closed with on-cluster evidence.
Docker host EVICTED again ("ephemeral local storage usage exceeds the total
limit of containers 1792Gi", ~4min after start, exit 143). Moving the vfs
data-root to /shared-data changed nothing: the pod spec shows
`shared-data: {"emptyDir":{}}`, and an emptyDir is part of the pod's local
ephemeral storage, so it counts against the very quota it was meant to dodge.
In the pod `df /shared-data` and `df /` report the identical 28T because both
are backed by the same node NVMe, which is what made it look like a hostPath.
There is no large hostPath to escape to either: /primus is 123G ext4 on one
node and a 12T NFS mount on another.
The real cost is vfs itself -- no layer dedup, so every layer and every
container is a full copy. /shared-data is a plain xfs mount, not an overlay,
and DOES accept an overlay upperdir (probe-verified on a privileged pod:
`mount -t overlay` with upperdir/workdir there returns MOUNT_OK, so xfs
ftype=1 holds). The original "overlayfs-on-overlayfs fails" reasoning only
ever applied to the container rootfs. So try overlay2, fall back to
fuse-overlayfs, and FAIL if neither is available -- vfs is not a fallback,
it is a guaranteed eviction, and a silent downgrade just reproduces the bug.
Each driver gets its own data-root subdir (dockerd refuses a data-root
holding another driver's tree) and the effective driver is logged.
Both baremetal-sglang legs exited 1 from run_leg's stall check while
demonstrably alive. The check measured absolute file mtimes under $session
only, so it charged the minutes spent inside the two non-streaming
`claude --print` turns to the leg and condemned it on the first loop
iteration: each kill lands exactly at last-$session-write + 600s (03:55:38
-> 04:06:07 and 04:03:15 -> 04:13:46). The 12h leg had written
$root/setup_sglang_retry.log 26s before being declared hung -- the agent's
launcher and setup/install logs live next to the workspace, not under
session/, and install.sh writes $root/.cache.
Extract the arithmetic into leg_idle_s(root, loop_start, now): scope the
liveness scan to $root and measure idleness from the LATER of last write and
loop start, so a slow build still keeps the leg alive and the pre-loop gap is
never charged to it, while a genuinely hung launch is still reaped in ~10m.
Also mirror both agent turns to $session/agent-<leg>.log. SaFE deletes a
failed leg's PyTorchJob immediately, taking the pod stdout with it; 3 of this
run's 5 workloads left no recoverable log and the root cause had to be
reconstructed from file mtimes.
Add test_pre_release_stall_liveness.py, which runs the real leg_idle_s out of
the script (pre-fix logic answers 629s on the observed case, post-fix 5s) and
pins that vfs never returns as a storage driver.
Co-authored-by: Cursor <cursoragent@cursor.com>
overlay2 fixed the docker-host eviction (200G plateau vs the 1792Gi limit, pod
alive well past the ~4min eviction point) and the stall fix kept the sglang legs
alive through a 5min source build. Run 1.0.1a0.dev202608280519+ci then exposed the
next layer: most legs never launched `optimize` at all.
`claude --print` is one turn, and the agent sometimes ends it with a progress note
instead of finishing the job. Live transcripts:
baremetal-vllm-3h setup: "Base tooling installed. Waiting on the vLLM ROCm wheel
install." (never printed `setup complete: ...`)
demo: "Good progress -- torch is in ... Waiting on the monitor."
baremetal-vllm-12h demo: "Install step started. Waiting for it to complete."
Everything the agent leaves running is a child of that turn, so it dies with it:
both legs were left with no claude and no optimize process, and the stall check
correctly reaped them 600s later. This is why the previous run's vllm legs passed
and this one's did not -- there the demo turn happened to run 16min and got as far
as `setsid nohup optimize`; here it returned after 2-4min. Non-deterministic agent
behaviour, so the harness has to converge it rather than hope.
Setup turn: the prompt's contract is a literal `setup complete: <mode>/<backend>`
line, so grep for it and re-drive the prompt (LEG_TURN_ATTEMPTS, default 3) until
the agent reports it, failing the leg only after that.
Demo turn: re-drive off the EXISTING stall signal instead of probing "is it
launched" right after the turn. The demo skill's launcher runs install.sh before
backgrounding optimize, so state.json can legitimately be 10+ minutes out and a
short-grace probe would double-launch; whereas "no state.json and nothing written
under the leg root for 600s" already means nothing is running. On that signal, ask
the agent to finish the job (LEG_DEMO_REDRIVES, default 2) before giving up.
Split the wait loop's clocks so the re-drive cannot extend the pod deadline:
start_ts still backs deadline_s and is never reassigned, while the new grace_ts
backs the stall window and restarts after each re-driven turn. The invariant
bootstrap deadline < SaFE pod timeout < poll GLOBAL_TIMEOUT_S is unchanged.
Both demo prompts now state that the turn is single and non-interactive, that
ending it on "install started" / "waiting on the monitor" kills the work, that
optimize must be detached with setsid nohup, and that the run must be confirmed
live (nested run dir + state.json + live PID) before the turn ends.
Tests: leg_run_started is exercised against a real session tree (a stray
state.json at the session top must not count as a launch), the setup marker is
cross-checked against all four setup prompts so the grep and the prompts cannot
drift, and the deadline clock is pinned against being reset by a re-drive.
Co-authored-by: Cursor <cursoragent@cursor.com>
… drop preempt Three things, all from run 1.0.1a0.dev202608280605+ci. 1. Setup was budgeted in TURNS, and that was my regression. `claude --print` is "print response and exit", and a framework install (vLLM ROCm wheel into an isolated venv, or SGLang compiled from source for gfx950) runs 10-30min -- far longer than one turn holds. So every agent backgrounds the install, polls its log, and eventually answers with a progress note, which ends the turn with setup incomplete. A cap of 3 turns amounted to ~4min of wall clock and killed two legs whose installs were provably still going: baremetal-vllm-3h died 3.5min in, mid "Phase 2 (isolated vLLM install) is underway" baremetal-sglang-12h died 5.7min in, mid "Still waiting on the kernel build" Budget it in time instead: LEG_SETUP_DEADLINE_S (45m) bounds the whole step, and the existing liveness signal decides failure -- a leg dies only once nothing has been written under the leg root for LEG_SETUP_STALL_S (10m), i.e. nothing is installing. LEG_SETUP_MAX_TURNS stays purely as a token-spend backstop. 2. Follow-up turns now RESUME the leg's conversation instead of re-feeding the prompt as a fresh turn. Re-feeding threw away everything the agent already knew, leaving it to rediscover which install it had launched and which log it was watching. `--session-id` opens a stable per-leg UUID (sha1 of leg+CI_VERSION, shaped into a v4 UUID -- distinct per leg, which matters on the docker host where four agents share one pod) and `--resume` continues it with a one-line nudge. The demo turn resumes the same session too, exactly as a human would keep one chat. Resume failure degrades to a standalone turn rather than failing the leg. That the detached install survives a turn boundary is verified: setup_vllm.log grew 71KB -> 77KB across one, and baremetal-vllm-12h converged to `setup complete: baremetal/vllm` on turn 3 and went on to launch optimize. 3. Remove the `preempt` job and pre-release-e2e-reap.sh. It could never work. A newer run cannot tear down an older one: while the concurrency group is held, GitHub keeps the new run at run-level `pending` with an EMPTY jobs array (run 33145644120 sat pending 20min with zero jobs), so no job of it exists to reclaim anything. And it ran on a GitHub-hosted runner while SAFE_API_BASE is an in-network NodePort, so every run logged `[preempt] could not list workloads; skipping reclaim` after a 30s curl timeout, having stopped nothing -- three runs checked, all identical. Keeping it only made the gate look protected. Teardown runs the other way round instead: poll fails fast on the first FAIL (POLL_FAIL_FAST, default on). It is a release gate -- the first FAIL already blocks the release, so waiting the rest out buys nothing while costing a 12h leg's GPUs and the only self-hosted runner, which is what left the next fix's run stuck at `pending` until the old run was cancelled by hand. Remaining legs are recorded as not judged rather than FAIL. Poll also sleeps in POLL_SLEEP_SLICE_S slices so a cancel lands in seconds instead of at the end of a full interval. Leaked pods stay covered by the dispatch-side reap, which -- unlike preempt -- runs in-network and only executes once this run holds the runner, i.e. once the previous run is provably done. Tests: setup budget must not be turn-based, follow-ups must resume, the session uuid is checked for shape/stability/uniqueness, all agent invocations must go through the single teeing helper, and the orchestration guard pins that no job touching SaFE runs on a GitHub-hosted runner and that fail-fast is armed on every FAIL path. Co-authored-by: Cursor <cursoragent@cursor.com>
Pre-release E2E — Complete (8/8 legs done)CI_VERSION
GATE: FAIL — one or more legs did not pass. Release blocked. |
…ory to 512Gi Fail-fast still marks the gate FAIL and releases the runner, but workloads that were still optimizing are left alive for post-mortem. Dispatch reap on the next run remains the backstop. Baremetal legs now request 512Gi memory and ephemeral storage after sglang-12h OOM at 128Gi. Co-authored-by: Cursor <cursoragent@cursor.com>
…vior Reformat test_pre_release_gate_orchestration and test_pre_release_stall_liveness so ruff format --check passes in CI. Co-authored-by: Cursor <cursoragent@cursor.com>
VERSION_TAG now hashes CI_VERSION with GITHUB_RUN_ID so dispatch reap can stop pods from a superseded push even when the wheel version is unchanged. Poll detects a newer queued workflow run for the same PR branch, releases the runner without stopping workloads, and leaves teardown to the successor. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Align poll gate with optimize CLI exit 0: final.json must carry a terminal stop_reason in the success set (time_exhausted, target_reached, etc.). TARGET_GAIN still flows to the demo skill at 100 but no longer gates PASS. Co-authored-by: Cursor <cursoragent@cursor.com>
Pre-release E2E — Complete (8/8 legs done)CI_VERSION
GATE: SUPERSEDED — newer run queued; workloads left for dispatch reap. |
Set nested container caps to 256g/512g for 3h/12h legs (64g shm) to avoid OOM on 14B-FP8 docker paths, and bump the privileged docker-host CPU request to 196 for four parallel agent/setup processes atop 4x32 container caps. Co-authored-by: Cursor <cursoragent@cursor.com>
Pre-release E2E — Complete (8/8 legs done)CI_VERSION
GATE: FAIL — one or more legs did not pass. Release blocked. |
Judge PASS/FAIL from state.json stop_reason instead of final.json, and complete bootstrap when optimize exits cleanly even if final.json is late. Sync 3h to framework-only (no --no-framework-agent, 0.90 framework pct), align 12h prompts with the 14b skill, and raise demo re-drive budget to 5. Co-authored-by: Cursor <cursoragent@cursor.com>
Pre-release E2E — Complete (8/8 legs done)CI_VERSION
GATE: SUPERSEDED — newer run queued; workloads left for dispatch reap. |
Mirror the 3h skill launch requirements: spell out phase budget flags and forbid --no-framework-agent / --no-kernel. Keep target-gain at 50 for users. Co-authored-by: Cursor <cursoragent@cursor.com>
Pre-release E2E — Complete (8/8 legs done)CI_VERSION
GATE: SUPERSEDED — newer run queued; workloads left for dispatch reap. |
Queue the privileged docker host first so dockerd startup and image pulls overlap with baremetal scheduling instead of starting after four 1-GPU pods. Co-authored-by: Cursor <cursoragent@cursor.com>
Pre-release E2E — Complete (8/8 legs done)CI_VERSION
GATE: FAIL — one or more legs did not pass. Release blocked. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds a pre-release end-to-end CI (
Pre-release E2E test) that exercises the full user path before a release: build the CI wheel, stage it plus the in-pod bootstrap/prompts to NFS, then run 8 demo legs (baremetal/docker × vLLM/SGLang × 3h/12h) as SaFE Authoring workloads driven through the Claude CLI + setup/demo skills — the same path a user takes. Independent of the per-PR smoke test (ci-e2e.yml).Design:
hyperloom-pre-release-e2e-ci-design.md.Release gate
target_gain = 100%is a hard gate (same params as the demo skills otherwise). A leg PASSes only when itsreports/final.jsonhasstop_reason == "target_reached"(≡cumulative_gain_validated >= 100), the workload didn't fail, bothfinal.json/final.mdexist, and crash/boot-failure counts are within tolerance.pre-release-e2e/<leg>), so the 3h legs surface in ~3–4h instead of waiting on the 14h barrier.Triggers
mainthat bumpspyprojectversion → FULL run (all 8 legs).mainthat only changes this CI's scripts/prompts/workflow (version unchanged) → SCRIPTS-ONLY run (the 4 fast 3h legs, to validate the logic change without burning a full 14h GPU round).workflow_dispatch→ chosen subset (default all 8), optional wheel reuse.Topology / isolation
.env(scrubbed on exit), never written to NFS.Config
All environment values are
${{ secrets.* }}/${{ vars.* }}references — nothing is hard-coded. Repo Secrets/Variables to populate before the first real run are listed in the workflow header (SAFE_API_KEY,ANTHROPIC_API_KEY;SAFE_API_BASE,SAFE_WORKSPACE_ID,AUTHORING_IMAGE,PRE_RELEASE_NFS_ROOT,MODEL_3H_PATH,MODEL_12H_PATH,CLAUDE_MODEL,CLAUDE_CLI_VERSION, …).Layout
Validation
bash -n.lint.ymlfor the authoritative pass.