Add pre-release E2E CI: full-path release gate with layered per-leg judging

.github/actionlint.yaml

@@ -0,0 +1,8 @@
+# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc.
+# SPDX-License-Identifier: MIT
+#
+# Declare self-hosted runner labels so actionlint does not flag them as unknown.
+# Used by ci-e2e.yml and pre-release-e2e-test.yml.
+self-hosted-runner:
+  labels:
+    - Hyperloom-e2e-ci

.github/pre-release/prompts/pre-release/demo-12h.md

@@ -0,0 +1,56 @@
+# Pre-release E2E — 12h demo leg
+
+You are running the Hyperloom pre-release E2E test non-interactively. Run the 12-hour
+demo to completion, then stop. Setup already ran successfully in this workspace.
+
+Invoke the `hyperloom-qwen3-14b-fp8-12h` demo skill with **one** override and otherwise
+its exact default flags.
+
+## Flags
+
+- **OVERRIDE:** use `--target-gain 100` (NOT the skill's default of 50). This shapes
+  optimize prompts only; the poll gate judges PASS/FAIL from `stop_reason`, not gain.
+- Keep every other required flag exactly as the skill defines them:
+
+  ```
+  --tp 1 --conc 64 --isl 1024 --osl 1024 --precision fp8 --max-hours 12
+  --max-minutes-framework-pct 0.43 --max-minutes-kernel-pct 0.42
+  ```
+
+  Do **not** pass `--no-framework-agent` or `--no-kernel` — the 12h demo runs the full
+  OPTIMIZE phase (FRAMEWORK_AGENT + KERNEL_AGENT).
+
+## Model path
+
+The skill will ask which model to use. Do **not** ask interactively — use
+`MODEL_PATH` from the repository-root `.env` (it is already set to the demo model,
+Qwen3-14B-FP8). Verify that path contains `config.json`; if it does, use it and
+continue without asking. Load LLM API keys/base URLs and `FRAMEWORK` from `.env`.
+
+## Hard constraints (automated release gate)
+
+- Do **not** modify any GPU-related environment variable or device visibility.
+- Do **not** choose GPUs via `rocm-smi`.
+- If `HYPERLOOM_RUN_MODE=docker`, run `optimize` **inside the container you started in
+  setup** via `docker exec -w "$REPO_ROOT" "$HYPERLOOM_CONTAINER_NAME" …` (per the demo
+  skill's docker mode). Do **not** start a new container and do **not** change its
+  device/isolation flags. Otherwise (baremetal) run directly and do not run `docker`.
+- Do **not** modify `USER_DATA_PATH`.
+- Do **not** print or copy secret values into output, reports, or logs.
+
+## Termination — do not end this turn until the run is launched
+
+This is a **single non-interactive turn**, and anything still running as a child of it is
+killed the moment the turn ends. So:
+
+1. Finish the install and the launch **inside this turn**. Do **not** end the turn with a
+   progress note such as "install started", "waiting on the pull", or "waiting on the
+   monitor" — that kills the work you just started and the leg ends up with nothing
+   running at all.
+2. Start `optimize` **detached** with `setsid nohup` (as the demo skill does) so it
+   survives the end of this turn.
+3. Before you finish, confirm the run is really live and report the paths: the nested
+   session run dir exists, `state.json` is present in it, and the optimizer PID is alive.
+
+Only then stop. The harness polls `state.json` for a clean terminal `stop_reason` to
+judge PASS/FAIL — do not fabricate a result.

.github/pre-release/prompts/pre-release/demo-3h.md

@@ -0,0 +1,57 @@
+# Pre-release E2E — 3h demo leg
+
+You are running the Hyperloom pre-release E2E test non-interactively. Run the 3-hour
+demo to completion, then stop. Setup already ran successfully in this workspace.
+
+Invoke the `hyperloom-qwen3-8b-3h` demo skill with **one** override and otherwise its
+exact default flags.
+
+## Flags
+
+- **OVERRIDE:** use `--target-gain 100` (NOT the skill's default of 30). This shapes
+  optimize prompts only; the poll gate judges PASS/FAIL from `stop_reason`, not gain.
+- Keep every other required flag exactly as the skill defines them:
+
+  ```
+  --tp 1 --conc 64 --isl 1024 --osl 1024 --precision bf16 --max-hours 3
+  --max-minutes-framework-pct 0.90 --max-minutes-sweep-pct 0.01
+  --no-kernel --no-enable-conc-sweep --no-enable-roofline
+  ```
+
+  Do **not** pass `--no-framework-agent` — the 3h demo runs FRAMEWORK_AGENT (OPTIMIZE)
+  only; kernel is skipped via `--no-kernel`.
+
+## Model path
+
+The skill will ask which model to use. Do **not** ask interactively — use
+`MODEL_PATH` from the repository-root `.env` (it is already set to the demo model).
+Verify that path contains `config.json`; if it does, use it and continue without
+asking. Load LLM API keys/base URLs and `FRAMEWORK` from `.env`.
+
+## Hard constraints (automated release gate)
+
+- Do **not** modify any GPU-related environment variable or device visibility.
+- Do **not** choose GPUs via `rocm-smi`.
+- If `HYPERLOOM_RUN_MODE=docker`, run `optimize` **inside the container you started in
+  setup** via `docker exec -w "$REPO_ROOT" "$HYPERLOOM_CONTAINER_NAME" …` (per the demo
+  skill's docker mode). Do **not** start a new container and do **not** change its
+  device/isolation flags. Otherwise (baremetal) run directly and do not run `docker`.
+- Do **not** modify `USER_DATA_PATH`.
+- Do **not** print or copy secret values into output, reports, or logs.
+
+## Termination — do not end this turn until the run is launched
+
+This is a **single non-interactive turn**, and anything still running as a child of it is
+killed the moment the turn ends. So:
+
+1. Finish the install and the launch **inside this turn**. Do **not** end the turn with a
+   progress note such as "install started", "waiting on the pull", or "waiting on the
+   monitor" — that kills the work you just started and the leg ends up with nothing
+   running at all.
+2. Start `optimize` **detached** with `setsid nohup` (as the demo skill does) so it
+   survives the end of this turn.
+3. Before you finish, confirm the run is really live and report the paths: the nested
+   session run dir exists, `state.json` is present in it, and the optimizer PID is alive.
+
+Only then stop. The harness polls `state.json` for a clean terminal `stop_reason` to
+judge PASS/FAIL — do not fabricate a result.

.github/pre-release/prompts/pre-release/setup-baremetal-sglang.md

@@ -0,0 +1,38 @@
+# Pre-release E2E — setup (baremetal + SGLang)
+
+You are running the Hyperloom pre-release E2E test non-interactively. Complete the
+setup step for a **baremetal + SGLang** leg, then stop. Do not run the demo yet.
+
+## Environment (already prepared)
+
+A `.env` file exists in the current workspace (`REPO_ROOT`) with these values already
+set: `ANTHROPIC_API_KEY`, `CLAUDE_MODEL`, `USER_DATA_PATH`,
+`HYPERLOOM_RUN_MODE=baremetal`, `FRAMEWORK=sglang`, `MODEL_PATH`, `TARGET_GAIN`,
+`DEMO_HOURS`. The wheel is already installed via `pip install --target .` so a
+`hyperloom/` package directory is present.
+
+## Fixed decisions
+
+Run the `hyperloom-setup` skill with these fixed decisions — do **not** ask
+interactive questions; use the values already in `.env` and the environment:
+
+- **Run mode:** baremetal (`HYPERLOOM_RUN_MODE` is already `baremetal`; keep it).
+- **Framework:** SGLang. Install the framework layer with the setup backend
+  (`--install-framework sglang`).
+- **LLM provider / model / `USER_DATA_PATH`:** use the values already in `.env`; do
+  not change them.
+
+## Hard constraints (automated release gate)
+
+- Do **not** modify any GPU-related environment variable (`ROCR_VISIBLE_DEVICES`,
+  `HIP_VISIBLE_DEVICES`, `GPUS_PER_NODE`, etc.). The pod already exposes exactly one
+  GPU; do not override device visibility.
+- Do **not** run `docker` and do **not** choose GPUs via `rocm-smi`. This is a
+  baremetal leg; setup runs on the host.
+- Do **not** print, echo, or copy secret values (API keys) into output or logs.
+- Do **not** modify `USER_DATA_PATH`.
+
+## Termination
+
+When setup completes successfully, stop. Report only `setup complete: baremetal/sglang`.
+If setup hard-fails, report the failure and stop.

.github/pre-release/prompts/pre-release/setup-baremetal-vllm.md

@@ -0,0 +1,38 @@
+# Pre-release E2E — setup (baremetal + vLLM)
+
+You are running the Hyperloom pre-release E2E test non-interactively. Complete the
+setup step for a **baremetal + vLLM** leg, then stop. Do not run the demo yet.
+
+## Environment (already prepared)
+
+A `.env` file exists in the current workspace (`REPO_ROOT`) with these values already
+set: `ANTHROPIC_API_KEY`, `CLAUDE_MODEL`, `USER_DATA_PATH`,
+`HYPERLOOM_RUN_MODE=baremetal`, `FRAMEWORK=vllm`, `MODEL_PATH`, `TARGET_GAIN`,
+`DEMO_HOURS`. The wheel is already installed via `pip install --target .` so a
+`hyperloom/` package directory is present.
+
+## Fixed decisions
+
+Run the `hyperloom-setup` skill with these fixed decisions — do **not** ask
+interactive questions; use the values already in `.env` and the environment:
+
+- **Run mode:** baremetal (`HYPERLOOM_RUN_MODE` is already `baremetal`; keep it).
+- **Framework:** vLLM. Install the framework layer with the setup backend
+  (`--install-framework vllm`, isolated framework env).
+- **LLM provider / model / `USER_DATA_PATH`:** use the values already in `.env`; do
+  not change them.
+
+## Hard constraints (automated release gate)
+
+- Do **not** modify any GPU-related environment variable (`ROCR_VISIBLE_DEVICES`,
+  `HIP_VISIBLE_DEVICES`, `GPUS_PER_NODE`, etc.). The pod already exposes exactly one
+  GPU; do not override device visibility.
+- Do **not** run `docker` and do **not** choose GPUs via `rocm-smi`. This is a
+  baremetal leg; setup runs on the host.
+- Do **not** print, echo, or copy secret values (API keys) into output or logs.
+- Do **not** modify `USER_DATA_PATH`.
+
+## Termination
+
+When setup completes successfully, stop. Report only `setup complete: baremetal/vllm`.
+If setup hard-fails, report the failure and stop.

.github/pre-release/prompts/pre-release/setup-docker-sglang.md

@@ -0,0 +1,114 @@
+# Pre-release E2E — setup (docker + SGLang)
+
+You are running the Hyperloom pre-release E2E test non-interactively. Complete the
+setup step for a **docker + SGLang** leg, then stop. Do not run the demo yet.
+
+> **IMPORTANT — you own the container.** You are on the privileged host pod. This is a
+> `docker` leg, so **you** must start the backend container yourself by following the
+> `hyperloom-setup` skill and the demo skill's **docker mode**: run `docker run` to start
+> a long-lived single-GPU container, then run setup **inside** it with `docker exec`.
+> Docker is already available on this host (a pod-local `dockerd` is running).
+
+## Environment (already prepared)
+
+A `.env` file exists in the current workspace (`REPO_ROOT`) with these values already
+set: `ANTHROPIC_API_KEY`, `CLAUDE_MODEL`, `USER_DATA_PATH`,
+`HYPERLOOM_RUN_MODE=docker`, `FRAMEWORK=sglang`, `MODEL_PATH`, `TARGET_GAIN`,
+`DEMO_HOURS`, `HYPERLOOM_CONTAINER_NAME`, `HYPERLOOM_SHM_SIZE`, plus the CI
+isolation values `E2E_RENDERD`, `E2E_KFD_GID`, `E2E_DRI_GID`, `E2E_LEG_CPUS`,
+`E2E_LEG_MEM`, `E2E_NFS_MOUNT`. The wheel is already installed via
+`pip install --target .` so a `hyperloom/` package directory is present.
+
+`HYPERLOOM_IMAGE` is **not** in `.env` — you choose it (see "Image selection" below).
+
+## Fixed decisions
+
+Follow the `hyperloom-setup` skill in **docker** mode with these fixed decisions — do
+**not** ask interactive questions; use the values already in `.env`:
+
+- **Run mode:** docker. Start the container yourself (see the hard constraints below for
+  the exact `docker run` flags), then `docker exec` the setup inside it. Do **not** set
+  `HYPERLOOM_DOCKER_TARGET_HOST` (run on the current host).
+- **Framework:** SGLang — provided by the container image; run setup with
+  `--install-framework none --yes` inside the container (do **not** `--install-framework
+  sglang`).
+- **LLM provider / model / `USER_DATA_PATH`:** use the values already in `.env`; do
+  not change them.
+
+## Image selection
+
+`HYPERLOOM_IMAGE` is not preset. You **MUST** use the **exact, verbatim** `docker.io/...`
+tag string that is written on the demo skill's matching `sglang` row in its **"Suggested
+Docker images"** section — nothing else. This is a hard release-gate constraint, not a
+suggestion:
+
+- **Do NOT freelance the tag.** Do not bump the version, do not pick a "newer" or
+  "latest" build, do not substitute a different tag from your memory, from Docker Hub, or
+  from anywhere other than the skill file. The pinned tag is the one the release is gated
+  on; a different tag is a **failure**, even if it also pulls successfully.
+- The skill lists a **different `sglang` image per GPU architecture**, so first detect the
+  architecture on this host, then read the **exact** tag for the matching row **from the
+  skill file itself** rather than typing it out (the demo skill's path is in
+  `HYPERLOOM_SKILL_PATH` in `.env`; otherwise it is the `SKILL.md` of the demo skill you
+  are running):
+
+  ```bash
+  gfx="$(/opt/rocm/bin/rocminfo 2>/dev/null | grep -oiE 'gfx9[0-9a-f]+' | head -1)"
+  case "$gfx" in
+    gfx950) row='MI355X' ;;   # MI355X
+    gfx942) row='MI300X' ;;   # MI300X
+    *) echo "ERROR: could not detect GPU arch (gfx='$gfx')"; exit 1 ;;
+  esac
+  HYPERLOOM_IMAGE="$(grep -E "^- \`sglang\` $row" "$HYPERLOOM_SKILL_PATH" | grep -oE 'docker\.io/[^`]+' | head -1)"
+  echo "arch=$gfx row=$row using image: $HYPERLOOM_IMAGE"
+  ```
+
+  This is architecture **detection only** — it selects the image tag, not which card the
+  leg runs on (the card is pinned by the isolation flags below).
+- If the arch cannot be detected, the extracted tag is empty, or the resulting image
+  cannot be pulled, **stop and report the failure** — do **not** substitute any other tag
+  to work around it.
+
+Export the extracted value as `HYPERLOOM_IMAGE` for the `docker run` below.
+
+## Hard constraints (automated release gate)
+
+Your `docker run` **MUST** use exactly the flags below. These **replace** the demo
+skill's default `--device /dev/dri` (all GPUs) and `--group-add video` (a group *name*
+the pod has no entry for) with single-card isolation and numeric GIDs. Every other
+aspect of the skill's docker flow (long-lived `--entrypoint tail … -f /dev/null`,
+mounting `$REPO_ROOT:$REPO_ROOT`, `docker exec` setup, running optimize inside) is
+unchanged. Read the values from `.env`:
+
+- **Container name:** `--name "$HYPERLOOM_CONTAINER_NAME"` (already unique per leg;
+  another leg may share this host's dockerd, so do not rename it to a fixed value).
+- **Image:** the `HYPERLOOM_IMAGE` you selected above (skill list + detected arch).
+- **Single-GPU isolation (REPLACES `--device /dev/dri`):**
+  `--device /dev/kfd --device /dev/dri/renderD${E2E_RENDERD}`
+- **Numeric group-add (REPLACES `--group-add video`):**
+  `--group-add ${E2E_KFD_GID} --group-add ${E2E_DRI_GID}`
+- **Resource caps:** `--cpus ${E2E_LEG_CPUS} --memory ${E2E_LEG_MEM} --shm-size ${HYPERLOOM_SHM_SIZE}`
+- **Security:** `--security-opt seccomp=unconfined`
+- **Device pin:** `-e HIP_VISIBLE_DEVICES=0 -e ROCR_VISIBLE_DEVICES=0`
+- **Mounts:** `-v "$REPO_ROOT:$REPO_ROOT" -v "${E2E_NFS_MOUNT}:${E2E_NFS_MOUNT}"` —
+  mounting **all** of `${E2E_NFS_MOUNT}` at the same absolute path is **required** so
+  that `MODEL_PATH` (a symlink into another subtree under `${E2E_NFS_MOUNT}`) resolves
+  inside the container. Do **not** mount only the model's parent directory.
+
+Also:
+
+- Do **not** add any other `--device`, and do **not** use `--group-add video`. You may run
+  `rocminfo`/`rocm-smi` to **detect the GPU architecture** for image selection, but do
+  **not** use them to **choose which GPU** the leg runs on: the single bound `renderD` node
+  + `HIP/ROCR_VISIBLE_DEVICES=0` are what pin this leg to its one card. Do not add
+  `HIP_VISIBLE_DEVICES`/`ROCR_VISIBLE_DEVICES` values other than `0`.
+- Do **not** print, echo, or copy secret values (API keys) into output or logs. The key
+  lives only in the pod-local `.env` (it reaches the container via `-v $REPO_ROOT:$REPO_ROOT`);
+  do **not** write it anywhere else, and never onto NFS outside that `.env`.
+- Do **not** modify `USER_DATA_PATH`.
+
+## Termination
+
+When setup completes successfully, stop and report only `setup complete: docker/sglang`.
+Leave the container **running** so the demo turn can `docker exec` into it. If setup
+hard-fails, report the failure and stop.