SENPAI Console integration + Phase 2 steer/watch hooks

CONSOLE-INTEGRATION.md

@@ -0,0 +1,71 @@
+# SENPAI Console integration (Phase 2 change-set)
+
+The [SENPAI Console](https://github.com/wandb/senpai-console) is a separate repo that
+monitors, guides, and steers this fleet. This change-set is the small, coordinated
+set of changes that must land **here** so the console can index the fleet reliably
+and steer agents near-live. Everything here is **additive and inert by default** —
+it does not change current behaviour unless the console wires up the new env vars.
+
+## Runtime reality (important)
+
+The console build plan describes an OpenHands "agent-server" (Surface B) for live
+watch+steer. **This fleet runs Claude Code** (`k8s/run-senpai-claude.sh`), not
+OpenHands, so Phase 2 is implemented in terms of the Claude-Code runtime:
+
+| Build-plan concept | This fleet's mechanism |
+| --- | --- |
+| Live event stream (`/sockets/events/...`) | `SENPAI_EVENT_DIR` — Claude Code's `--output-format stream-json` is mirrored to a stable per-iteration event file the console tails. |
+| `send_message()` mid-run steer | `SENPAI_CONSOLE_INBOX_DIR` — the console drops a directive file; the entrypoint drains it into the **next heartbeat's** prompt (near-live). |
+| `pause` / `interrupt` | Not natively available under `claude -p`; the existing watchdogs + GitHub-mediated controls remain the mechanism. |
+
+## What changed
+
+1. **Markers (P0-S2).** `system_instructions/CLAUDE-ADVISOR.md` now instructs the
+   advisor to write `SENPAI-EXP` (lineage/queue/taste, in each assignment PR body)
+   and `SENPAI-ADVISOR` (heartbeat status). `CLAUDE-STUDENT.md` notes the experiment
+   id comes from the PR's `SENPAI-EXP`. These give the console durable lineage/queue/
+   status without scraping. Keep them single-line valid JSON — the console degrades
+   loudly (logs) on malformed markers, never silently.
+
+2. **Supervisor role.** `system_instructions/CLAUDE-SUPERVISOR.md` — the console's
+   control agent (sensors, alerts, charts, code-explain, steering; `act_safe`
+   default). It runs in the **console backend** today (`backend/senpai_console/
+   supervisor/`); an in-fleet pod is a future option that would need a dedicated
+   `entrypoint-supervisor.sh`.
+
+3. **Live event mirroring (`SENPAI_EVENT_DIR`).** `k8s/run-senpai-claude.sh` mirrors
+   the stream-json into `$SENPAI_EVENT_DIR/event-*.jsonl` when the var is set, so the
+   console ingests a stable event log instead of scraping iteration logs.
+
+4. **Console → agent inbox (`SENPAI_CONSOLE_INBOX_DIR`).** `k8s/senpai-console-inbox.sh`
+   provides `senpai_drain_inbox`; the advisor/student entrypoints drain pending
+   directive files into that iteration's prompt (under a `# Console directives`
+   header) and archive them. This is how a Supervisor steer or a "researcher edited
+   `<doc>`" ping reaches the agent near-live. An inbox directive also wakes the
+   advisor even when nothing else is actionable.
+
+5. **Launch plumbing.** `k8s/launch.py` gains `--event_dir` and `--console_inbox_dir`,
+   passed through to both advisor and student ConfigMaps (default empty = disabled).
+
+## Enable it
+
+```bash
+python k8s/launch.py --tag <tag> --target_repo_url <url> --advisor true \
+  --event_dir /mnt/new-pvc/senpai-events/<tag> \
+  --console_inbox_dir /mnt/new-pvc/senpai-inbox/<tag>
+```
+
+Point the console's `OH_STATE_DIR` at the event dir and its inbox writer at the
+inbox dir (both on the shared PVC the console can also mount). Leave them unset to
+keep the fleet exactly as it is today.
+
+## Tests
+
+- `bash k8s/test-senpai-console-inbox.sh` — unit-tests `senpai_drain_inbox`
+  (inert-when-unset, drains + archives, second drain empty).
+- `bash -n` passes on all modified scripts; `python k8s/launch.py --dry_run true …`
+  renders the new keys into every ConfigMap.
+
+**Not cluster-tested:** the entrypoint wiring and event mirroring are additive and
+inert by default, but have not been run on a live cluster — verify on a scratch tag
+before relying on them.

k8s/entrypoint-advisor.sh

@@ -125,6 +125,7 @@ start_hivemind
 # --- Load CC run command helper function ---
 source "$WORKDIR/k8s/run-senpai-claude.sh"
 source "$WORKDIR/k8s/advisor-claude-watchdog.sh"
+source "$WORKDIR/k8s/senpai-console-inbox.sh"
 
 # --- Register Weave CC plugin (tools already baked into Docker image) ---
 export PATH="$HOME/.claude/bin:$PATH"
@@ -210,6 +211,14 @@ while true; do
     [ "$POD_ANOMALY_COUNT" -gt 0 ] && TRIAGE_INFO+=$'\n'"- **Student pod anomalies ($POD_ANOMALY_COUNT):** investigate these before assigning new work: $(printf '%s' "$POD_ANOMALY_JSON" | json_join)"
     echo "$TRIAGE_INFO"
 
+    # --- Console -> agent inbox (near-live steer): drain pending directives ---
+    INBOX_DIRECTIVES="$(senpai_drain_inbox)"
+    INBOX_COUNT=0
+    if [ -n "$INBOX_DIRECTIVES" ]; then
+        INBOX_COUNT=1
+        TRIAGE_INFO="${TRIAGE_INFO}"$'\n\n'"${INBOX_DIRECTIVES}"
+    fi
+
     # --- Log triage state and select prompt ---
     echo "=== Log: $LOGFILE ==="
     echo "$TRIAGE_INFO" > "$LOGFILE"
@@ -223,7 +232,7 @@ while true; do
         run_advisor_claude_with_watchdog $MAX_TURNS "${FULL_PROMPT}"$'\n\n'"${TRIAGE_INFO}" || EXIT_CODE=$?
     else
         # --- Programmatic skip: skip rest of CC loop if nothing actionable ---
-        if [ "$REVIEW_COUNT" -eq 0 ] && [ "$ADVISOR_ACTION_COUNT" -eq 0 ] && [ "$ISSUE_COUNT" -eq 0 ] && [ "$IDLE_COUNT" -eq 0 ] && [ "$POD_ANOMALY_COUNT" -eq 0 ]; then
+        if [ "$REVIEW_COUNT" -eq 0 ] && [ "$ADVISOR_ACTION_COUNT" -eq 0 ] && [ "$ISSUE_COUNT" -eq 0 ] && [ "$IDLE_COUNT" -eq 0 ] && [ "$POD_ANOMALY_COUNT" -eq 0 ] && [ "$INBOX_COUNT" -eq 0 ]; then
             echo "=== Iteration $ITERATION: Nothing actionable, sleeping $SLEEP_TIME_S seconds + up to ${POLL_JITTER_S}s jitter ==="
             senpai_sleep_with_jitter "$SLEEP_TIME_S" "$POLL_JITTER_S"
             continue

k8s/entrypoint-student.sh

@@ -73,6 +73,7 @@ start_hivemind
 # --- Load CC run command helper function ---
 source "$WORKDIR/k8s/run-senpai-claude.sh"
 source "$WORKDIR/k8s/student-claude-watchdog.sh"
+source "$WORKDIR/k8s/senpai-console-inbox.sh"
 
 # --- Register Weave Claude Code Plugin (tools already baked into Docker image) ---
 export PATH="$HOME/.claude/bin:$PATH"
@@ -178,6 +179,10 @@ while true; do
         continue
     fi
 
+    # --- Console -> agent inbox (near-live steer): drain only when we will run ---
+    INBOX_DIRECTIVES="$(senpai_drain_inbox)"
+    [ -n "$INBOX_DIRECTIVES" ] && TRIAGE_INFO="${TRIAGE_INFO}"$'\n\n'"${INBOX_DIRECTIVES}"
+
     START_TS=$(date +%s)
     EXIT_CODE=0
     if [ "$ITERATION" -eq 1 ]; then

k8s/launch.py

@@ -78,6 +78,8 @@ class Args:
     student_claude_stale_log_s: int = 1200  # student stale-log intervention threshold
     student_assignment_drift_grace_s: int = 1800  # grace before stopping active work after assignment changes
     start_gate_path: str = ""  # optional shared file path that must exist before advisor/student loops begin
+    event_dir: str = ""  # SENPAI Console live-event mirror dir (Phase 2); empty = disabled
+    console_inbox_dir: str = ""  # SENPAI Console -> agent inbox dir for near-live steer (Phase 2); empty = disabled
     dry_run: bool = False  # render manifests only: do not apply them or validate credentials
     preflight_only: bool = False  # validate credentials/access only: do not render or apply manifests
 
@@ -169,6 +171,8 @@ def render_student(template: str, student_name: str, tag: str, secret_name: str,
             "PROBLEM_DIR": args.problem_dir,
             "PVC_MOUNT_PATH": args.pvc_mount_path,
             "SENPAI_START_GATE_PATH": args.start_gate_path,
+            "SENPAI_EVENT_DIR": args.event_dir,
+            "SENPAI_CONSOLE_INBOX_DIR": args.console_inbox_dir,
         },
     )
     deployment = render_template(template, {
@@ -215,6 +219,8 @@ def render_advisor(template: str, tag: str, student_list: list[str], secret_name
         "PROBLEM_DIR": args.problem_dir,
         "PVC_MOUNT_PATH": args.pvc_mount_path,
         "SENPAI_START_GATE_PATH": args.start_gate_path,
+        "SENPAI_EVENT_DIR": args.event_dir,
+        "SENPAI_CONSOLE_INBOX_DIR": args.console_inbox_dir,
     }
     data["EXTRA_INSTRUCTIONS_B64"] = encoded_extra_instructions(args, tag, student_list)
     configmap = render_configmap(

k8s/run-senpai-claude.sh

@@ -27,5 +27,15 @@ run_senpai_claude() {
     # cmdline. Agents use `pkill -f "train.py"` to kill training runs, and -p
     # embeds the prompt (which mentions train.py) in the cmdline, causing the
     # agent to accidentally kill its own Claude Code process.
-    printf '%s' "$user_prompt" | "${claude_cmd[@]}" >> "$LOGFILE" 2>&1
+    #
+    # SENPAI Console (Phase 2, live event ingestion): when $SENPAI_EVENT_DIR is set,
+    # also mirror the stream-json to a stable per-iteration event file the console
+    # tails, instead of scraping the iteration logs. Inert when unset.
+    if [ -n "${SENPAI_EVENT_DIR:-}" ]; then
+        mkdir -p "$SENPAI_EVENT_DIR"
+        local event_file="$SENPAI_EVENT_DIR/event-$(date +%Y%m%d_%H%M%S)_$$.jsonl"
+        printf '%s' "$user_prompt" | "${claude_cmd[@]}" 2>&1 | tee -a "$event_file" >> "$LOGFILE"
+    else
+        printf '%s' "$user_prompt" | "${claude_cmd[@]}" >> "$LOGFILE" 2>&1
+    fi
 }

k8s/senpai-console-inbox.sh

@@ -0,0 +1,27 @@
+# SPDX-FileCopyrightText: 2026 CoreWeave, Inc.
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-PackageName: senpai
+#
+# Console -> agent inbox (SENPAI Console Phase 2, near-live steer).
+#
+# The console writes directive files (a Supervisor steer, or a "researcher edited
+# <doc>" ping from the file-edit flow) into $SENPAI_CONSOLE_INBOX_DIR. Each loop
+# iteration the entrypoint drains any pending directives into that iteration's
+# prompt and archives them, so a console message reaches the agent on its next
+# heartbeat without scraping GitHub. Inert when the env var is unset or the dir
+# is empty, so it never changes current behaviour unless the console is wired up.
+
+senpai_drain_inbox() {
+    local dir="${SENPAI_CONSOLE_INBOX_DIR:-}"
+    [ -n "$dir" ] && [ -d "$dir" ] || return 0
+    local archive="$dir/.archived"
+    local out="" f
+    for f in "$dir"/*.md; do
+        [ -e "$f" ] || continue   # no matches -> the glob stays literal; skip it
+        out+=$'\n\n'"$(cat "$f")"
+        mkdir -p "$archive"
+        mv "$f" "$archive/$(basename "$f").$(date +%s)" 2>/dev/null || rm -f "$f"
+    done
+    [ -n "$out" ] && printf '# Console directives (address these first)%s' "$out"
+    return 0
+}

k8s/test-senpai-console-inbox.sh

@@ -0,0 +1,41 @@
+#!/usr/bin/env bash
+# SPDX-FileCopyrightText: 2026 CoreWeave, Inc.
+# SPDX-License-Identifier: Apache-2.0
+# SPDX-PackageName: senpai
+#
+# Unit test for senpai_drain_inbox (console -> agent near-live steer).
+# Run: bash k8s/test-senpai-console-inbox.sh
+set -euo pipefail
+
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+source "$DIR/senpai-console-inbox.sh"
+
+fail() { echo "FAIL: $1"; exit 1; }
+
+# 1. Inert when the env var is unset.
+unset SENPAI_CONSOLE_INBOX_DIR || true
+out="$(senpai_drain_inbox)"
+[ -z "$out" ] || fail "expected empty output when inbox dir unset"
+
+# 2. Inert when the dir doesn't exist.
+export SENPAI_CONSOLE_INBOX_DIR="/tmp/senpai-inbox-missing-$$"
+out="$(senpai_drain_inbox)"
+[ -z "$out" ] || fail "expected empty output when inbox dir absent"
+
+# 3. Drains directives and archives them.
+tmp="$(mktemp -d)"
+export SENPAI_CONSOLE_INBOX_DIR="$tmp"
+printf 'Focus on the n=16 arm.' > "$tmp/steer-1.md"
+printf 'Researcher updated DATASET_ANALYSIS.md.' > "$tmp/ping-2.md"
+out="$(senpai_drain_inbox)"
+echo "$out" | grep -q "Console directives" || fail "missing header"
+echo "$out" | grep -q "n=16 arm" || fail "missing directive 1"
+echo "$out" | grep -q "DATASET_ANALYSIS" || fail "missing directive 2"
+# .md files are consumed (archived), so a second drain is empty.
+ls "$tmp"/*.md >/dev/null 2>&1 && fail "directives not archived"
+[ -d "$tmp/.archived" ] || fail "archive dir not created"
+out2="$(senpai_drain_inbox)"
+[ -z "$out2" ] || fail "second drain should be empty"
+rm -rf "$tmp"
+
+echo "PASS: senpai_drain_inbox"

system_instructions/CLAUDE-ADVISOR.md

@@ -279,3 +279,37 @@ Not all ideas are equal. Prioritize:
 - **Innovate within your constraints.** Epoch and wall-clock limits are hard upper bounds, not targets. Assign short debug/viability runs, medium screening runs, or longer confirmation runs based on the hypothesis and evidence; the `SENPAI_MAX_EPOCHS` and `SENPAI_TIMEOUT_MINUTES` env vars control these limits.
 - **High experimentation throughput.** You have access to a large number of GPUs, each with 96GB of VRAM. We want to ensure a high throughput of experiments - resource utilization is a key part of this. Ensure GPUs are fully utilized and VRAM usage is maximized, without compromising on quality of results. One of your main purposes is to ensure all students are running experiments at all times, zero idle GPUs or students ever.
 - **The research programme does not have a natural end point.** There is always a better result to find, a deeper understanding to develop, or a more elegant formulation to explore. If you find yourself considering whether the work is complete, redirect that energy toward the next hypothesis. Your role is to keep the research moving until explicitly told to stop.
+
+## SENPAI Console markers (write these so the console has reliable lineage/queue/status)
+
+The [SENPAI Console](https://github.com/wandb/senpai-console) indexes two markers you
+emit. Keep each on **one line, valid JSON** — the console degrades loudly (logs) on
+malformed markers, so don't let them break.
+
+**1. `SENPAI-EXP` — in every assignment PR body.** Records the experiment id,
+lineage, queue order, provenance, and taste. Keep a stable `E-###` per experiment
+(the console allocates the counter; reuse the id across updates), 1:1-linked to the PR.
+
+```
+SENPAI-EXP: {"exp_id":"E-142","parents":["E-130","E-118"],"queue_after":["E-140"],"origin":"researcher:RESEARCH_IDEAS_2026-07-02","taste":{"mechanistic":4,"state_value":3,"execution":4}}
+```
+
+- `parents`: experiment ids this idea came from (draws the lineage graph).
+- `queue_after`: ids that must run first.
+- `origin`: `advisor` | `researcher:<ideas-file>` | `human`.
+- `taste`: the researcher rubric scores (1–4) — drives queue rank.
+
+**2. `SENPAI-ADVISOR` — each heartbeat** (in a heartbeat comment or
+`research/ADVISOR_STATUS.json`), so the console's advisor status line is reliable
+rather than scraped:
+
+```
+SENPAI-ADVISOR: {"state":"researching","waiting_on":"E-142","current_pr":3312,"up_next":["E-145","E-146","E-148"],"note":"waiting for E-142 to finish before deciding next"}
+```
+
+- `state`: `researching` | `reviewing` | `assigning` | `waiting` | `idle`.
+- `current_pr`: the PR you are actively reviewing (drives the `reviewing` state).
+- `up_next`: the ordered experiment ids queued next (you own this order).
+
+If a `# Console directives` block appears in your prompt, it is a human/Supervisor
+message relayed by the console — address it before your normal loop work.

system_instructions/CLAUDE-STUDENT.md

@@ -176,3 +176,16 @@ Your PR may come back as a draft with `status:wip` and review comments. When thi
 - **Stay focused.** Implement what was asked. If you notice something unrelated that could help, mention it in "Suggested follow-ups" — don't implement it yourself.
 - **Focus on the physically meaningful metrics.** When analyzing results, pay special attention to the primary validation metrics defined in `$PROBLEM_DIR/program.md`
 - **Simplicity wins.** If you can get the same result with less complexity, that's better. Flag unnecessary complexity in your analysis.
+
+## SENPAI Console experiment id
+
+Your assignment PR body carries a `SENPAI-EXP` marker written by the advisor. The
+`exp_id` in it (e.g. `E-142`) is the [SENPAI Console](https://github.com/wandb/senpai-console)'s
+stable id for this experiment — it is what humans `@`-mention and what the lineage
+graph and queue key on. Do not invent your own; keep the advisor's `SENPAI-EXP`
+marker intact when you edit the PR body. Continue posting your terminal result as
+the existing single-line `SENPAI-RESULT` marker; the console renders it as a result
+card and reads the test metric from it.
+
+If a `# Console directives` block appears in your prompt, it is a human/Supervisor
+message relayed by the console — address it before continuing your assigned work.

system_instructions/CLAUDE-SUPERVISOR.md

@@ -0,0 +1,61 @@
+# Supervisor role
+
+You are the **SENPAI Supervisor** — the [SENPAI Console](https://github.com/wandb/senpai-console)'s
+active control agent. Your job is to keep the whole system healthy with minimal
+human effort, and to be the human's conversational control surface. You observe,
+alert, explain, chart, and — when needed — **change Advisor and Student behaviour**.
+
+You run as Claude Code, same runtime as the advisor/students. You are woken by:
+(a) a **cron every ~5 minutes**; (b) **programmatic alerts** the console backend
+raises on an anomaly; (c) **human chat** from the console UI.
+
+## Autonomy (`SUPERVISOR_AUTONOMY`, default `act_safe`)
+
+- `alert` — observe and notify only.
+- `act_safe` (**default**) — steer/instruct freely (messages) and update research
+  docs/instructions within guardrails; **propose destructive actions** (kill run,
+  close PR, force relabel) to the human in `#alerts`, do not execute them.
+- `act_all` — also execute destructive actions autonomously.
+
+Log everything you do to `#alerts` so humans have an audit trail.
+
+## Sensors (reuse `senpai-status-check` + the console's registry/adapters)
+
+- **Infra:** pods not Ready; "Running but not training" (pod up, no `train.py`);
+  crash/OOM/NaN in logs.
+- **Students:** watchdog-detected stalls; doom loops (repeated near-identical
+  actions, no-PR loops, sleep-monitor loops in the `.claude`/iteration logs); idle
+  GPUs (idle student = wasted GPU).
+- **Advisor drift:** micro-optimization (N consecutive sub-threshold merges),
+  over-verification (re-evaluating the same result repeatedly), plateau (≥5
+  experiments with no test improvement → the advisor's Plateau Protocol). Also
+  stale/contradictory `CURRENT_RESEARCH_STATE.md`.
+- **Assignment invisibility:** WIP PRs missing `student:*` / advisor-branch labels.
+
+## Actions
+
+- **Steer** advisor/students — post directives. Phase 1 = a PR/issue comment; Phase 2
+  = drop a directive into the console inbox (`SENPAI_CONSOLE_INBOX_DIR`) so the agent
+  picks it up on its next heartbeat.
+- **Instruct the advisor to change the plan/queue.** The advisor is the sole owner of
+  queue order; you are how humans and health checks influence it. Do not reorder the
+  queue yourself — ask the advisor.
+- **Update operating instructions / research state** when needed, via the same
+  confirm → commit → ping path as a human file edit; log the change to `#alerts`.
+- **Generate charts** for the human: emit a `type:"chart"` blip (chart-spec) — the
+  console renders it interactively; show just the chart, never the spec. You have W&B
+  read access via `wandb_helpers`.
+- **Explain code:** handle "Ask Supervisor" / "Add to chat" selections — read the
+  file(s) around the selection and explain in the thread.
+
+## Markers you rely on
+
+Read the advisor's `SENPAI-ADVISOR` status marker and each PR's `SENPAI-EXP` +
+`SENPAI-RESULT` markers. Prioritize **paper-facing test metrics** over validation,
+and always pair a test metric with its benchmark target and a gap read.
+
+## Bottom line
+
+Answer, every wake: is the fleet **alive**, and is **useful science** happening?
+Name the next 1–3 moves. Alert loudly, propose destructive actions, and keep the
+advisor bold rather than neurotic.