Add campaign operational supervisor

[morganmcg1]

Summary

Adds an opt-in, campaign-scoped operational supervisor for Kubernetes Senpai
deployments. It wakes every 15 minutes, builds one timestamped health snapshot,
retains the latest three snapshots, and runs a fresh bounded OpenHands turn. A
separate capability-free assessment runs at most every six hours to detect
sustained research drift against the advisor guidance actually deployed in the
managed role.

This head contains current main at 901be4b3 and the complete PR #3472 head
at 8feef8b8. The release candidate is 23b40e7785f10c2d8b412a4a51c3c57d58553dba.

Operational flow

1. Collect only campaign-owned evidence: open PRs based on this advisor branch,
   exact W&B run IDs recorded by configured students, and exact labeled role
   pods.
2. Reduce evidence to configured identities, allowlisted states, counts,
   timestamps, measurements, and opaque fingerprints. Missing evidence remains
   unknown; raw PR/W&B/log/error text is not persisted or placed in the
   privileged operational prompt.
3. Run one fresh operational turn with an unrestricted native terminal and one
   typed senpai_operations tool.
4. Persist the snapshot, cadence, bounded mutation audit, repair receipts, and
   completion tombstones in a campaign-dedicated SQLite PVC.
5. When the six-hour slot is due, run a separate non-persistent assessor with
   raw research evidence, no credentials or action capabilities, and one
   enum-only output tool. Only strategic_drift maps to a fixed, code-owned,
   state-bound advisor-principles nudge.

Cycles are single-flight and never overlap. A long wake delays the next wake;
it does not create concurrent supervisors or an accumulating model history.

Authority and trust boundaries

• The credentialed control container owns campaign state, model/GitHub/W&B
  credentials, and namespace-scoped Kubernetes credentials. Model-visible
  operation results contain only closed statuses, counts, UUIDs, generations,
  booleans, and fingerprints; backend text and identifiers are not echoed.
• Native terminal commands execute over an abstract Unix socket in a separate
  shell container with a private mutable home/workspace and no Kubernetes
  token, GitHub token, W&B key, provider secret, campaign state, shared PID
  namespace, or access to the control container filesystem. Senpai does not
  filter Git or shell syntax there.
• The unrestricted shell remains a deliberate residual trust boundary: if the
  model chooses to read arbitrary PR text, logs, files, or command output, that
  data must be treated as hostile and can reintroduce prompt-injection risk.
• Typed actions can target only the configured advisor/students. Nudges,
  same-UUID context resets, and controller restarts are state-bound,
  idempotent, cooldown-limited, and durably audited.
• A context reset preserves the conversation UUID, append-only raw trace,
  workspace, and pending events while starting a clean active model branch. It
  never deletes selected messages or rewinds an event log.
• A planned restart is consumed only by the process owner and is refused while
  jobs/delegations are active or activity is unknown. The source generation is
  rechecked immediately before replacement.
• Arbitrary role repairs execute in secret-free role sidecars. PID 1 first
  quiesces the controller and descendants, proves that no TCP listener remains,
  authenticates over SO_PEERCRED, and issues a one-use resume capability.
  Commands are exact-target, timeout/output bounded, replay-safe, and retained
  as durable receipts/tombstones.

Kubernetes release and durability

• Supervisor state uses a dedicated, explicitly acknowledged SQLite-safe RWO
  or RWOP PVC, not the student-writable dataset volume.
• A supervisor-only upgrade does not restart advisor/student research roles
  when their versioned management and repair protocols are compatible.
• Before mutation, the launcher pins cluster/namespace identity, acquires a
  campaign transaction Lease, and captures the five mutable resources in a
  private rollback journal. Failed upgrades quiesce the failed Deployment,
  restore security resources first, then restore and verify the old Deployment.
• Restore uses raw namespaced Kubernetes POST/PUT endpoints so kubectl cannot
  rewrite last-applied-configuration; the captured postcondition remains
  exact while PUT retains optimistic resourceVersion concurrency.
• Immutable Secret/ConfigMap artifacts and persistent SQLite state are never
  rolled back. Stale Lease epochs cannot restore over a newer release.
• The supervisor kill switch removes only its Deployment and keeps research
  roles, repair sidecars, state, capacity, RBAC, and network policy intact.
• Supervisor-capable pods require an enforcing CNI and a NetworkPolicy that
  denies IPv4/IPv6 link-local metadata access while retaining bounded DNS
  exceptions. The launcher requires an explicit dedicated-namespace and CNI
  acknowledgement.

Compatibility and launcher hardening

• Legacy delegation and training event payloads remain loadable or are migrated
  before OpenHands model validation, so existing advisor/student conversations
  can resume across the upgrade.
• Prompt composition, campaign-scope changes, research-review failures, SQLite
  initialization, inbox adoption, interrupted operations, and failed-turn
  delivery have bounded, restart-idempotent recovery paths rather than retry
  loops.
• A repair response is now a readiness barrier: PID 1 retains the result in an
  anonymous, unlinked file descriptor until the worker and descendants are
  reconciled, the listener is rebound, and the idle heartbeat is published.
  Startup health also remains non-ready until the initial listener is live.
• Runtime CI exports the locked production dependency graph with the same
  heavyweight training packages pruned by the advisor image. The advisor build
  explicitly imports W&B, and render-only canary helpers lazily load the
  CLI-only argument parser so host-side rollback capture has no hidden parser
  dependency.
• PR Add Docker and AWS GPU launch backends #3472's Docker, AWS GPU, native, and AWS Mac launchers are integrated.
  The operational supervisor itself remains Kubernetes-only; other backends
  reject the flag until their scoped management transports are implemented.
• AWS Mac launch state is campaign-bound and authenticated with exact SSH host
  keys. Campaign-owned SSH security groups remove and verify default egress
  before instance launch. Cleanup waits for confirmed EC2 termination and
  retains access material while any node is unresolved; it does not release
  Dedicated Hosts. Native teardown uses an exact role inventory and retries
  loaded LaunchDaemon bootout even after plist removal.

Prior review findings

The substantive review of the earlier e581371 head is now addressed:

• the real supervisor entrypoint boots from an immutable checkout and rollout
  readiness is verified;
• unrestricted shell authority is separated from command secrets and role
  credentials;
• durable supervisor state is isolated from student workloads;
• oversized research review input and review failures are bounded;
• legacy persisted delegation/training payloads remain resumable;
• empty/default namespaces fail the dedicated-namespace preflight;
• boot, repeated-wake, campaign-scope, final delegation recheck, prompt
  substitution, compatibility, and documentation contracts have regression
  coverage.

Validation

• Authoritative host suite: 1594 passed, 22 skipped.
• GitHub Linux runtime suite on the release candidate: 1611 passed, 6 skipped.
• Production-like Linux repair-executor suite: 26 passed, including immediate
  sequential requests, shutdown-during-spawn, crashed-worker reconciliation,
  and anonymous result-channel coverage.
• Student production image smoke passed the immutable runtime and isolated
  repair lifecycle.
• Exact-SHA Kind canary on 23b40e7785f10c2d8b412a4a51c3c57d58553dba
  passed enforcing metadata isolation, terminal wake isolation, durable typed
  repair, authenticated quiescence, process-owner restart, failed-upgrade exact
  rollback, interrupted-operation recovery, dedicated SQLite persistence, and
  cleanup. The exact canary cluster was removed by the workflow.
• Focused final rollback/canary contract suite: 47 passed.
• uv lock --check: 405 packages resolved.
• Python compilation, shell syntax, and git diff --check passed.
• Independent final reviews of the AWS/native lifecycle, privileged research
  boundary, repair failure paths, response/rebind barrier, portable anonymous
  result channel, and exact raw-API rollback reported no remaining blockers.

Current support

The operational supervisor is production-ready for the documented Kubernetes
topology. Docker, AWS GPU, and AWS Mac supervisor transports remain explicitly
planned and disabled; the README/SPEC describe the required broker boundaries
without claiming that those transports exist today.