diff --git a/changelog.d/changed-substrate-runtime-docs.md b/changelog.d/changed-substrate-runtime-docs.md new file mode 100644 index 0000000000..fadfda564d --- /dev/null +++ b/changelog.d/changed-substrate-runtime-docs.md @@ -0,0 +1 @@ +- Document Agent Substrate as the opt-in fourth `sandbox.runtime` (ADR-0026 phase 2): ADR amendment with the contract mapping, deviations and when to choose it over OpenShell, a `sandbox-isolation.md` section, `deploy/substrate.md` install and verification notes with the hive cluster survey, and the integrations table entry. diff --git a/src/deploy/README.md b/src/deploy/README.md index 6b3fb54d45..db3234b79d 100644 --- a/src/deploy/README.md +++ b/src/deploy/README.md @@ -20,6 +20,7 @@ Use it indirectly through the dashboard terminal link. If a terminal pane says i - `k8s/` — the Kustomize base (`kustomization.yaml`) with namespace, deployment, service, PVC, Secret, ConfigMap and RBAC manifests, plus the custom error backend (`error-pages.yaml`), the hub backup CronJob (`backup-cronjob.yaml`) and the node-prep DaemonSet (`node-prep/`), which are applied separately. The OpenShift Route lives in `kustomize/overlays/openshift/`. - `inference/` — sample in-cluster OpenAI-compatible inference deployment and RBAC. - `openshell.md` — cluster-side notes for the opt-in `sandbox.runtime: openshell` (ADR-0026): prerequisites, the upstream Helm install, the gateway settings and PVC labels hive's runtime depends on, and how to verify. No manifests are vendored. +- `substrate.md` — cluster-side notes for the opt-in `sandbox.runtime: substrate` (ADR-0026 phase 2): the Agent Substrate operator install, WorkerPool and SandboxConfig, the pod-certificate feature gates, the router client bundle hive needs, and how to verify. No manifests are vendored. - `docker-compose.architect.yaml`, `hive-quickstart.yaml`, `hive-level*.yaml`, `architect-only.yaml`, `hive.yaml` — example deployment/configuration manifests. - `blue-green-deploy.sh`, `bootstrap-lxc.sh`, `create-lxc.sh` — operational scripts for **Docker Compose** deployments (LXC bootstrap and a blue-green Compose upgrade). Not "non-Kubernetes" generally: none of the three shells out to `podman` or reads `HIVE_DEPLOY_RUNTIME`, so none of them applies to the Quadlet stack in `quadlet/` below. See [Deployment helper scripts](../docs/deployment-scripts.md) for each script's scope and where a Podman operator should go instead. - `quadlet/` — Podman Quadlet units for the standalone deployment (ADR-0017). `hive.container` and `hive-data.volume` start the Hive service under `systemd`, rootful or rootless; `hive.network` and `hive-gateway.container` put the authenticating gateway in front of it with the same published-port split as the Compose stack — 3001 published, 7681 never. See [the operator guide](../docs/podman-standalone-quadlet.md). Docker Compose remains the default runtime and is unaffected. diff --git a/src/deploy/substrate.md b/src/deploy/substrate.md new file mode 100644 index 0000000000..941d32d2c3 --- /dev/null +++ b/src/deploy/substrate.md @@ -0,0 +1,118 @@ +# Installing Agent Substrate for `sandbox.runtime: substrate` + +Cluster-side notes for the opt-in Substrate sandbox runtime +([ADR-0026 phase 2](../docs/adr/0026-sandbox-runtime-selection.md#implementation-notes-phase-2-agent-substrate), +epic [#17695](https://github.com/hivecommons/hive/issues/17695); operator view +in [sandbox-isolation.md](../docs/sandbox-isolation.md#running-sandboxed-kicks-in-a-substrate-actor)). +Hive installs nothing: Substrate is a cluster-admin install, once per cluster, +and hive only talks to its Control API and router. Upstream is the source of +truth for the install — [agent-substrate/substrate → docs/guides/operator-install.md](https://github.com/agent-substrate/substrate/blob/main/docs/guides/operator-install.md), +the [API guide](https://github.com/agent-substrate/substrate/blob/main/docs/reference/api-guide.md) +and [sandbox classes](https://github.com/agent-substrate/substrate/blob/main/docs/reference/sandbox-classes.md) — +and no manifests are vendored here; pin the release you install and record it +alongside the hive config. + +## Prerequisites to check before installing + +| Requirement | Why | How to check | +|---|---|---| +| Kubernetes with `ClusterTrustBundle`, `ClusterTrustBundleProjection` and `PodCertificateRequest` feature gates and `certificates.k8s.io/v1beta1` enabled | actor and client identities are pod certificates (SPIFFE); the podcert controller does not start without them. Still off by default as of Kubernetes 1.36 | `kubectl get --raw /metrics \| grep 'kubernetes_feature_enabled{name="PodCertificateRequest"'` and `kubectl api-resources --api-group=certificates.k8s.io`. Managed control planes (OKE, LKE) need the provider to enable them; OpenShift needs a `FeatureGate` CR | +| Standard Linux nodes with **cgroup v2** for the `gvisor` class | `ateworker-gvisor` runs `runsc` inside the worker pod; there is no kernel floor beyond ≥ 5.8 | `kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kernelVersion}'`; `stat -fc %T /sys/fs/cgroup` on a node → `cgroup2fs` | +| `/dev/kvm` on nodes for the `microvm` class | Kata on Cloud Hypervisor guests; `atelet` advertises the `ate.dev/kvm` device only where it exists | `kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.allocatable.ate\.dev/kvm}{"\n"}{end}'` after install | +| Nodes trusted to run worker pods with `SYS_ADMIN` and `NET_ADMIN` | workers are `privileged: false` but keep those two for mounts and networking | on OpenShift, an SCC granting them to the `ate-system` worker ServiceAccount | +| A CNI that **enforces** egress `NetworkPolicy` | Substrate's `atenet` dataplane carries actor egress; the worker pod itself must not be a side door | create a deny-all policy in a scratch namespace and confirm a connection is refused | +| PostgreSQL and S3-compatible object storage | Control API state; actor snapshots (`snapshot_on_idle`) | `ate-setup` deploys bundled ones for a lab; use managed services in production | +| Cluster-admin for the install | CRDs (`workerpools`, `sandboxconfigs`, `egresspolicies`, …), ValidatingAdmissionPolicies, RBAC, PKI | — | + +### Which hive clusters qualify (survey 2026-11-11) + +| cluster | kernel / Kubernetes | `gvisor` | `microvm` | +|---|---|---|---| +| hive-oke | OL9, UEK8 6.12, k8s 1.34.10 | yes, once the podcert feature gates are enabled on the OKE control plane | needs KVM-capable shapes (bare metal or nested virtualisation) | +| lke648397, hive-ci | Debian 13, 6.12 | yes; confirm the feature gates — LKE exposes no control-plane flags | KVM to confirm per plan | +| vllm-d, a-ks-wec2 | RHCOS 9.x, 5.14 | possible: SCC for the worker capabilities + `FeatureGate` CR (TechPreviewNoUpgrade) | only on KVM-capable nodes | + +## Install + +Build or download `ate-setup` from a pinned release of +`github.com/agent-substrate/substrate`, write the install document, and deploy: + +```yaml +# substrate-install.yaml +apiVersion: install.ate.dev/v1alpha1 +kind: SubstrateInstall +namespace: ate-system +atenet: + dataplane: envoy +images: + repo: /substrate + tag: +``` + +```shell +ate-setup deploy ate-system --config substrate-install.yaml \ + --credential-provider='{"name":"k8s.io"}' \ + --ingress-allowed-spiffe-ids=spiffe://cluster.local/ns//sa/ +``` + +`--ingress-allowed-spiffe-ids` must name hive's own pod identity: that is what +lets hive open `CONNECT` tunnels through `atenet-router` to the bridge in each +actor. Unknown keys in the document are rejected, not ignored. + +Then create the pieces hive's runtime depends on, in Substrate terms: + +- **An atespace** for hive (`sandbox.substrate.atespace`, default `default`), + and a principal for hive's token (`token_env` / `token_file`) that may + create and delete `ActorTemplate`, `Actor` and `EgressPolicy` and list + workers in it. `kubectl ate` shows the exact RBAC verbs for the release. +- **A `WorkerPool`** with `spec.sandboxClasses[].name: gvisor` (or `microvm`) + resolving the default `SandboxConfig` (`gvisor-default`), labelled so that + `sandbox.substrate.worker_selector` matches it; hive only uses pools whose + workers report `ACTIVE` and advertise a sandbox class. Taint the node pool + `ate.dev/sandboxClass=:NoSchedule` to reserve it; worker pods + tolerate it. +- **Projected files in the hive pod**: a ServiceAccount token with audience + `api.ate-system.svc` (→ `token_file`), the Control API CA via + `clusterTrustBundle` with signer `servicedns.podcert.ate.dev/identity` + (→ `ca_file`), and a `podCertificate` with signer + `podidentity.podcert.ate.dev/identity` (→ `router_client_bundle`) plus the + router CA (→ `router_ca_file`). The authentication guide upstream has the + volume stanza. +- **A digest-pinned sandbox image** containing `crux` (for the + `substrate-bridge` entrypoint) and the agent CLI, set as + `sandbox.substrate.image`; tags are refused at boot and at kick. Publishing + it is tracked in [#17697](https://github.com/hivecommons/hive/issues/17697). +- **Egress**: hive writes its own `EgressPolicy` per actor (`github.com`, + `api.github.com`, `allow_hosts`). Do **not** also configure Substrate's + credential injection for GitHub — hive's MITM proxy brokers credentials and + the two would conflict. + +## Verify + +```shell +kubectl -n ate-system get deploy,ds # ateapi, atenet-router, podcert controller, atelet ready +kubectl get crd | grep ate.dev # workerpools, sandboxconfigs, egresspolicies, … +kubectl get workerpool -A # at least one pool with ACTIVE workers +kubectl get sandboxconfig gvisor-default -o yaml | grep defaultVersion +kubectl get nodes -o custom-columns=NAME:.metadata.name,KVM:.status.allocatable.ate\.dev/kvm +kubectl ate create actor smoke -a --template && kubectl ate delete actor smoke -a +``` + +From the hive side, start with `runtime: substrate` and watch the boot log: the +preflight warns when the Control API is unreachable, has no matching `ACTIVE` +worker, or its workers report no sandbox class. Then run one kick and confirm +in the router log that the actor's SPIFFE id made exactly the allowed +connections and that a connect to any other host was refused — that denial is +the point of the runtime. Latency and image numbers go back into this page +from the prototype ([#17697](https://github.com/hivecommons/hive/issues/17697)); +the fuller prover (CRDs, operator, feature gates, KVM) is +[#17698](https://github.com/hivecommons/hive/issues/17698). + +## Upgrades + +Substrate is pre-1.0; follow its version-skew policy (N ↔ N+1 only) and pin +the release and the sandbox image by digest. A `SandboxConfig` upgrade adds a +`runsc` version and keeps the previous one so suspended actors still restore. +Hive's generated client (`crux/sandbox/substrateapi`) tracks the fields it +uses; bump it with `generate.sh` against the matching upstream tag when the +Control API changes. diff --git a/src/docs/adr/0026-sandbox-runtime-selection.md b/src/docs/adr/0026-sandbox-runtime-selection.md index 3a6e062fde..153111ef6f 100644 --- a/src/docs/adr/0026-sandbox-runtime-selection.md +++ b/src/docs/adr/0026-sandbox-runtime-selection.md @@ -1,6 +1,6 @@ # ADR-0026: Sandbox runtime selection -Status: Accepted (phase 1 implemented, runtime opt-in) — #17642, hivecommons/hive#17646 +Status: Accepted (phase 1 OpenShell implemented, runtime opt-in — #17642, hivecommons/hive#17646; phase 2 Agent Substrate implemented, runtime opt-in — #17695, hivecommons/hive#17741) ## Context @@ -218,6 +218,74 @@ runs on, read on 2026-11-10: LKE (`lke648397`, `hive-ci`) run 6.12 on Debian 13 with Calico and qualify; OKE UEK7 (5.15), and the RHCOS 9.x OpenShift clusters (5.14) do not. +## Implementation notes (phase 2: Agent Substrate) + +Phase 2 (#17695, hivecommons/hive#17741) adds a **fourth** runtime, +`sandbox.runtime: substrate`, behind the same `sandbox.Launcher` seam. It is +[Agent Substrate](https://github.com/agent-substrate/substrate) (CNCF Sandbox, +donated by Google, Apache-2.0): a Kubernetes-hosted *actor* runtime where an +`ActorTemplate` (image, command, env, limits, volumes) is instantiated as an +`Actor` through a gRPC Control API (`ate.control.v1`), scheduled on a +`WorkerPool` whose pods run the workload under gVisor (`runsc`, class +`gvisor`) or in a KVM microVM (Kata on Cloud Hypervisor, class `microvm`). All +egress leaves through `atenet-router` under an `EgressPolicy`, and every actor +holds a SPIFFE identity issued through Kubernetes pod certificates. It was not +in the 2026-10-25 comparison; the write-up that led here is the Substrate +vs. crux research note, and the operator view is in +[sandbox-isolation.md](../sandbox-isolation.md#running-sandboxed-kicks-in-a-substrate-actor) +and [deploy/substrate.md](../../deploy/substrate.md). Like OpenShell it is +opt-in and inert unless selected. + +### How it maps onto the contract + +| Contract item | Substrate | What phase 2 does | +|---|---|---| +| Kernel-enforced confinement | gVisor user-space kernel or a microVM; worker pods are `privileged: false` but keep `SYS_ADMIN`/`NET_ADMIN` | selects the class through `worker_selector`; no change to hive | +| Default-deny egress | `atenet-egress` `EgressPolicy` per actor; rules are TLS-passthrough or terminate per host:port, no per-method read-only mode | `SubstrateEgressPolicy`: `github.com` and `api.github.com` on 443 plus `sandbox.substrate.allow_hosts`, TLS passthrough; everything else (and all UDP) denied | +| Credentials never in the sandbox | zero-credential model: actors have a SPIFFE id, credentials are injected at the router | unchanged hive brokering — the agent still gets `HTTPS_PROXY` to the hive MITM proxy (ADR-0002); Substrate's injection is **not** used, the two are complementary | +| Workspace in, commits out | `durableDir` volumes and object-storage snapshots; no exec/attach API | an in-actor HTTP **bridge** (`crux substrate-bridge`) reached through the router's mTLS `CONNECT` tunnel: push tar → `POST /run` → pull tar; the hive-side workspace stays authoritative | +| Cleanup and timeouts | `DeleteActor` / `SuspendActor` / `DeleteActorTemplate` | one template + actor per kick, both deleted on success, failure and timeout (fresh bounded context, `ExitCode -1`, 16 MiB cap, as `job`/`openshell`) | + +### Deviations from Substrate's reference model + +| Substrate assumes | Phase 2 does | +|---|---| +| long-lived actors and reusable templates | per-kick `ActorTemplate` + `Actor`, named `hive-kick--` | +| Go client `go.ate.dev/substrate` | **not imported**: it needs Go 1.27 (hive is on 1.26) and a very large module graph. A trimmed, wire-compatible proto subset is generated in-repo (`crux/sandbox/substrateapi`, field numbers pinned by tests); zero new Go modules, +578 KB on the `crux` binary | +| image by tag or digest | digest-pinned `sandbox.substrate.image` only, enforced at boot (`AgentSandboxGateWarnings`) and at kick | +| no `policy_profile` concept | passed to the actor as `HIVE_POLICY_PROFILE`; Substrate has no annotations | +| snapshot/resume as the normal lifecycle | `snapshot_on_idle: true` suspends (snapshot to object storage) instead of deleting; resume-on-next-kick is not implemented (#17697) | +| `api.github.com` read-only | TLS passthrough like `github.com`; the actor holds no token and hive does every write after the kick | + +### Cluster prerequisites and which hive clusters qualify + +Substrate's real requirement is **not** a kernel floor: the gVisor class runs +`runsc` inside the worker pod on any standard Linux node with cgroup v2; the +microVM class needs `/dev/kvm` (nested virtualisation on cloud shapes). Both +need the operator (`ate-setup deploy ate-system`: Control API, `atelet` +DaemonSet, `atenet-router`, podcert controller, PostgreSQL, snapshot object +storage) and Kubernetes with the `ClusterTrustBundle`, +`ClusterTrustBundleProjection` and `PodCertificateRequest` feature gates — still +off by default as of 1.36 — because actor identities are pod certificates. +Survey on 2026-11-11: + +| cluster | kernel / Kubernetes | `gvisor` class | `microvm` class | +|---|---|---|---| +| hive-oke | OL9, UEK8 6.12, k8s 1.34.10 | yes once the feature gates are on (OKE control plane: needs a support request or a self-managed control plane) | needs KVM-capable (bare-metal or nested-virt) shapes | +| lke648397, hive-ci | Debian 13, 6.12 | yes; LKE exposes no control-plane flags, so the podcert gates are the blocker — check with `kubectl get --raw /metrics \| grep kubernetes_feature_enabled` | KVM to confirm per plan | +| vllm-d, a-ks-wec2 | RHCOS 9.x, 5.14 | possible: the kernel is fine, but the worker's `SYS_ADMIN`/`NET_ADMIN` needs an SCC and the gates a `FeatureGate` CR (TechPreviewNoUpgrade) | only on KVM-capable nodes | + +### When to choose it over OpenShell + +- Choose **OpenShell** when the cluster already runs a ≥ 6.2 kernel with an + enforcing CNI and you want per-binary, per-method egress policy and the + shared-PVC workspace with no image changes. +- Choose **Substrate** when you want a stronger boundary than Landlock + + seccomp (gVisor or a VM), snapshot/suspend of idle agents, or SPIFFE + identities for the sandboxed workload, and you can run its operator, + PostgreSQL and object storage. Its egress policy is coarser (host:port, no + read-only API mode) and the workspace is copied, not shared. + ## Consequences - Hive gains a sandbox boundary that is checked before the agent runs, without @@ -233,8 +301,12 @@ with Calico and qualify; OKE UEK7 (5.15), and the RHCOS 9.x OpenShift clusters hive's policy. - Telemetry must be disabled explicitly. A preflight check should refuse a gateway that reports it enabled. +- Substrate (phase 2) adds a second operator-side stack (Control API, + atelet, atenet-router, PostgreSQL, object storage) and a copied workspace; + the hive-side credential model is untouched. Its preflight (#17698) and + prototype (#17697) follow the OpenShell ones. - If the prototype fails a gate, this record is rewritten to decide "no change" with the measured reason, and the Podman and Job runtimes remain the only ones. -Refs #2804, #6311, #13806, #14092. +Refs #2804, #6311, #13806, #14092, #17642, #17695. diff --git a/src/docs/integrations.md b/src/docs/integrations.md index f4479b2e3c..fed5bf8cad 100644 --- a/src/docs/integrations.md +++ b/src/docs/integrations.md @@ -94,6 +94,7 @@ per-backend confinement matrix. Podman logoRootless Podman agent sandboxOpt-insandbox-isolation.md Kubernetes logoKubernetes Job sandbox runtimeOpt-insandbox-isolation.md NVIDIA logoNVIDIA OpenShell sandbox runtimeIn progress (opt-in, phase 1)sandbox-isolation.md · #17642 +CNCF logoAgent Substrate sandbox runtime (CNCF Sandbox)In progress (opt-in, phase 2)sandbox-isolation.md · #17695 OpenSandboxAAIF OpenSandbox (evaluation)Planned#14092 diff --git a/src/docs/sandbox-isolation.md b/src/docs/sandbox-isolation.md index 55edeb2784..e66150d406 100644 --- a/src/docs/sandbox-isolation.md +++ b/src/docs/sandbox-isolation.md @@ -286,6 +286,49 @@ The gateway, workspace, token, TLS and claim fields are cluster facts and fall b **Limitations (phase 1).** Kubernetes driver only (no Podman-driver path for standalone hosts yet); no Secret references into the sandbox; `policy_profile` is advisory; kernel and claim-label prerequisites are not preflighted; latency and image size are not yet measured ([#17643](https://github.com/hivecommons/hive/issues/17643)). +## Running sandboxed kicks in a Substrate actor + +`sandbox.runtime: substrate` runs an agent's sandboxed kicks as an [Agent Substrate](https://github.com/agent-substrate/substrate) **actor** ([ADR-0026 phase 2](adr/0026-sandbox-runtime-selection.md#implementation-notes-phase-2-agent-substrate), epic [#17695](https://github.com/hivecommons/hive/issues/17695)). Substrate is a CNCF Sandbox project donated by Google: a `WorkerPool` of pods runs workloads under gVisor (`runsc`) or in a KVM microVM, every actor gets a SPIFFE identity, and all egress leaves through `atenet-router` under an `EgressPolicy`. It is the fourth runtime and, like OpenShell, opt-in; nothing changes for a hive that does not select it. + +Everything above about the kick path still holds; only step 2 changes. The launcher is `sandbox.SubstrateLauncher`, which talks to Substrate's gRPC **Control API** (`ate.control.v1`) with a trimmed, wire-compatible client generated in-repo (`crux/sandbox/substrateapi`; the upstream Go module needs a newer Go than hive builds with). + +```yaml +agent_sandbox: + runtime: substrate + substrate: + endpoint: api.ate-system.svc:443 # Control API, host:port or https:// + atespace: hive # Substrate's tenancy unit, default "default" + token_env: ATE_TOKEN # bearer token env var (or token_file for a projected SA token) + ca_file: /var/run/ate/ca.crt # Control API CA; insecure_tls: true only for a lab + router_url: https://atenet-router.ate-system.svc:8444 + router_client_bundle: /var/run/ate/credential-bundle.pem # hive's own mTLS client identity + router_ca_file: /var/run/ate/router-ca.crt + image: ghcr.io/hivecommons/crux-sandbox@sha256:… # digest-pinned; tags are refused + cpu_millicores: 2000 + memory_bytes: 4294967296 + worker_selector: {ate.dev/sandboxClass: gvisor} # pick the pool (gvisor | microvm) + snapshot_on_idle: false + allow_hosts: [models.example.com, hub.example.com:8443] + timeout_s: 1800 +agents: + reviewer: + sandbox: + substrate: + snapshot_on_idle: true # per-agent overrides merge onto the global block +``` + +**One actor per kick.** Hive creates an `ActorTemplate` (one container running the bridge command, the workspace as a `durableDir`, the operator's limits and `worker_selector`) and an `Actor` from it, both named `hive-kick--`, attaches the egress policy, waits for the actor to report `RUNNING`, and then talks to a small HTTP **bridge** inside the actor (`crux substrate-bridge`, overridable with `bridge_command`, port `bridge_port`) through the router's mTLS `CONNECT` tunnel: it pushes the workspace as a tar, runs the kick with the sanitized environment, and pulls the workspace back. The hive-side workspace remains the source of truth and feeds the push broker exactly as for Podman. Substrate has no exec API and hive shares no PVC with the worker nodes, which is why the workspace is copied rather than mounted; the copy is bounded by the kick timeout. + +**The egress policy.** Hive derives an `atenet-egress` policy from the hosts its proxy brokers for a Podman sandbox — `github.com` and `api.github.com` on 443 — plus `allow_hosts` as `host` or `host:port`, all as TLS passthrough; URLs, userinfo and wildcards are rejected at validation, before any actor is created. Everything else, including all UDP and raw IP connects, is denied. Substrate's policy language has no per-method read-only mode, so `api.github.com` is reachable; the actor holds no token (`SanitizedEnv`) and hive performs every GitHub write itself after the kick. `network_mode: none` drops every rule. Credential brokering is unchanged: the agent inside the actor still receives `HTTPS_PROXY` pointing at the hive MITM proxy ([ADR-0002](adr/0002-github-proxy.md)); Substrate's own zero-credential injection is complementary and not used by hive. + +**Lifecycle.** Success, failure and the kick timeout all delete the actor and then its template, on a fresh bounded context when the kick's own has expired, with exit code `-1` and `context deadline exceeded` on timeout, like a timed-out Job. With `snapshot_on_idle: true` the actor is **suspended** (snapshotted to Substrate's object storage) instead of deleted; the template is still deleted and the snapshot is not yet resumed on the next kick ([#17697](https://github.com/hivecommons/hive/issues/17697)). `policy_profile` is passed to the actor as `HIVE_POLICY_PROFILE` for in-actor tooling; Substrate has no annotations. + +**What hive checks, and what it cannot.** At boot and on reload (`config.AgentSandboxGateWarnings`) a substrate-runtime agent is warned about, and its kick refused, when `endpoint` or `router_url` is missing or `image` is not digest-pinned. Hive also preflights each distinct Control API (read-only `ListWorkers`, cached for five minutes, bounded to 30 s) and warns when it is unreachable, has no `ACTIVE` worker matching `worker_selector`, or its workers advertise no sandbox class. It cannot see the CRDs, the operator, the pod-certificate feature gates or `/dev/kvm` from inside its pod; those are the operator prerequisites in [deploy/substrate.md](../deploy/substrate.md) and the prover in [#17698](https://github.com/hivecommons/hive/issues/17698). + +**The terminal pane.** A substrate-runtime agent has no tmux session and, until [#17699](https://github.com/hivecommons/hive/issues/17699), no live pane: the ▶ terminal shows the last persisted transcript like the job viewer between kicks. + +**Limitations (phase 2).** Workspace copied, not shared; `crux substrate-bridge` entrypoint and a published sandbox image land with the prototype ([#17697](https://github.com/hivecommons/hive/issues/17697)); no resume of suspended actors; `api.github.com` not read-only at the policy layer; latency and image size not yet measured. + ## Health and metrics for sandboxed agents A sandboxed agent (Podman or Job runtime) has no tmux session, no in-pod process and no `pluk` log between kicks: the manager parks it in the `idle` state and each kick runs to completion in a container or Job. The dashboard therefore must not read "not running" as "session down" for these agents, and the token collector cannot find their CLI session files because the CLI's home directory lives in the sandbox, not in the hive pod. @@ -315,7 +358,7 @@ On-demand agents, paused agents and `pause`/`continuous` cadences are never over - The Job runtime is a first slice: one Job per kick, workspace shared through a PVC, no per-kick Secret minting. A clone-inside-the-Job transport for clusters without ReadWriteMany storage is future work. - Live Podman execution is covered by skip-when-absent tests; CI still needs a rootless-Podman runner lane for always-on integration coverage. - Sandboxed inference depends on an operator-provided restricted network/proxy policy. `none` is stronger but not yet usable for all model backends. -- A stronger runtime than plain containers is under evaluation: [ADR-0026](adr/0026-sandbox-runtime-selection.md) (Proposed) compares NVIDIA OpenShell and OpenSandbox against this contract. +- Stronger runtimes than plain containers ship opt-in: [ADR-0026](adr/0026-sandbox-runtime-selection.md) (Accepted) covers NVIDIA OpenShell (phase 1) and Agent Substrate (phase 2); both still need their prototype measurements and cluster preflights. ## Agent guardrails (defense in depth)