clubanderson · Nov 11, 2026 · Nov 11, 2026 · Nov 11, 2026 · Nov 11, 2026 · Nov 11, 2026
diff --git a/changelog.d/changed-substrate-runtime-docs.md b/changelog.d/changed-substrate-runtime-docs.md
@@ -0,0 +1 @@
+- Document Agent Substrate as the opt-in fourth `sandbox.runtime` (ADR-0026 phase 2): ADR amendment with the contract mapping, deviations and when to choose it over OpenShell, a `sandbox-isolation.md` section, `deploy/substrate.md` install and verification notes with the hive cluster survey, and the integrations table entry.
@@ -20,6 +20,7 @@ Use it indirectly through the dashboard terminal link. If a terminal pane says i
 - `k8s/` — the Kustomize base (`kustomization.yaml`) with namespace, deployment, service, PVC, Secret, ConfigMap and RBAC manifests, plus the custom error backend (`error-pages.yaml`), the hub backup CronJob (`backup-cronjob.yaml`) and the node-prep DaemonSet (`node-prep/`), which are applied separately. The OpenShift Route lives in `kustomize/overlays/openshift/`.
 - `inference/` — sample in-cluster OpenAI-compatible inference deployment and RBAC.
 - `openshell.md` — cluster-side notes for the opt-in `sandbox.runtime: openshell` (ADR-0026): prerequisites, the upstream Helm install, the gateway settings and PVC labels hive's runtime depends on, and how to verify. No manifests are vendored.
+- `substrate.md` — cluster-side notes for the opt-in `sandbox.runtime: substrate` (ADR-0026 phase 2): the Agent Substrate operator install, WorkerPool and SandboxConfig, the pod-certificate feature gates, the router client bundle hive needs, and how to verify. No manifests are vendored.
 - `docker-compose.architect.yaml`, `hive-quickstart.yaml`, `hive-level*.yaml`, `architect-only.yaml`, `hive.yaml` — example deployment/configuration manifests.
 - `blue-green-deploy.sh`, `bootstrap-lxc.sh`, `create-lxc.sh` — operational scripts for **Docker Compose** deployments (LXC bootstrap and a blue-green Compose upgrade). Not "non-Kubernetes" generally: none of the three shells out to `podman` or reads `HIVE_DEPLOY_RUNTIME`, so none of them applies to the Quadlet stack in `quadlet/` below. See [Deployment helper scripts](../docs/deployment-scripts.md) for each script's scope and where a Podman operator should go instead.
 - `quadlet/` — Podman Quadlet units for the standalone deployment (ADR-0017). `hive.container` and `hive-data.volume` start the Hive service under `systemd`, rootful or rootless; `hive.network` and `hive-gateway.container` put the authenticating gateway in front of it with the same published-port split as the Compose stack — 3001 published, 7681 never. See [the operator guide](../docs/podman-standalone-quadlet.md). Docker Compose remains the default runtime and is unaffected.

@@ -0,0 +1,118 @@
+# Installing Agent Substrate for `sandbox.runtime: substrate`
+
+Cluster-side notes for the opt-in Substrate sandbox runtime
+([ADR-0026 phase 2](../docs/adr/0026-sandbox-runtime-selection.md#implementation-notes-phase-2-agent-substrate),
+epic [#17695](https://github.com/hivecommons/hive/issues/17695); operator view
+in [sandbox-isolation.md](../docs/sandbox-isolation.md#running-sandboxed-kicks-in-a-substrate-actor)).
+Hive installs nothing: Substrate is a cluster-admin install, once per cluster,
+and hive only talks to its Control API and router. Upstream is the source of
+truth for the install — [agent-substrate/substrate → docs/guides/operator-install.md](https://github.com/agent-substrate/substrate/blob/main/docs/guides/operator-install.md),
+the [API guide](https://github.com/agent-substrate/substrate/blob/main/docs/reference/api-guide.md)
+and [sandbox classes](https://github.com/agent-substrate/substrate/blob/main/docs/reference/sandbox-classes.md) —
+and no manifests are vendored here; pin the release you install and record it
+alongside the hive config.
+
+## Prerequisites to check before installing
+
+| Requirement | Why | How to check |
+|---|---|---|
+| Kubernetes with `ClusterTrustBundle`, `ClusterTrustBundleProjection` and `PodCertificateRequest` feature gates and `certificates.k8s.io/v1beta1` enabled | actor and client identities are pod certificates (SPIFFE); the podcert controller does not start without them. Still off by default as of Kubernetes 1.36 | `kubectl get --raw /metrics \| grep 'kubernetes_feature_enabled{name="PodCertificateRequest"'` and `kubectl api-resources --api-group=certificates.k8s.io`. Managed control planes (OKE, LKE) need the provider to enable them; OpenShift needs a `FeatureGate` CR |
+| Standard Linux nodes with **cgroup v2** for the `gvisor` class | `ateworker-gvisor` runs `runsc` inside the worker pod; there is no kernel floor beyond ≥ 5.8 | `kubectl get nodes -o jsonpath='{.items[*].status.nodeInfo.kernelVersion}'`; `stat -fc %T /sys/fs/cgroup` on a node → `cgroup2fs` |
+| `/dev/kvm` on nodes for the `microvm` class | Kata on Cloud Hypervisor guests; `atelet` advertises the `ate.dev/kvm` device only where it exists | `kubectl get nodes -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.status.allocatable.ate\.dev/kvm}{"\n"}{end}'` after install |
+| Nodes trusted to run worker pods with `SYS_ADMIN` and `NET_ADMIN` | workers are `privileged: false` but keep those two for mounts and networking | on OpenShift, an SCC granting them to the `ate-system` worker ServiceAccount |
+| A CNI that **enforces** egress `NetworkPolicy` | Substrate's `atenet` dataplane carries actor egress; the worker pod itself must not be a side door | create a deny-all policy in a scratch namespace and confirm a connection is refused |
+| PostgreSQL and S3-compatible object storage | Control API state; actor snapshots (`snapshot_on_idle`) | `ate-setup` deploys bundled ones for a lab; use managed services in production |
+| Cluster-admin for the install | CRDs (`workerpools`, `sandboxconfigs`, `egresspolicies`, …), ValidatingAdmissionPolicies, RBAC, PKI | — |
+
+### Which hive clusters qualify (survey 2026-11-11)
+
+| cluster | kernel / Kubernetes | `gvisor` | `microvm` |
+|---|---|---|---|
+| hive-oke | OL9, UEK8 6.12, k8s 1.34.10 | yes, once the podcert feature gates are enabled on the OKE control plane | needs KVM-capable shapes (bare metal or nested virtualisation) |
+| lke648397, hive-ci | Debian 13, 6.12 | yes; confirm the feature gates — LKE exposes no control-plane flags | KVM to confirm per plan |
+| vllm-d, a-ks-wec2 | RHCOS 9.x, 5.14 | possible: SCC for the worker capabilities + `FeatureGate` CR (TechPreviewNoUpgrade) | only on KVM-capable nodes |
+
+## Install
+
+Build or download `ate-setup` from a pinned release of
+`github.com/agent-substrate/substrate`, write the install document, and deploy:
+
+```yaml
+# substrate-install.yaml
+apiVersion: install.ate.dev/v1alpha1
+kind: SubstrateInstall
+namespace: ate-system
+atenet:
+  dataplane: envoy
+images:
+  repo: <your mirror>/substrate
+  tag: <pinned release>
+```
+
+```shell
+ate-setup deploy ate-system --config substrate-install.yaml \
+  --credential-provider='{"name":"k8s.io"}' \
+  --ingress-allowed-spiffe-ids=spiffe://cluster.local/ns/<hive namespace>/sa/<hive ServiceAccount>
+```
+
+`--ingress-allowed-spiffe-ids` must name hive's own pod identity: that is what
+lets hive open `CONNECT` tunnels through `atenet-router` to the bridge in each
+actor. Unknown keys in the document are rejected, not ignored.
+
+Then create the pieces hive's runtime depends on, in Substrate terms:
+
+- **An atespace** for hive (`sandbox.substrate.atespace`, default `default`),
+  and a principal for hive's token (`token_env` / `token_file`) that may
+  create and delete `ActorTemplate`, `Actor` and `EgressPolicy` and list
+  workers in it. `kubectl ate` shows the exact RBAC verbs for the release.
+- **A `WorkerPool`** with `spec.sandboxClasses[].name: gvisor` (or `microvm`)
+  resolving the default `SandboxConfig` (`gvisor-default`), labelled so that
+  `sandbox.substrate.worker_selector` matches it; hive only uses pools whose
+  workers report `ACTIVE` and advertise a sandbox class. Taint the node pool
+  `ate.dev/sandboxClass=<class>:NoSchedule` to reserve it; worker pods
+  tolerate it.
+- **Projected files in the hive pod**: a ServiceAccount token with audience
+  `api.ate-system.svc` (→ `token_file`), the Control API CA via
+  `clusterTrustBundle` with signer `servicedns.podcert.ate.dev/identity`
+  (→ `ca_file`), and a `podCertificate` with signer
+  `podidentity.podcert.ate.dev/identity` (→ `router_client_bundle`) plus the
+  router CA (→ `router_ca_file`). The authentication guide upstream has the
+  volume stanza.
+- **A digest-pinned sandbox image** containing `crux` (for the
+  `substrate-bridge` entrypoint) and the agent CLI, set as
+  `sandbox.substrate.image`; tags are refused at boot and at kick. Publishing
+  it is tracked in [#17697](https://github.com/hivecommons/hive/issues/17697).
+- **Egress**: hive writes its own `EgressPolicy` per actor (`github.com`,
+  `api.github.com`, `allow_hosts`). Do **not** also configure Substrate's
+  credential injection for GitHub — hive's MITM proxy brokers credentials and
+  the two would conflict.
+
+## Verify
+
+```shell
+kubectl -n ate-system get deploy,ds                      # ateapi, atenet-router, podcert controller, atelet ready
+kubectl get crd | grep ate.dev                            # workerpools, sandboxconfigs, egresspolicies, …
+kubectl get workerpool -A                                 # at least one pool with ACTIVE workers
+kubectl get sandboxconfig gvisor-default -o yaml | grep defaultVersion
+kubectl get nodes -o custom-columns=NAME:.metadata.name,KVM:.status.allocatable.ate\.dev/kvm
+kubectl ate create actor smoke -a <atespace> --template <any template> && kubectl ate delete actor smoke -a <atespace>
+```
+
+From the hive side, start with `runtime: substrate` and watch the boot log: the
+preflight warns when the Control API is unreachable, has no matching `ACTIVE`
+worker, or its workers report no sandbox class. Then run one kick and confirm
+in the router log that the actor's SPIFFE id made exactly the allowed
+connections and that a connect to any other host was refused — that denial is
+the point of the runtime. Latency and image numbers go back into this page
+from the prototype ([#17697](https://github.com/hivecommons/hive/issues/17697));
+the fuller prover (CRDs, operator, feature gates, KVM) is
+[#17698](https://github.com/hivecommons/hive/issues/17698).
+
+## Upgrades
+
+Substrate is pre-1.0; follow its version-skew policy (N ↔ N+1 only) and pin
+the release and the sandbox image by digest. A `SandboxConfig` upgrade adds a
+`runsc` version and keeps the previous one so suspended actors still restore.
+Hive's generated client (`crux/sandbox/substrateapi`) tracks the fields it
+uses; bump it with `generate.sh` against the matching upstream tag when the
+Control API changes.
diff --git a/src/docs/adr/0026-sandbox-runtime-selection.md b/src/docs/adr/0026-sandbox-runtime-selection.md
@@ -1,6 +1,6 @@
 # ADR-0026: Sandbox runtime selection
 
-Status: Accepted (phase 1 implemented, runtime opt-in) — #17642, hivecommons/hive#17646
+Status: Accepted (phase 1 OpenShell implemented, runtime opt-in — #17642, hivecommons/hive#17646; phase 2 Agent Substrate implemented, runtime opt-in — #17695, hivecommons/hive#17741)
 
 ## Context
 
@@ -218,6 +218,74 @@ runs on, read on 2026-11-10: LKE (`lke648397`, `hive-ci`) run 6.12 on Debian 13
 with Calico and qualify; OKE UEK7 (5.15), and the RHCOS 9.x OpenShift clusters
 (5.14) do not.
 
+## Implementation notes (phase 2: Agent Substrate)
+
+Phase 2 (#17695, hivecommons/hive#17741) adds a **fourth** runtime,
+`sandbox.runtime: substrate`, behind the same `sandbox.Launcher` seam. It is
+[Agent Substrate](https://github.com/agent-substrate/substrate) (CNCF Sandbox,
+donated by Google, Apache-2.0): a Kubernetes-hosted *actor* runtime where an
+`ActorTemplate` (image, command, env, limits, volumes) is instantiated as an
+`Actor` through a gRPC Control API (`ate.control.v1`), scheduled on a
+`WorkerPool` whose pods run the workload under gVisor (`runsc`, class
+`gvisor`) or in a KVM microVM (Kata on Cloud Hypervisor, class `microvm`). All
+egress leaves through `atenet-router` under an `EgressPolicy`, and every actor
+holds a SPIFFE identity issued through Kubernetes pod certificates. It was not
+in the 2026-10-25 comparison; the write-up that led here is the Substrate
+vs. crux research note, and the operator view is in
+[sandbox-isolation.md](../sandbox-isolation.md#running-sandboxed-kicks-in-a-substrate-actor)
+and [deploy/substrate.md](../../deploy/substrate.md). Like OpenShell it is
+opt-in and inert unless selected.
+
+### How it maps onto the contract
+
+| Contract item | Substrate | What phase 2 does |
+|---|---|---|
+| Kernel-enforced confinement | gVisor user-space kernel or a microVM; worker pods are `privileged: false` but keep `SYS_ADMIN`/`NET_ADMIN` | selects the class through `worker_selector`; no change to hive |
+| Default-deny egress | `atenet-egress` `EgressPolicy` per actor; rules are TLS-passthrough or terminate per host:port, no per-method read-only mode | `SubstrateEgressPolicy`: `github.com` and `api.github.com` on 443 plus `sandbox.substrate.allow_hosts`, TLS passthrough; everything else (and all UDP) denied |
+| Credentials never in the sandbox | zero-credential model: actors have a SPIFFE id, credentials are injected at the router | unchanged hive brokering — the agent still gets `HTTPS_PROXY` to the hive MITM proxy (ADR-0002); Substrate's injection is **not** used, the two are complementary |
+| Workspace in, commits out | `durableDir` volumes and object-storage snapshots; no exec/attach API | an in-actor HTTP **bridge** (`crux substrate-bridge`) reached through the router's mTLS `CONNECT` tunnel: push tar → `POST /run` → pull tar; the hive-side workspace stays authoritative |
+| Cleanup and timeouts | `DeleteActor` / `SuspendActor` / `DeleteActorTemplate` | one template + actor per kick, both deleted on success, failure and timeout (fresh bounded context, `ExitCode -1`, 16 MiB cap, as `job`/`openshell`) |
+
+### Deviations from Substrate's reference model
+
+| Substrate assumes | Phase 2 does |
+|---|---|
+| long-lived actors and reusable templates | per-kick `ActorTemplate` + `Actor`, named `hive-kick-<agent>-<unix>` |
+| Go client `go.ate.dev/substrate` | **not imported**: it needs Go 1.27 (hive is on 1.26) and a very large module graph. A trimmed, wire-compatible proto subset is generated in-repo (`crux/sandbox/substrateapi`, field numbers pinned by tests); zero new Go modules, +578 KB on the `crux` binary |
+| image by tag or digest | digest-pinned `sandbox.substrate.image` only, enforced at boot (`AgentSandboxGateWarnings`) and at kick |
+| no `policy_profile` concept | passed to the actor as `HIVE_POLICY_PROFILE`; Substrate has no annotations |
+| snapshot/resume as the normal lifecycle | `snapshot_on_idle: true` suspends (snapshot to object storage) instead of deleting; resume-on-next-kick is not implemented (#17697) |
+| `api.github.com` read-only | TLS passthrough like `github.com`; the actor holds no token and hive does every write after the kick |
+
+### Cluster prerequisites and which hive clusters qualify
+
+Substrate's real requirement is **not** a kernel floor: the gVisor class runs
+`runsc` inside the worker pod on any standard Linux node with cgroup v2; the
+microVM class needs `/dev/kvm` (nested virtualisation on cloud shapes). Both
+need the operator (`ate-setup deploy ate-system`: Control API, `atelet`
+DaemonSet, `atenet-router`, podcert controller, PostgreSQL, snapshot object
+storage) and Kubernetes with the `ClusterTrustBundle`,
+`ClusterTrustBundleProjection` and `PodCertificateRequest` feature gates — still
+off by default as of 1.36 — because actor identities are pod certificates.
+Survey on 2026-11-11:
+
+| cluster | kernel / Kubernetes | `gvisor` class | `microvm` class |
+|---|---|---|---|
+| hive-oke | OL9, UEK8 6.12, k8s 1.34.10 | yes once the feature gates are on (OKE control plane: needs a support request or a self-managed control plane) | needs KVM-capable (bare-metal or nested-virt) shapes |
+| lke648397, hive-ci | Debian 13, 6.12 | yes; LKE exposes no control-plane flags, so the podcert gates are the blocker — check with `kubectl get --raw /metrics \| grep kubernetes_feature_enabled` | KVM to confirm per plan |
+| vllm-d, a-ks-wec2 | RHCOS 9.x, 5.14 | possible: the kernel is fine, but the worker's `SYS_ADMIN`/`NET_ADMIN` needs an SCC and the gates a `FeatureGate` CR (TechPreviewNoUpgrade) | only on KVM-capable nodes |
+
+### When to choose it over OpenShell
+
+- Choose **OpenShell** when the cluster already runs a ≥ 6.2 kernel with an
+  enforcing CNI and you want per-binary, per-method egress policy and the
+  shared-PVC workspace with no image changes.
+- Choose **Substrate** when you want a stronger boundary than Landlock +
+  seccomp (gVisor or a VM), snapshot/suspend of idle agents, or SPIFFE
+  identities for the sandboxed workload, and you can run its operator,
+  PostgreSQL and object storage. Its egress policy is coarser (host:port, no
+  read-only API mode) and the workspace is copied, not shared.
+
 ## Consequences
 
 - Hive gains a sandbox boundary that is checked before the agent runs, without
@@ -233,8 +301,12 @@ with Calico and qualify; OKE UEK7 (5.15), and the RHCOS 9.x OpenShift clusters
   hive's policy.
 - Telemetry must be disabled explicitly. A preflight check should refuse a
   gateway that reports it enabled.
+- Substrate (phase 2) adds a second operator-side stack (Control API,
+  atelet, atenet-router, PostgreSQL, object storage) and a copied workspace;
+  the hive-side credential model is untouched. Its preflight (#17698) and
+  prototype (#17697) follow the OpenShell ones.
 - If the prototype fails a gate, this record is rewritten to decide "no
   change" with the measured reason, and the Podman and Job runtimes remain the
   only ones.
 
-Refs #2804, #6311, #13806, #14092.
+Refs #2804, #6311, #13806, #14092, #17642, #17695.
diff --git a/src/docs/integrations.md b/src/docs/integrations.md
@@ -94,6 +94,7 @@ per-backend confinement matrix.
 <tr><td><img src="assets/logos/podman.svg" height="40" alt="Podman logo"></td><td>Rootless Podman agent sandbox</td><td>Opt-in</td><td><a href="sandbox-isolation.md#current-wiring">sandbox-isolation.md</a></td></tr>
 <tr><td><img src="assets/logos/kubernetes.svg" height="40" alt="Kubernetes logo"></td><td>Kubernetes Job sandbox runtime</td><td>Opt-in</td><td><a href="sandbox-isolation.md#running-sandboxed-kicks-as-kubernetes-jobs">sandbox-isolation.md</a></td></tr>
 <tr><td><img src="assets/logos/nvidia.svg" height="40" alt="NVIDIA logo"></td><td>NVIDIA OpenShell sandbox runtime</td><td>In progress (opt-in, phase 1)</td><td><a href="sandbox-isolation.md#running-sandboxed-kicks-in-an-openshell-sandbox">sandbox-isolation.md</a> · <a href="https://github.com/hivecommons/hive/issues/17642">#17642</a></td></tr>
+<tr><td><img src="assets/logos/cncf.svg" height="40" alt="CNCF logo"></td><td>Agent Substrate sandbox runtime (CNCF Sandbox)</td><td>In progress (opt-in, phase 2)</td><td><a href="sandbox-isolation.md#running-sandboxed-kicks-in-a-substrate-actor">sandbox-isolation.md</a> · <a href="https://github.com/hivecommons/hive/issues/17695">#17695</a></td></tr>
 <tr><td><img src="assets/logos/opensandbox-badge.svg" height="40" alt="OpenSandbox"></td><td>AAIF OpenSandbox (evaluation)</td><td>Planned</td><td><a href="https://github.com/hivecommons/hive/issues/14092">#14092</a></td></tr>
 </table>
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		- Document Agent Substrate as the opt-in fourth `sandbox.runtime` (ADR-0026 phase 2): ADR amendment with the contract mapping, deviations and when to choose it over OpenShell, a `sandbox-isolation.md` section, `deploy/substrate.md` install and verification notes with the hive cluster survey, and the integrations table entry.