Repository navigation
KIP-30: Sandbox MCP productionization (M0 done, M1-M3 proposed) - #633
Merged
Merged
Conversation
Contributor
Test Results973 tests +35 946 ✅ +35 7m 1s ⏱️ +3s Results for commit 51f4a58. ± Comparison against base commit bdea883. This pull request removes 1 and adds 36 tests. Note that renamed tests count towards both.♻️ This comment has been updated with latest results. |
xiaods
force-pushed
the
kip-30-sandbox-mcp-productionization
branch
4 times, most recently
from
October 15, 2026 14:34
4faba16 to
a822c80
Compare
…P-30 M2) mcp-serve bootstraps its own mTLS client identity via the same Login RPC the CLI uses and caches it in ConfigMap k8e-mcp/mcp-client-cert, so restarts and rollouts reuse one identity per Deployment and re-enroll (with the same private key) only inside the 30-day renewal window of the 90-day TTL. Trust stays fail-closed: without an explicit CA, a cached identity or an explicit --insecure-bootstrap opt-in, the API key is never sent. - pkg/sandbox/client: export Login/CreateCSR/ValidateIssuedCertificate so in-cluster enrollment reuses the CLI path instead of a copy. - pkg/sandboxmcp: EnrollmentConfig/ClientIdentity, ConfigMap identity cache (first-writer-wins across replicas), ensureGatewayIdentity; new --bootstrap-api-key/--insecure-bootstrap/--cert-cache flags (env variants too); gateway-cert/key kept as the mutually exclusive explicit pair path; Kubernetes now connects before the gateway dial so the identity cache client exists at enrollment time. - K8E_MCP_TENANT scopes every session the adapter creates (empty keeps the server-side default normalization). - manifests + init script: mcp-bootstrap Secret (sandbox CA + one API key) replaces the operator-generated client pair; MCP_BOOTSTRAP_API_KEY selects the enrollment key, RBAC names the identity ConfigMap explicitly. Signed-off-by: Xiao Deshi <xiaods@gmail.com>
xiaods
force-pushed
the
kip-30-sandbox-mcp-productionization
branch
from
October 16, 2026 01:12
a822c80 to
4eb93ac
Compare
The KIP-30 M2 init script stages only the sandbox CA and one API key (k8e-mcp/mcp-bootstrap); --gateway-cert/--gateway-key are gone because mcp-serve enrolls its own pair. Mock kubectl now answers the sandbox-apikeys keys.json lookup so the first-active-key selection runs, and the assertions cover mcp-bootstrap instead of the removed Secret. Signed-off-by: Xiao Deshi <xiaods@gmail.com>
KIP-8 的后继：让 MCP 适配器在默认安装和 3 节点集群下都能真正跑起来。
- M0（已实现并在单机集群验证，817e98dd）：server 默认 skip
sandbox-mcp 清单（目录级 + 文件级，fail-closed 移除旧 staged
文件），默认安装不再产 FailedMount/ImagePullBackOff 坏 pod；init
脚本保持唯一 apply 入口，README 记录升级时一次性 cleanup。
- M1：k8e-mcp 镜像进 release 流水线（GHCR +
%{SYSTEM_DEFAULT_REGISTRY}% 模板），HTTPRoute 增加 sectionName:
http 父引用，control-plane taint toleration。
- M2：mcp-serve 用 API key 自举客户端身份（Login → CSR → 签发），
私钥不出 pod，证书缓存进 ConfigMap；tenant 是隔离单元（API key
只是凭证，store.go Record 无 tenant 字段），mcp-serve 通过
K8E_MCP_TENANT 透传租户，server 把空 TenantID 归一化为 default。
- M3：sandbox-grpc-gateway EndpointSlice 移入 leader-gated
sandboxmatrix controller（每 server 一个 endpoint）；
TLS mirror loop 把 sandbox-server 证书对镜像进 sandbox-e2b；
warm pool 节点/RuntimeClass 感知；CLI 打印可粘贴 MCP 配置块 +
tls export-ca。
- 记录已验证的 IP-only 自签 HTTPS 全链路（advertise 公网 IP 进 SAN →
sandbox CA 签发 → mirror 进 sandbox-e2b → 客户端信任 sandbox CA）。
同步更新 docs/README.md 索引（Sandbox matrix 区）与 numbering note。
Signed-off-by: Xiao Deshi <xiaods@gmail.com>
xiaods
force-pushed
the
kip-30-sandbox-mcp-productionization
branch
from
October 16, 2026 01:50
4eb93ac to
6a7f3c0
Compare
Owner
Author
Implementation statusMilestones per KIP-30, verified on this branch's latest CI run:
Review focus for M2
CI: all jobs green (Build & Test, MCP interop, cross-compile, E2E L1/L2, SonarCloud). |
The k8e Cilium Gateway https listener needs a Secret named
sandbox-e2b that nothing creates on a default install, so the
listener stays Invalid until an operator stages a pair by hand (M0).
The server already owns a cert whose leaf carries the advertise
IP/hostname SANs and is signed by the sandbox CA every client
trusts; this loop copies that pair into the Secret so the Gateway is
valid out of the box on any install with a routable public IP/hostname.
- Ownership is opt-in by absence: the loop takes over only when the
Secret is absent or already carries its managed-by label. A
hand-created sandbox-e2b (cert-manager, purchased cert, operator
file) is never touched — the same conflict semantics the MCP init
script already practice.
- Leader-gated (same sandboxmatrix-controller Lease): exactly one
node writes the Secret, no last-writer-wins drift between servers.
- Triggers: loop start, fsnotify on the cert pair (the gRPC server
rotates sandbox-server.{crt,key} on SAN change or inside the
30-day renewal window), and a 10-minute safety-net sweep.
- A half-written rotation file (unparsable PEM) never reaches the
API — the previous Secret stays in place.
- --skip-sandbox-tls-mirror / K8E_SKIP_SANDBOX_TLS_MIRROR is the
documented escape hatch for externally managed Secrets.
Signed-off-by: Xiao Deshi <xiaods@gmail.com>
The staged e2b-gateway.yaml carried a static single-address EndpointSlice resolved from whichever node staged it first — correct for single-server installs, wrong for every HA one: a second server never appears, a dead first server's address lingers, and an IP change silently goes stale until restart. - Move the sandbox-grpc-gateway slice into the leader-gated sandboxmatrix controller: the leader lists server nodes (control-plane/etcd labeled, the same set the HA dynamiclistener CN filter uses), collects their routable Internal/ExternalIPs, and writes one endpoint each. Node join/leave and IP changes land within a reconcile tick; exactly one node writes, so no drift. - The staged manifest drops the gateway slice (e2b-server keeps its static one — it is a single host-process service). The controller adopts the object with app.kubernetes.io/managed-by: k8e-sandboxmatrix; updates retry on conflict. - No server addresses yet (first boot) keeps the current slice rather than writing an empty one. - --skip-sandbox-endpointslice-controller / K8E_SKIP_ENDPOINTSLICE_CONTROLLER is the documented escape hatch for externally managed slices (MetalLB-style VIP fronting). Signed-off-by: Xiao Deshi <xiaods@gmail.com>
A gvisor|kata|firecracker warm pool could only run on nodes that support that runtime, but nothing told the reconciler: pods landed on every node, the unsupported ones sat Pending (RuntimeClass not found) until the recycle pass deleted and recreated them against the same nodes — a churn loop while the pool reported NotReady for a reason nobody wrote down. - Node annotation sandbox.k8e.io/runtime-classes: gvisor,kata is mirrored to boolean node labels (sandbox.k8e.io/runtime-class-<class>) by the leader; warm pods carry a required nodeAffinity on their pool's class — but only when at least one node is annotated, so unannotated single-node installs keep scheduling exactly as before. - WarmPoolSpec gains nodeSelector (merged into the pod spec) for operator-level node pinning. - Pending warm pods the scheduler gave up on get the scheduler's own message stamped on the pod and surfaced as notReadyReasons on the SandboxMatrix status, instead of a silent delete/recreate churn. - Unschedulable pods are retired once and the next create targets a node that can actually run the class. Signed-off-by: Xiao Deshi <xiaods@gmail.com>
Close the UX gap of a header-based credential channel: an API key is
only half an MCP client's configuration.
- `apikey create` / `apikey list` print a copy-pasteable MCP config
block after the key JSON:
{"url": "https://sbx.example.com/mcp", "headers": {"Authorization":
"Bearer e2b_<key>"}}. The URL resolves from K8E_MCP_URL, then the
sandbox advertise hostname (KIP-22), then the active profile
endpoint host (KIP-17); when none is known the block is omitted
rather than guessed. `apikey list` prints it for a live key too, so
reconnecting a client later needs no doc lookup.
- `tls export-ca` writes the sandbox CA to ~/.k8e/sandbox-ca.crt (or
--output PATH / - for stdout), reading the on-server
/var/lib/k8e/server/tls/sandbox-ca.crt or the sandbox-matrix
sandbox-ca ConfigMap when running off-server. This reduces the
IP-only bootstrap to two CLI commands and gives KIP-28's
/.well-known distribution a non-well-known fallback.
Signed-off-by: Xiao Deshi <xiaods@gmail.com>
Owner
Author
M3 implemented (this branch)All four M3 milestones from KIP-30 are now on this branch, each a reviewable commit on top of the M2 implementation:
Review focus for M3
Note: the KIP-30 document's M3 section stays as the design record; the implementation notes above supersede any naming drift (e.g. the KIP says |
tls_mirror.go referenced tlsDirOverride, a var declared in a _test.go file, so the non-test build failed (go vet ./... in CI). tlsDir is a package var now; tests repoint it directly through withTLSDir, and the now-redundant tls_mirror_tlsdir_test.go is removed. Signed-off-by: Xiao Deshi <xiaods@gmail.com>
|
Owner
Author
|
All checks green on 51f4a58 (the build fix included). The M3 branch is complete: M0–M3 all implemented on
Ready for review; happy to split any milestone into its own PR if the reviewers prefer smaller diffs. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
KIP-30 提案，KIP-8 的后继：让 MCP 适配器在默认安装和 3 节点集群下都能真正跑起来。
M0（已实现并在单机集群验证，commit 817e98d）
sandbox-mcp清单（目录级 + 文件级，fail-closed 移除旧 staged 文件），默认安装不再产 FailedMount/ImagePullBackOff 坏 podM1（待开工）
%{SYSTEM_DEFAULT_REGISTRY}%模板），air-gap 仍可用导入镜像sectionName: http父引用（默认 HTTP-only 安装下路由不再被拒）M2（待开工）
mcp-serve用 API key 自举客户端身份（Login → CSR → 签发），私钥不出 pod，证书缓存进 ConfigMap（mcp-stateRole 扩 ConfigMap 读写）M3（待开工）
sandbox-grpc-gatewayEndpointSlice 移入 leader-gated sandboxmatrix controller，每 server 一个 endpoint（消除现状的 last-writer-wins 漂移）；K8E_SKIP_ENDPOINTSLICE_CONTROLLER保留外部管理出口sandbox-server.{crt,key}镜像进sandbox-e2b（label/field-manager 认领，手制 Secret 永不覆盖）apikey create打印可粘贴的 MCP 配置块（含 Bearer header）；tls export-ca已验证（KIP §Verification）：IP-only 自签 HTTPS 全链路 —— advertise 公网 IP 进 SAN、sandbox CA 签发、mirror 进 sandbox-e2b、外部客户端
--cacert sandbox-ca.crtssl_verify_result=0。风险与开放问题见 KIP §Risks（air-gap 镜像归属、EndpointSlice 成员来源、多 server 下 mirror 哪一对叶子证书、bootstrap 证书 TTL、warm-pool 注入 vs 探针）。
请审核，通过后再开工 M1。