[Proposal] add sandbox-pool management proposal#431
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
|
Welcome @lichuqiang! It looks like this is your first PR to volcano-sh/agentcube 🎉 |
|
/cc @RainbowMango |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a design proposal for SandboxPool Resource Pool Management, which outlines a two-layer architecture to declaratively manage and elastically scale node-level sandbox resource pools. Feedback on the proposal includes fixing a markdown rendering issue with un-opened code blocks, using resource.Quantity instead of string for CPU and memory fields to align with Kubernetes API standards, and refining the state machine logic to ensure all node conditions are re-evaluated when a node is recovered rather than unconditionally transitioning to a Ready state.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
There was a problem hiding this comment.
Pull request overview
Adds a new design proposal documenting the “slow track” SandboxPool resource pool management architecture, intended to address AgentCube’s current Kubernetes control-plane bottlenecks (startup latency and scale) discussed in #430.
Changes:
- Introduces a new proposal document describing a two-layer SandboxPool architecture (controller + node-local placeholder-agent).
- Defines proposed CRD data models (SandboxPoolClass/SandboxPool), status/conditions ownership (SSA FieldOwner), and a phase state machine.
- Documents operational flows (create/update/delete), RBAC, webhook validation rules, and a phased implementation plan.
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #431 +/- ##
==========================================
+ Coverage 58.41% 58.50% +0.08%
==========================================
Files 36 36
Lines 3463 3463
==========================================
+ Hits 2023 2026 +3
+ Misses 1231 1228 -3
Partials 209 209
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Could this PR use My understanding is that #430 is a broader architecture discussion, while this proposal focuses on the slow resource track / SandboxPool management. Using |
|
@lichuqiang please commit signoff statement too , dco also failling |
abso , this is discussion issue not a bug or need to fix , u should first purpose and discuss then open pr @lichuqiang |
|
I want to better understand the relationship with the existing agent-sandbox WarmPool path. Today |
| | Risk | Impact | Mitigation | | ||
| |------|--------|-----------| | ||
| | Phase update delay (≤30s) | Delay between Conditions change and Phase update | Alert rule sets `for ≥ 1m` buffer to avoid flapping | | ||
| | Conditions stuck at stale values when placeholder-agent unreachable | Phase may not reflect latest state | NodeNotFound covers node deletion; long unreachability causes Node to become NotReady, indirectly covering | |
There was a problem hiding this comment.
I have one question about this stale-status mitigation.
My reading is that placeholder-agent owns the node-local conditions and patches status every 30s, while the controller owns phase and NodeNotFound. If the Kubernetes Node still exists but placeholder-agent has stopped or can no longer patch status, the last successful PlaceholderPodReady=True / NodeCtlHealthy=True values may stay in status.
Could the proposal make the stale-status rule explicit for that case? For example, should the controller derive Degraded / Unready from a lastHeartbeat timeout, Node readiness, condition timestamps, or a controller-owned PlaceholderAgentHealthy condition?
This would make the phase aggregation and fault-injection tests clearer.
There was a problem hiding this comment.
Good suggestion. Updated related content
@acsoto The topic discussed in issue 430 and the current agentcube implementation represent more of a two-generation architecture relationship. |
| | ValidatingWebhook | 1.16 GA | None | | ||
| | CRD v1 | 1.16 GA | None | | ||
| | SSA FieldOwner | 1.18 GA | None | | ||
| | VPA InPlaceResize | 1.27 Alpha / 1.31 GA | FeatureGate: InPlacePodVerticalScaling | |
There was a problem hiding this comment.
I think the compatibility assumption here needs clarification because the proposal combines two behaviors that the native Kubernetes resize path does not currently support together.
The proposal makes both Static Pod resource locking and no-rebuild VPA InPlaceResize part of the v1alpha1 core scope. However, Kubernetes KEP-1287 explicitly lists "The pod is a static pod" as an Infeasible in-place resize case. Its metadata records v1.27 alpha, v1.33 beta, and v1.35 stable, rather than v1.31 GA.
Could the proposal clarify which mechanism Phase 3 intends to use?
- Native Kubernetes
/resize: Static Pods are unsupported. - Local Static Pod manifest update: if kubelet rebuilds the Pod, this is not a no-rebuild resize, and the resource-locking behavior during replacement needs to be defined.
- A custom
placeholder-agent/ runtime path: could the proposal describe how Kubernetes-visible requests, mirror Pod state, and node-ctl's actual limits remain consistent, instead of treating it as the nativeInPlacePodVerticalScalingpath?
This distinction changes the compatibility table, implementation contract, and e2e acceptance criteria.
There was a problem hiding this comment.
You're right — "VPA" here is more of an analogy, and should not depend on the community's pod VPA feature. Static pod manifest resource spec updates actually go through a delete-and-recreate process, but during the window kubelet can correctly account for the static pod's resource usage, ensuring that the SandboxPool's resources are not consumed by mis-scheduled pods during that period (this is what we care about most). I've added the relevant explanation in the "Update Flow" section.
| memory: 16Mi | ||
| ``` | ||
|
|
||
| kubelet routes CRI calls to placeholder-agent's Socket (`/run/sandbox-pool/cri.sock`) based on the RuntimeClass handler name. The overhead values remain unchanged during VPA resize. |
There was a problem hiding this comment.
I have one question about the node-side RuntimeClass / CRI integration contract.
The updated proposal makes custom CRI interception part of the resource-adjustment path. It also says that kubelet routes CRI calls to /run/sandbox-pool/cri.sock based on the placeholder RuntimeClass handler, with placeholder-agent implementing the CRI server.
In kubelet v1.36.2, RuntimeClass is resolved to a runtimeHandler and passed to the already configured runtimeService.RunPodSandbox(...); it does not select a second CRI endpoint. With containerd, the handler selects a configured runtime/shim (or sandboxer integration) behind the same CRI service.
Could the proposal specify the missing integration layer between the placeholder handler and placeholder-agent?
- Is
placeholder-agentintended to integrate as a containerd runtime v2 shim / sandboxer? - Is there a CRI proxy in front of containerd that dispatches by
runtime_handler? - Or is the node-wide kubelet CRI endpoint expected to point directly to
placeholder-agent, with normal workloads forwarded elsewhere?
This determines which API placeholder-agent must implement, how normal Pods continue to use the default runtime, and whether the proposed per-Pod routing can work without replacing the node's CRI path.
There was a problem hiding this comment.
Good question. The expectation here is to serve as the runtime implementation behind containerd, registered as a plugin into containerd. Therefore placeholder-agent implements the v2 shim interface.
I have already corrected the relevant content in the documentation.
Signed-off-by: lichuqiang <lichuqiang@huawei.com>
|
|
||
| - Provide declarative CRD APIs (SandboxPoolClass + SandboxPool) to manage node-level sandbox resource pools. | ||
| - Reliably lock scheduling resources through the Static Pod model; mirror pods are automatically rebuilt after accidental deletion (typically within seconds, depending on kubelet sync interval). | ||
| - Support adjusting placeholder Pod resource quotas via containerd runtime shim. This is NOT Kubernetes in-place Pod resize (InPlacePodVerticalScaling) — the placeholder Pod is rebuilt through Static Pod manifest update (delete/recreate), but running sandboxes managed by node-ctl are not interrupted during the rebuild. |
What type of PR is this?
/kind feature
What this PR does / why we need it:
As discussed in #430, the current agentcube architecture has limitations in certain scenarios. To break through the
bottlenecks of the Kubernetes architectural paradigm in workload startup speed and managed resource scale, we envision an architectural evolution of agentcube that establishes a dual fast/slow resource track. The slow track builds sandbox resource pool
management capabilities on top of the Kubernetes CRD framework, supporting the partitioning of resources on K8s nodes as resource pools for sandbox execution. In parallel, a lightweight, fast, Kubernetes-independent sandbox lifecycle management platform is
established to achieve more extreme performance and a larger management scale.
This proposal focuses on the slow track. The system implements declarative management of sandbox resources through a two-layer architecture:
synchronization, and Phase aggregation.
checking, and status reporting.
Core capabilities include:
deletion
Integrated with the fast-track data plane, this ultimately enables co-location of regular Pods and sandboxes on the same node.
Which issue(s) this PR fixes:
Refs #430
Does this PR introduce a user-facing change?:
Architecture evolution