ai-dynamo · ozbarshalom · Aug 20, 2026 · Aug 20, 2026 · Aug 20, 2026 · Aug 20, 2026
@@ -1,93 +1,188 @@
 # Snapshot
 
-> **This project is under construction.**
+Snapshot is a Kubernetes-native checkpoint and restore system for NVIDIA GPU
+workloads. It checkpoints a fully initialized GPU pod — its running process, with
+CPU and GPU memory — and restores that state on any compatible node, so a pod
+becomes ready in seconds instead of minutes.
 
+Snapshot provides the checkpoint and restore primitives for GPU pods.
+Orchestration — which pods to checkpoint, when, and how the checkpoints are
+restored — is left to the systems that integrate it.
 
-Snapshot is a Kubernetes-native checkpoint and restore system for NVIDIA GPU workloads.
-It enables AI frameworks and platforms to capture a fully initialized GPU worker and restore that state on any compatible node, allowing new pods to become ready in seconds instead of minutes.
+> [!NOTE]
+> Snapshot's APIs may still change, so it is not yet recommended for
+> production-critical workloads.
 
-Snapshot focuses on one responsibility: reliably capturing and restoring running GPU workloads. Higher-level decisions - such as which workloads to checkpoint, when to create snapshots, or when to restore them - are left to the systems integrating with Snapshot.
+## The Problem
 
+In inference serving, a replica can't answer a single request until it is fully
+initialized — model weights loaded into GPU memory, CUDA and runtime libraries
+initialized, execution kernels warmed up, and computation graphs compiled. For
+large models, this **cold start** takes minutes.
 
-## Why Snapshot?
+That cost is paid over and over. Every replica added to meet demand, every
+scale-up from zero, every restart or reschedule pays the full cold start again
+before it can serve traffic:
 
-GPU inference workers are expensive to start. Before serving a single request, a worker typically needs to load large model weights into GPU memory, initialize CUDA and other runtime libraries, warm up execution kernels, and compile or optimize computation graphs.
+- New replicas take minutes to become ready, so autoscaling lags behind demand.
+- Teams over-provision idle GPUs just to absorb demand spikes.
+- Restarts and reschedules stall serving capacity exactly when it is needed.
 
-For large models, this initialization can take several minutes. Every new replica, pod restart, reschedule, or scale-up event repeats the entire process, paying that cost from scratch.
-Snapshot eliminates most of this overhead by restoring a previously initialized worker instead of starting a new one.
+## The Solution
 
-## How it Works
+Snapshot checkpoints a fully initialized pod once and restores it on demand, so a
+new replica comes online in seconds instead of minutes.
 
-Snapshot exposes checkpoint and restore as Kubernetes resources.
+- **Checkpoint** — pause a running pod and save its complete execution state (CPU
+  and GPU memory) as a portable artifact.
+- **Restore** — start a new pod from that artifact on any node with matching GPU
+  hardware and driver versions, skipping model loading and warm-up; the process
+  resumes from where it was checkpointed.
 
-#### Capture
+## When to use it
 
-To create a snapshot, a caller identifies the pod to checkpoint. Snapshot pauses the running process and captures its complete execution state, including both CPU memory and GPU memory, into a persistent artifact.
-This artifact is not a container image, a filesystem snapshot, or a volume snapshot. Instead, it represents the complete in-memory state of a live, fully initialized GPU worker.
+- **Autoscaling inference** — scale out from an existing snapshot: bring the N+1
+  replica and beyond online in seconds to keep pace with demand.
+- **Scale-to-zero** — park idle models at zero replicas and restore them quickly
+  when capacity is needed again.
+- **Faster restarts and reschedules** — recover a pod's initialized state after a
+  restart or a move to another node.
 
-#### Restore
+Snapshot currently focuses on inference cold-start; further use cases are on the
+roadmap.
 
-To restore a worker, a new pod references a previously captured snapshot artifact. During pod startup, Snapshot restores the captured process state directly into the container, bypassing model loading, kernel warm-up, and other initialization steps. The restored process resumes execution from the exact point where it was captured.
-Snapshots are portable across compatible machines and can be restored on any node with matching GPU hardware and driver versions. They are not tied to the node where they were originally created.
+## Who it's for
 
-&nbsp;
+Snapshot is a building block for the teams that build and operate inference
+infrastructure:
+
+- **Developers** building Kubernetes controllers, operators, or serving platforms.
+- **MLOps and platform engineers** who assemble deployment pipelines declaratively
+  with GitOps or workflow tools.
+
+## Prerequisites
+
+Before installing Snapshot, make sure the following are in place:
+
+- A Kubernetes cluster with NVIDIA GPU nodes
+- containerd or CRI-O as the container runtime
+- [NVIDIA GPU Operator](https://github.com/NVIDIA/gpu-operator) 26.3 or newer, with CUDA driver 580 or newer and MIG disabled
+- A `ReadWriteMany` (RWX) storage class
+- The [Helm](https://helm.sh/docs/intro/install) CLI
+- A cluster that permits privileged pods for the node agent — see [Security](docs/operations/security.md)
+
+## Installation
+
+Snapshot installs as a single per-cluster Helm release — a control-plane operator
+plus a privileged node agent (DaemonSet) on GPU nodes. Install it in its own
+namespace, and run GPU workloads in separate namespaces.
+
+Snapshot can be installed:
+
+- **From a release** (recommended)
+- **From source** (build the images and install locally)
+
+### From a release
+
+Find the latest version on the [releases page](https://github.com/ai-dynamo/snapshot/releases),
+then install the published chart, replacing `<VERSION>`:
+
+```bash
+helm install snapshot oci://ghcr.io/ai-dynamo/snapshot/snapshot \
+  --version <VERSION> \
+  --namespace snapshot --create-namespace
+```
+
+By default the chart provisions its own RWX checkpoint volume, shared by every
+checkpoint. See [Storage](docs/operations/storage.md) for the volume model and options
+(including reusing an existing claim), and [Installation](docs/operations/install.md)
+for install and uninstall.
+
+### From source
+
+Follow the instructions in [Building from source](docs/development/build-from-source.md).
+
+## How to use it
+
+Snapshot is driven entirely through Kubernetes resources, with standard tooling.
+Create a `PodSnapshot` to checkpoint a running pod, and annotate a new pod with
+`nvidia.com/restore-from` to restore it. Higher-level systems wire these
+primitives into their own control loop.
 
-## APIs
 | Resource | Scope | Role |
 |----------|-------|------|
-| `PodSnapshot` | Namespaced | Created by callers to request a capture or reference an artifact for restore. |
-| `PodSnapshotContent` | Cluster-scoped | System-managed record of the physical artifact, bound to a `PodSnapshot`. Created by the Snapshot operator, never by the caller. |
-| `nvidia.com/restore-from` | Namespaced | Added as a pod annotation to trigger restore from a named `PodSnapshot` in the same namespace. |
-| `nvidia.com/restore-container-map` | Namespaced | Optional comma-separated `source=destination` mappings used to clone the single captured container into one or more restore containers. |
+| `PodSnapshot` | Namespaced | Created by callers to request a checkpoint, or to reference an artifact for restore. |
+| `PodSnapshotContent` | Cluster-scoped | System-managed record of the physical artifact, bound to a `PodSnapshot`. Created by the operator, never by the caller. |
+| `SnapshotJob` | Namespaced | Runs a pod from a template and checkpoints it into a `PodSnapshot` once ready — a self-contained checkpoint job. |
+| `nvidia.com/restore-from` | Namespaced | Pod annotation that triggers a restore from a named `PodSnapshot` in the same namespace. |
+
+Under the hood, a control-plane operator and a per-node agent perform the CRIU
+and `cuda-checkpoint` work; see [Architecture](docs/reference/architecture.md).
+The [API reference](docs/reference/api.md) covers the resources and the
+checkpoint/restore lifecycle.
+
+Once Snapshot is installed, follow the **[usage guides](docs/guides/README.md)**
+to checkpoint and restore a pod.
+
+## Limitations
 
-&nbsp;
+Current limitations:
 
+- Single-GPU workloads only.
+- x86_64 nodes only.
+- vGPU is not supported.
+- Runs only on NVIDIA GPUs supported by the required CUDA driver.
 
-## Architecture
+Multi-GPU and Arm support are on the roadmap.
 
-Snapshot consists of two main components.
+## Documentation
 
-#### Operator
+**Get started**
 
-The Kubernetes operator manages the control plane.
+- [Usage guides](docs/guides/README.md) — build a snapshot-ready image per inference framework, then checkpoint and restore.
 
-It is responsible for:
+**Reference**
 
-* Orchestrating checkpoint and restore operations.
-* Tracking snapshot lifecycle.
-* Exposing status through Kubernetes resources.
-* Managing cleanup.
+- [API](docs/reference/api.md) — `PodSnapshot`, `PodSnapshotContent`, `SnapshotJob`, and the `restore-from` annotation.
+- [Architecture](docs/reference/architecture.md) — operator and node-agent design, and the checkpoint/restore internals.
+- [CLI (`snapshotctl`)](docs/reference/cli.md) — lower-level checkpoint/restore from a pod manifest.
 
+**Operations**
 
-#### Node Agent
+- [Installation](docs/operations/install.md) — Helm install and uninstall.
+- [Storage](docs/operations/storage.md) — the shared checkpoint volume and how to configure it.
+- [Troubleshooting](docs/operations/troubleshooting.md) — common failures and where to look.
+- [Security](docs/operations/security.md) — the privileged agent, seccomp, and Pod Security.
 
-A privileged node agent runs on every GPU node.
+**Development**
 
-It performs the actual checkpoint and restore operations by invoking CRIU and cuda-checkpoint against live processes.
+- [Building from source](docs/development/build-from-source.md) — build the images and install locally.
+- [Benchmarks](docs/development/benchmarks.md) — how startup performance is measured.
 
-The node agent is intentionally an implementation detail. Clients never communicate with it directly.
+**More**
 
-&nbsp;
+- [Limitations & known issues](docs/limitations.md) — current limitations and what's on the roadmap.
 
-## Design Principles
+## Adopters
 
-Snapshot owns the mechanics of checkpoint and restore—not the policy.
+[NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo), the open-source
+inference-serving stack, integrates Snapshot for GPU cold-start. On Dynamo, Snapshot is available through it directly — see
+[Snapshotting GPU Workers](https://docs.nvidia.com/dynamo/latest/kubernetes/operations/cold-start-optimizations/dynamo-snapshot)
+in the Dynamo docs.
 
-Systems integrating with Snapshot decide:
+## Contributing
 
-* Which workloads should be checkpointed.
-* When snapshots should be created.
-* When they should be restored.
-* How failures should be handled.
+Contributions are welcome under the project's [Apache 2.0 license](LICENSE). See
+[CONTRIBUTING.md](CONTRIBUTING.md) — all commits must be signed off (DCO).
 
-Snapshot executes those requests and exposes the resulting state.
+## Security
 
-Everything Snapshot manages is represented as Kubernetes resources. Snapshot metadata, capture progress, restore status, and lifecycle information are all observable through the Kubernetes API using standard Kubernetes tooling.
+To report a security vulnerability, follow the process in [SECURITY.md](SECURITY.md).
 
-Clients interact exclusively through the Kubernetes API. No platform-specific APIs, direct node communication, or custom protocols are required.
+## Feedback
 
-&nbsp;
+Feedback and issues are welcome — please [open an issue](https://github.com/ai-dynamo/snapshot/issues).
 
-## Status
+## License
 
-The project is in early development. API types and control plane components are scaffolded but not yet feature-complete. Not ready for production use.
+Snapshot is licensed under the [Apache License 2.0](LICENSE).
@@ -0,0 +1,3 @@
+# Benchmarks
+
+_Documentation for this page is in progress._
@@ -0,0 +1,76 @@
+# Building from source
+
+This guide builds the Snapshot operator and node-agent images from a checkout of
+this repository and installs the chart against them. Most users should install
+[from a release](../../README.md#from-a-release) instead — build from source when developing Snapshot or testing unreleased changes.
+
+## Prerequisites
+
+In addition to the [runtime prerequisites](../../README.md#prerequisites), the build needs:
+
+- Go (matching the version pinned in the modules)
+- Docker with Buildx
+- A container registry the cluster can pull from, and push access to it
+- `kubectl` and `helm` configured against the cluster
+
+The node agent is **x86_64 (amd64) only** — `cuda-checkpoint` ships no other
+architecture — so its image builds for `linux/amd64`.
+
+## 1. Clone the repository
+
+```bash
+git clone https://github.com/ai-dynamo/snapshot.git
+cd snapshot
+```
+
+## 2. Build the images
+
+The root `Makefile` builds both images. Override `REGISTRY` and `VERSION` to tag
+them for the registry:
+
+```bash
+make docker-build-agent docker-build-operator \
+  REGISTRY=<registry> \
+  VERSION=<tag>
+```
+
+This produces `<registry>/agent:<tag>` and
+`<registry>/operator:<tag>`.
+
+## 3. Push the images
+
+Push both images to a registry the cluster can pull from:
+
+```bash
+docker push <registry>/agent:<tag>
+docker push <registry>/operator:<tag>
+```
+
+## 4. Install the chart against the built images
+
+Install the chart from the checkout, pointing the operator and agent images at
+the built images:
+
+```bash
+helm install snapshot ./charts/snapshot \
+  --namespace snapshot --create-namespace \
+  --set image.operator.repository=<registry>/operator \
+  --set image.operator.tag=<tag> \
+  --set image.agent.repository=<registry>/agent \
+  --set image.agent.tag=<tag>
+```
+
+See [Installation](../operations/install.md) for storage and uninstall options.
+
+## Development workflow
+
+Common `make` targets from the repo root:
+
+- `make build` — compile the agent and operator
+- `make test` — run unit tests across the `api`, `agent`, and `operator` modules
+- `make lint` — run linters
+- `make helm-lint` — lint the Helm chart
+- `make check` — the full pre-merge gate (generate, license headers, fmt, tidy, lint, and more)
+
+See [CONTRIBUTING.md](../../CONTRIBUTING.md) for the contribution process and DCO
+sign-off.
@@ -0,0 +1,38 @@
+# Usage guides
+
+Using Snapshot is a three-stage flow:
+
+1. **Build and deploy** a snapshot-ready image for the inference framework. Start
+   from the framework's runtime image, add a small program that cooperates with
+   Snapshot's checkpoint/restore lifecycle, and deploy it as a replica. Snapshot's agent
+   injects the restore tooling at runtime.
+2. **Checkpoint** the running replica — with a `PodSnapshot` or a `SnapshotJob`.
+3. **Restore** into new pods — with the `nvidia.com/restore-from` annotation.
+
+Stages 2 and 3 are the same for every framework; only the image and deployment in
+stage 1 differ.
+
+> [!NOTE]
+> These guides use `kubectl` to show the resources and the flow. In production, a
+> controller or platform creates and watches these resources through the Kubernetes
+> API as part of its own control loop — `kubectl` here is just for illustration and
+> for trying things out by hand.
+
+## 1. Build and deploy
+
+Per inference framework:
+
+- [vLLM](vllm.md)
+- [SGLang](sglang.md)
+- [TensorRT-LLM](tensorrt-llm.md)
+
+## 2. Checkpoint
+
+- [Checkpoint a replica](checkpoint.md)
+
+## 3. Restore
+
+- [Restore a replica](restore.md)
+
+See [Installation](../operations/install.md) for cluster prerequisites and the
+[API reference](../reference/api.md) for full resource detail.
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1,3 @@
		# Benchmarks

		_Documentation for this page is in progress._