diff --git a/README.md b/README.md index 5a794c9..582abe8 100644 --- a/README.md +++ b/README.md @@ -1,93 +1,188 @@ # Snapshot -> **This project is under construction.** +Snapshot is a Kubernetes-native checkpoint and restore system for NVIDIA GPU +workloads. It checkpoints a fully initialized GPU pod — its running process, with +CPU and GPU memory — and restores that state on any compatible node, so a pod +becomes ready in seconds instead of minutes. +Snapshot provides the checkpoint and restore primitives for GPU pods. +Orchestration — which pods to checkpoint, when, and how the checkpoints are +restored — is left to the systems that integrate it. -Snapshot is a Kubernetes-native checkpoint and restore system for NVIDIA GPU workloads. -It enables AI frameworks and platforms to capture a fully initialized GPU worker and restore that state on any compatible node, allowing new pods to become ready in seconds instead of minutes. +> [!NOTE] +> Snapshot's APIs may still change, so it is not yet recommended for +> production-critical workloads. -Snapshot focuses on one responsibility: reliably capturing and restoring running GPU workloads. Higher-level decisions - such as which workloads to checkpoint, when to create snapshots, or when to restore them - are left to the systems integrating with Snapshot. +## The Problem +In inference serving, a replica can't answer a single request until it is fully +initialized — model weights loaded into GPU memory, CUDA and runtime libraries +initialized, execution kernels warmed up, and computation graphs compiled. For +large models, this **cold start** takes minutes. -## Why Snapshot? +That cost is paid over and over. Every replica added to meet demand, every +scale-up from zero, every restart or reschedule pays the full cold start again +before it can serve traffic: -GPU inference workers are expensive to start. Before serving a single request, a worker typically needs to load large model weights into GPU memory, initialize CUDA and other runtime libraries, warm up execution kernels, and compile or optimize computation graphs. +- New replicas take minutes to become ready, so autoscaling lags behind demand. +- Teams over-provision idle GPUs just to absorb demand spikes. +- Restarts and reschedules stall serving capacity exactly when it is needed. -For large models, this initialization can take several minutes. Every new replica, pod restart, reschedule, or scale-up event repeats the entire process, paying that cost from scratch. -Snapshot eliminates most of this overhead by restoring a previously initialized worker instead of starting a new one. +## The Solution -## How it Works +Snapshot checkpoints a fully initialized pod once and restores it on demand, so a +new replica comes online in seconds instead of minutes. -Snapshot exposes checkpoint and restore as Kubernetes resources. +- **Checkpoint** — pause a running pod and save its complete execution state (CPU + and GPU memory) as a portable artifact. +- **Restore** — start a new pod from that artifact on any node with matching GPU + hardware and driver versions, skipping model loading and warm-up; the process + resumes from where it was checkpointed. -#### Capture +## When to use it -To create a snapshot, a caller identifies the pod to checkpoint. Snapshot pauses the running process and captures its complete execution state, including both CPU memory and GPU memory, into a persistent artifact. -This artifact is not a container image, a filesystem snapshot, or a volume snapshot. Instead, it represents the complete in-memory state of a live, fully initialized GPU worker. +- **Autoscaling inference** — scale out from an existing snapshot: bring the N+1 + replica and beyond online in seconds to keep pace with demand. +- **Scale-to-zero** — park idle models at zero replicas and restore them quickly + when capacity is needed again. +- **Faster restarts and reschedules** — recover a pod's initialized state after a + restart or a move to another node. -#### Restore +Snapshot currently focuses on inference cold-start; further use cases are on the +roadmap. -To restore a worker, a new pod references a previously captured snapshot artifact. During pod startup, Snapshot restores the captured process state directly into the container, bypassing model loading, kernel warm-up, and other initialization steps. The restored process resumes execution from the exact point where it was captured. -Snapshots are portable across compatible machines and can be restored on any node with matching GPU hardware and driver versions. They are not tied to the node where they were originally created. +## Who it's for -  +Snapshot is a building block for the teams that build and operate inference +infrastructure: + +- **Developers** building Kubernetes controllers, operators, or serving platforms. +- **MLOps and platform engineers** who assemble deployment pipelines declaratively + with GitOps or workflow tools. + +## Prerequisites + +Before installing Snapshot, make sure the following are in place: + +- A Kubernetes cluster with NVIDIA GPU nodes +- containerd or CRI-O as the container runtime +- [NVIDIA GPU Operator](https://github.com/NVIDIA/gpu-operator) 26.3 or newer, with CUDA driver 580 or newer and MIG disabled +- A `ReadWriteMany` (RWX) storage class +- The [Helm](https://helm.sh/docs/intro/install) CLI +- A cluster that permits privileged pods for the node agent — see [Security](docs/operations/security.md) + +## Installation + +Snapshot installs as a single per-cluster Helm release — a control-plane operator +plus a privileged node agent (DaemonSet) on GPU nodes. Install it in its own +namespace, and run GPU workloads in separate namespaces. + +Snapshot can be installed: + +- **From a release** (recommended) +- **From source** (build the images and install locally) + +### From a release + +Find the latest version on the [releases page](https://github.com/ai-dynamo/snapshot/releases), +then install the published chart, replacing ``: + +```bash +helm install snapshot oci://ghcr.io/ai-dynamo/snapshot/snapshot \ + --version \ + --namespace snapshot --create-namespace +``` + +By default the chart provisions its own RWX checkpoint volume, shared by every +checkpoint. See [Storage](docs/operations/storage.md) for the volume model and options +(including reusing an existing claim), and [Installation](docs/operations/install.md) +for install and uninstall. + +### From source + +Follow the instructions in [Building from source](docs/development/build-from-source.md). + +## How to use it + +Snapshot is driven entirely through Kubernetes resources, with standard tooling. +Create a `PodSnapshot` to checkpoint a running pod, and annotate a new pod with +`nvidia.com/restore-from` to restore it. Higher-level systems wire these +primitives into their own control loop. -## APIs | Resource | Scope | Role | |----------|-------|------| -| `PodSnapshot` | Namespaced | Created by callers to request a capture or reference an artifact for restore. | -| `PodSnapshotContent` | Cluster-scoped | System-managed record of the physical artifact, bound to a `PodSnapshot`. Created by the Snapshot operator, never by the caller. | -| `nvidia.com/restore-from` | Namespaced | Added as a pod annotation to trigger restore from a named `PodSnapshot` in the same namespace. | -| `nvidia.com/restore-container-map` | Namespaced | Optional comma-separated `source=destination` mappings used to clone the single captured container into one or more restore containers. | +| `PodSnapshot` | Namespaced | Created by callers to request a checkpoint, or to reference an artifact for restore. | +| `PodSnapshotContent` | Cluster-scoped | System-managed record of the physical artifact, bound to a `PodSnapshot`. Created by the operator, never by the caller. | +| `SnapshotJob` | Namespaced | Runs a pod from a template and checkpoints it into a `PodSnapshot` once ready — a self-contained checkpoint job. | +| `nvidia.com/restore-from` | Namespaced | Pod annotation that triggers a restore from a named `PodSnapshot` in the same namespace. | + +Under the hood, a control-plane operator and a per-node agent perform the CRIU +and `cuda-checkpoint` work; see [Architecture](docs/reference/architecture.md). +The [API reference](docs/reference/api.md) covers the resources and the +checkpoint/restore lifecycle. + +Once Snapshot is installed, follow the **[usage guides](docs/guides/README.md)** +to checkpoint and restore a pod. + +## Limitations -  +Current limitations: +- Single-GPU workloads only. +- x86_64 nodes only. +- vGPU is not supported. +- Runs only on NVIDIA GPUs supported by the required CUDA driver. -## Architecture +Multi-GPU and Arm support are on the roadmap. -Snapshot consists of two main components. +## Documentation -#### Operator +**Get started** -The Kubernetes operator manages the control plane. +- [Usage guides](docs/guides/README.md) — build a snapshot-ready image per inference framework, then checkpoint and restore. -It is responsible for: +**Reference** -* Orchestrating checkpoint and restore operations. -* Tracking snapshot lifecycle. -* Exposing status through Kubernetes resources. -* Managing cleanup. +- [API](docs/reference/api.md) — `PodSnapshot`, `PodSnapshotContent`, `SnapshotJob`, and the `restore-from` annotation. +- [Architecture](docs/reference/architecture.md) — operator and node-agent design, and the checkpoint/restore internals. +- [CLI (`snapshotctl`)](docs/reference/cli.md) — lower-level checkpoint/restore from a pod manifest. +**Operations** -#### Node Agent +- [Installation](docs/operations/install.md) — Helm install and uninstall. +- [Storage](docs/operations/storage.md) — the shared checkpoint volume and how to configure it. +- [Troubleshooting](docs/operations/troubleshooting.md) — common failures and where to look. +- [Security](docs/operations/security.md) — the privileged agent, seccomp, and Pod Security. -A privileged node agent runs on every GPU node. +**Development** -It performs the actual checkpoint and restore operations by invoking CRIU and cuda-checkpoint against live processes. +- [Building from source](docs/development/build-from-source.md) — build the images and install locally. +- [Benchmarks](docs/development/benchmarks.md) — how startup performance is measured. -The node agent is intentionally an implementation detail. Clients never communicate with it directly. +**More** -  +- [Limitations & known issues](docs/limitations.md) — current limitations and what's on the roadmap. -## Design Principles +## Adopters -Snapshot owns the mechanics of checkpoint and restore—not the policy. +[NVIDIA Dynamo](https://github.com/ai-dynamo/dynamo), the open-source +inference-serving stack, integrates Snapshot for GPU cold-start. On Dynamo, Snapshot is available through it directly — see +[Snapshotting GPU Workers](https://docs.nvidia.com/dynamo/latest/kubernetes/operations/cold-start-optimizations/dynamo-snapshot) +in the Dynamo docs. -Systems integrating with Snapshot decide: +## Contributing -* Which workloads should be checkpointed. -* When snapshots should be created. -* When they should be restored. -* How failures should be handled. +Contributions are welcome under the project's [Apache 2.0 license](LICENSE). See +[CONTRIBUTING.md](CONTRIBUTING.md) — all commits must be signed off (DCO). -Snapshot executes those requests and exposes the resulting state. +## Security -Everything Snapshot manages is represented as Kubernetes resources. Snapshot metadata, capture progress, restore status, and lifecycle information are all observable through the Kubernetes API using standard Kubernetes tooling. +To report a security vulnerability, follow the process in [SECURITY.md](SECURITY.md). -Clients interact exclusively through the Kubernetes API. No platform-specific APIs, direct node communication, or custom protocols are required. +## Feedback -  +Feedback and issues are welcome — please [open an issue](https://github.com/ai-dynamo/snapshot/issues). -## Status +## License -The project is in early development. API types and control plane components are scaffolded but not yet feature-complete. Not ready for production use. +Snapshot is licensed under the [Apache License 2.0](LICENSE). diff --git a/docs/development/benchmarks.md b/docs/development/benchmarks.md new file mode 100644 index 0000000..3e148e8 --- /dev/null +++ b/docs/development/benchmarks.md @@ -0,0 +1,3 @@ +# Benchmarks + +_Documentation for this page is in progress._ diff --git a/docs/development/build-from-source.md b/docs/development/build-from-source.md new file mode 100644 index 0000000..1340bfa --- /dev/null +++ b/docs/development/build-from-source.md @@ -0,0 +1,76 @@ +# Building from source + +This guide builds the Snapshot operator and node-agent images from a checkout of +this repository and installs the chart against them. Most users should install +[from a release](../../README.md#from-a-release) instead — build from source when developing Snapshot or testing unreleased changes. + +## Prerequisites + +In addition to the [runtime prerequisites](../../README.md#prerequisites), the build needs: + +- Go (matching the version pinned in the modules) +- Docker with Buildx +- A container registry the cluster can pull from, and push access to it +- `kubectl` and `helm` configured against the cluster + +The node agent is **x86_64 (amd64) only** — `cuda-checkpoint` ships no other +architecture — so its image builds for `linux/amd64`. + +## 1. Clone the repository + +```bash +git clone https://github.com/ai-dynamo/snapshot.git +cd snapshot +``` + +## 2. Build the images + +The root `Makefile` builds both images. Override `REGISTRY` and `VERSION` to tag +them for the registry: + +```bash +make docker-build-agent docker-build-operator \ + REGISTRY= \ + VERSION= +``` + +This produces `/agent:` and +`/operator:`. + +## 3. Push the images + +Push both images to a registry the cluster can pull from: + +```bash +docker push /agent: +docker push /operator: +``` + +## 4. Install the chart against the built images + +Install the chart from the checkout, pointing the operator and agent images at +the built images: + +```bash +helm install snapshot ./charts/snapshot \ + --namespace snapshot --create-namespace \ + --set image.operator.repository=/operator \ + --set image.operator.tag= \ + --set image.agent.repository=/agent \ + --set image.agent.tag= +``` + +See [Installation](../operations/install.md) for storage and uninstall options. + +## Development workflow + +Common `make` targets from the repo root: + +- `make build` — compile the agent and operator +- `make test` — run unit tests across the `api`, `agent`, and `operator` modules +- `make lint` — run linters +- `make helm-lint` — lint the Helm chart +- `make check` — the full pre-merge gate (generate, license headers, fmt, tidy, lint, and more) + +See [CONTRIBUTING.md](../../CONTRIBUTING.md) for the contribution process and DCO +sign-off. diff --git a/docs/guides/README.md b/docs/guides/README.md new file mode 100644 index 0000000..e34f215 --- /dev/null +++ b/docs/guides/README.md @@ -0,0 +1,38 @@ +# Usage guides + +Using Snapshot is a three-stage flow: + +1. **Build and deploy** a snapshot-ready image for the inference framework. Start + from the framework's runtime image, add a small program that cooperates with + Snapshot's checkpoint/restore lifecycle, and deploy it as a replica. Snapshot's agent + injects the restore tooling at runtime. +2. **Checkpoint** the running replica — with a `PodSnapshot` or a `SnapshotJob`. +3. **Restore** into new pods — with the `nvidia.com/restore-from` annotation. + +Stages 2 and 3 are the same for every framework; only the image and deployment in +stage 1 differ. + +> [!NOTE] +> These guides use `kubectl` to show the resources and the flow. In production, a +> controller or platform creates and watches these resources through the Kubernetes +> API as part of its own control loop — `kubectl` here is just for illustration and +> for trying things out by hand. + +## 1. Build and deploy + +Per inference framework: + +- [vLLM](vllm.md) +- [SGLang](sglang.md) +- [TensorRT-LLM](tensorrt-llm.md) + +## 2. Checkpoint + +- [Checkpoint a replica](checkpoint.md) + +## 3. Restore + +- [Restore a replica](restore.md) + +See [Installation](../operations/install.md) for cluster prerequisites and the +[API reference](../reference/api.md) for full resource detail. diff --git a/docs/guides/checkpoint.md b/docs/guides/checkpoint.md new file mode 100644 index 0000000..e9465ee --- /dev/null +++ b/docs/guides/checkpoint.md @@ -0,0 +1,102 @@ +# Checkpoint a replica + +Checkpointing saves an initialized replica's state as a checkpoint artifact. There +are two ways to do it, depending on the use case: + +| Method | Choose it when… | Implication | +|--------|-----------------|-------------| +| **`PodSnapshot`** | The running replica can be controlled and tracked — for example, by a controller or platform that manages inference pods | The most efficient path — it checkpoints a replica that stays serving. It needs orchestration: bringing the replica up, waiting until it is ready, then triggering the checkpoint. | +| **`SnapshotJob`** | The running pod cannot be tracked directly — for example, in a pipeline that submits the work | Snapshot runs the whole flow: it creates the replica, checkpoints it, and tears it down. Self-contained, but the source is discarded, so every replica (including the first) comes up via [restore](restore.md). | + +These examples use `kubectl` to show the flow. In production, an integrating +controller or platform creates and watches these resources through the Kubernetes +API as part of its control loop. + +## Prerequisites + +- Snapshot is [installed](../operations/install.md) in the cluster. +- The pod to checkpoint is a **snapshot-ready pod**, fully initialized (weights + loaded, kernels warmed up). A [snapshot-ready image](README.md) is necessary but + not sufficient — the pod spec itself must also carry what Snapshot relies on to + checkpoint it: + - the `/snapshot-control` volume mount, the control directory Snapshot signals + through; + - the `securityContext` (seccomp profile) that checkpointing requires; + - a readiness gate on `/snapshot-control/ready-for-snapshot`, so the pod reports + Ready only once it is safe to checkpoint; + - the `nvidia.com/snapshot-is-checkpoint-source: "true"` pod label. + +The build-and-deploy guides include a complete, working example of such a pod for +each framework — see the `deployment.yaml` referenced from the [vLLM](vllm.md), +[SGLang](sglang.md), and [TensorRT-LLM](tensorrt-llm.md) guides. Use that pod spec +as the reference: a `PodSnapshot` targets a pod deployed this way, and a +`SnapshotJob`'s `podTemplate` must carry the same fields. + +## Option 1 — `PodSnapshot` (checkpoint a running replica) + +Point at a replica that is already up and serving. Create a `PodSnapshot` naming its +pod and the container to checkpoint: + +```yaml +apiVersion: nvidia.com/v1alpha1 +kind: PodSnapshot +metadata: + name: vllm-snapshot + namespace: my-inference +spec: + source: + podRef: + name: vllm-source- + containers: + - main +``` + +```bash +kubectl apply -f vllm-snapshot.yaml +kubectl wait --for=condition=Ready podsnapshot/vllm-snapshot \ + -n my-inference --timeout=30m +``` + +The operator binds a cluster-scoped `PodSnapshotContent` and records the artifact. +Because the replica keeps running and serving, this is the faster path — the +trade-off is the orchestration it requires: bringing the replica up, waiting for +readiness, then triggering the checkpoint. + +## Option 2 — `SnapshotJob` (checkpoint a throwaway replica) + +`SnapshotJob` runs a replica from a pod template, checkpoints it once ready, and +completes from the resulting `PodSnapshot` — removing the source replica. There is +no long-running replica to manage, which fits pipeline use cases. + +```yaml +apiVersion: nvidia.com/v1alpha1 +kind: SnapshotJob +metadata: + name: vllm-snapshot-job + namespace: my-inference +spec: + podSnapshotTemplate: + targetContainers: + - main + # podTemplate must be a full snapshot-ready pod spec — see Prerequisites and the + # build-and-deploy deployment.yaml (checkpoint-source label, securityContext, + # /snapshot-control mount, and the ready-for-snapshot readiness gate). + podTemplate: + spec: + containers: + - name: main + image: /vllm-snapshot: +``` + +```bash +kubectl apply -f vllm-snapshot-job.yaml +kubectl wait --for=condition=Completed snapshotjob/vllm-snapshot-job \ + -n my-inference --timeout=30m + +# the resulting PodSnapshot to restore from: +kubectl get snapshotjob vllm-snapshot-job -n my-inference \ + -o jsonpath='{.status.podSnapshotName}' +``` + +Because the source replica is deleted, every serving replica — including the +first — is brought up via [restore](restore.md). diff --git a/docs/guides/restore.md b/docs/guides/restore.md new file mode 100644 index 0000000..5979c98 --- /dev/null +++ b/docs/guides/restore.md @@ -0,0 +1,45 @@ +# Restore a replica + +Restoring starts a new replica from a snapshot instead of cold-starting it. A new +pod carries the `nvidia.com/restore-from` annotation, naming the `PodSnapshot` to +restore from; the node agent restores the checkpointed state into the container during +pod startup. + +## Prerequisites + +- A ready `PodSnapshot` exists (see [Checkpoint a replica](checkpoint.md)). +- The new pod uses the same [snapshot-ready image](README.md) and matching replica + configuration. + +## Example + +Add the annotation to the replica pod to restore: + +```yaml +apiVersion: v1 +kind: Pod +metadata: + name: vllm-restored + namespace: my-inference + annotations: + nvidia.com/restore-from: vllm-snapshot +spec: + containers: + - name: main + image: /vllm-snapshot: + # ...the replica configuration that was checkpointed +``` + +```bash +kubectl apply -f vllm-restored.yaml +kubectl get pod vllm-restored -n my-inference -w +``` + +The node agent adds a `snapshot/Restored` condition to the pod once the restore +completes — watch it, along with pod readiness, to confirm. If the restored +workload serves an API, sending a request is a good end-to-end check that it +resumed correctly. + +The restored process resumes from the checkpointed state, skipping model loading +and warm-up. In practice, higher-level systems add this annotation to the pods +they create, rather than applying pods by hand. diff --git a/docs/guides/sglang.md b/docs/guides/sglang.md new file mode 100644 index 0000000..642dc1b --- /dev/null +++ b/docs/guides/sglang.md @@ -0,0 +1,180 @@ +# Build and deploy an SGLang replica + +Snapshot restores a replica by injecting its checkpointed state into a +snapshot-ready image: an SGLang runtime image prepared with the application and container +layout Snapshot expects. The Snapshot agent injects the restore tooling at +runtime. + +## Build + +Start with an SGLang image that includes SGLang, CUDA, and +`torch_memory_saver`. Add one program that prepares SGLang for checkpoint and +resumes it after restore. Select the model when deploying the source pod. + +### 1. Download the example files + +Download [`app.py`](sglang/app.py), +[`Dockerfile.sglang`](sglang/Dockerfile.sglang), +[`model-cache-pvc.yaml`](sglang/model-cache-pvc.yaml), and +[`deployment.yaml`](sglang/deployment.yaml) from the repository: + +```bash +mkdir -p sglang-snapshot-image +cd sglang-snapshot-image + +curl --fail --location \ + --output app.py \ + https://raw.githubusercontent.com/ai-dynamo/snapshot/main/docs/guides/sglang/app.py + +curl --fail --location \ + --output Dockerfile.sglang \ + https://raw.githubusercontent.com/ai-dynamo/snapshot/main/docs/guides/sglang/Dockerfile.sglang + +curl --fail --location \ + --output model-cache-pvc.yaml \ + https://raw.githubusercontent.com/ai-dynamo/snapshot/main/docs/guides/sglang/model-cache-pvc.yaml + +curl --fail --location \ + --output deployment.yaml \ + https://raw.githubusercontent.com/ai-dynamo/snapshot/main/docs/guides/sglang/deployment.yaml +``` + +The program creates a direct `sglang.Engine`, runs one generation, and calls +`TokenizerManager.pause_generation()` followed by +`Engine.release_memory_occupation()`. It writes `ready-for-snapshot` only after +both operations succeed. + +The Deployment enables SGLang's memory saver and CPU weight backup through the +program. An init container downloads the selected model into a persistent +cache. The source application then loads that cache with `HF_HUB_OFFLINE=1` so +the checkpointed process has no open Hugging Face connections. + +After restore, the checkpointed process calls +`Engine.resume_memory_occupation()` and +`TokenizerManager.continue_generation()`. It runs another generation and +starts an API on port 8000. It writes `sglang-restore-ready` only after the +generation succeeds and the API is listening. To validate the restored replica, +send a `POST` request to `/generate` with a JSON body such as +`{"prompt":"What is the capital of Italy?"}`. + +The Dockerfile starts from the tested SGLang image, creates +`/snapshot-control`, and adds `app.py`. The source and restore pods must use the +same immutable image, mount the Snapshot control volume at +`/snapshot-control`, and mount the same model cache at `/hf-cache`. + +### 2. Build the image + +```bash +export SGLANG_RUNTIME_IMAGE=lmsysorg/sglang@sha256:9e148f5ac788e856a06166bd6347a831831eb9fcfab4d1770874823a7c29a1a1 +export SGLANG_SNAPSHOT_IMAGE=/sglang-snapshot: + +docker build \ + --platform linux/amd64 \ + --build-arg SGLANG_RUNTIME_IMAGE="$SGLANG_RUNTIME_IMAGE" \ + -f Dockerfile.sglang \ + -t "$SGLANG_SNAPSHOT_IMAGE" . + +docker push "$SGLANG_SNAPSHOT_IMAGE" +``` + +The `docker push` command uploads the newly built image to the registry named +in `$SGLANG_SNAPSHOT_IMAGE`. Step 3 deploys that image as the source pod. Use +the same full image name and tag for restored pods. + +Verify that the packaged image contains SGLang, `torch_memory_saver`, and +`app.py`: + +```bash +docker run --rm \ + --platform linux/amd64 \ + --entrypoint python3 \ + "$SGLANG_SNAPSHOT_IMAGE" \ + -c 'import pathlib; import sglang; import torch_memory_saver; assert pathlib.Path("/app/app.py").is_file()' +``` + +The command produces no output when all three components are present. Any +failure prints an error and returns a non-zero exit status. + +### 3. Deploy SGLang + +Set the namespace where the SGLang pod will run: + +```bash +export SGLANG_NAMESPACE= +kubectl get namespace "$SGLANG_NAMESPACE" +``` + +Set the model through the `SNAPSHOT_MODEL` environment variable in both the +init container and the main container in +[`deployment.yaml`](sglang/deployment.yaml): + +```yaml +env: + - name: SNAPSHOT_MODEL + value: Qwen/Qwen3-0.6B +``` + +The example configures a context length of 10240 tokens for a 24 GiB NVIDIA A10 +GPU. Reduce `SGLANG_CONTEXT_LENGTH` for a smaller GPU or increase it only after +validating the resulting memory use. + +> [!NOTE] +> This example runs SGLang directly through `sglang.Engine` rather than +> `sglang.launch_server`, so the standard server's command-line arguments do not +> apply. The model is selected with `SNAPSHOT_MODEL`, and other runtime settings +> are supplied through SGLang's [environment variables](https://docs.sglang.ai/references/environment_variables.html) +> set in the Deployment's Pod template. + +Create the persistent model cache: + +```bash +kubectl apply \ + --namespace "$SGLANG_NAMESPACE" \ + --filename model-cache-pvc.yaml +``` + +Use [`deployment.yaml`](sglang/deployment.yaml) to deploy the image built in +step 2: + +```bash +kubectl set image \ + --local \ + --filename deployment.yaml \ + model-cache="$SGLANG_SNAPSHOT_IMAGE" \ + main="$SGLANG_SNAPSHOT_IMAGE" \ + --output yaml | + kubectl apply \ + --namespace "$SGLANG_NAMESPACE" \ + --filename - +``` + +The command replaces both example image values in `deployment.yaml` with +`$SGLANG_SNAPSHOT_IMAGE` before creating the Deployment. It does not modify the +local file. The init container downloads the model when its cache marker does +not exist. The main container then starts SGLang from the offline cache. + +Wait until the SGLang replica finishes initialization and becomes safe to +checkpoint: + +```bash +kubectl rollout status \ + --namespace "$SGLANG_NAMESPACE" \ + deployment/sglang-source \ + --timeout=30m +``` + +List the generated Pod: + +```bash +kubectl get pods \ + --namespace "$SGLANG_NAMESPACE" \ + --selector app=sglang-source +``` + +Use that Pod name in the `PodSnapshot` created during the next step. The +readiness probe succeeds after `app.py` writes `ready-for-snapshot`. + +## Next steps + +- [Checkpoint a replica](checkpoint.md) +- [Restore a replica](restore.md) diff --git a/docs/guides/sglang/Dockerfile.sglang b/docs/guides/sglang/Dockerfile.sglang new file mode 100644 index 0000000..1468b08 --- /dev/null +++ b/docs/guides/sglang/Dockerfile.sglang @@ -0,0 +1,23 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +ARG SGLANG_RUNTIME_IMAGE=lmsysorg/sglang@sha256:9e148f5ac788e856a06166bd6347a831831eb9fcfab4d1770874823a7c29a1a1 +FROM ${SGLANG_RUNTIME_IMAGE} + +ARG TARGETARCH=amd64 + +ENV HF_HUB_DISABLE_XET=1 + +USER root + +RUN set -eux; \ + if [ "${TARGETARCH}" != "amd64" ]; then \ + echo "Snapshot requires x86_64" >&2; \ + exit 1; \ + fi; \ + mkdir -p /snapshot-control + +WORKDIR /app +COPY app.py ./ +ENTRYPOINT ["python3", "-u", "/app/app.py"] +CMD ["--mode", "snapshot"] diff --git a/docs/guides/sglang/app.py b/docs/guides/sglang/app.py new file mode 100644 index 0000000..b0cafe2 --- /dev/null +++ b/docs/guides/sglang/app.py @@ -0,0 +1,159 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +import argparse +import json +import os +import time +from http.server import BaseHTTPRequestHandler, HTTPServer +from pathlib import Path +from typing import Any + +CONTROL_DIR = Path(os.environ.get("SNAPSHOT_CONTROL_DIR", "/snapshot-control")) + + +def parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser() + parser.add_argument("--mode", choices=("prime", "snapshot"), default="snapshot") + return parser.parse_args() + + +def configure_capture_environment() -> None: + os.environ.update( + { + "NCCL_CUMEM_ENABLE": "0", + "NCCL_NVLS_ENABLE": "0", + "NCCL_IB_DISABLE": "1", + "NCCL_RAS_ENABLE": "0", + "TORCH_NCCL_ENABLE_MONITORING": "0", + "TORCH_NCCL_DUMP_ON_TIMEOUT": "0", + "HF_HUB_OFFLINE": "1", + } + ) + + +def create_engine(snapshot_mode: bool) -> Any: + import sglang as sgl + + return sgl.Engine( + model_path=os.environ["SNAPSHOT_MODEL"], + context_length=int(os.environ.get("SGLANG_CONTEXT_LENGTH", "10240")), + enable_memory_saver=snapshot_mode, + enable_weights_cpu_backup=snapshot_mode, + log_level="info", + ) + + +def generate_text(engine: Any, prompt: str) -> str: + result = engine.generate( + prompt, + sampling_params={"temperature": 0, "max_new_tokens": 16}, + ) + text = result.get("text", "").strip() + if not text: + raise RuntimeError("SGLang produced empty output") + return text + + +def pause_generation(engine: Any) -> None: + from sglang.srt.managers.io_struct import PauseGenerationReqInput + + engine.loop.run_until_complete( + engine.tokenizer_manager.pause_generation( + PauseGenerationReqInput(mode="abort") + ) + ) + + +def continue_generation(engine: Any) -> None: + from sglang.srt.managers.io_struct import ContinueGenerationReqInput + + engine.loop.run_until_complete( + engine.tokenizer_manager.continue_generation(ContinueGenerationReqInput()) + ) + + +def serve_api(engine: Any, restored_text: str) -> None: + class GenerateHandler(BaseHTTPRequestHandler): + def do_POST(self) -> None: + if self.path != "/generate": + self.send_error(404) + return + + try: + length = int(self.headers.get("Content-Length", "0")) + payload = json.loads(self.rfile.read(length)) + prompt = payload["prompt"] + if not isinstance(prompt, str) or not prompt.strip(): + raise ValueError("prompt must be a non-empty string") + text = generate_text(engine, prompt) + except (json.JSONDecodeError, KeyError, TypeError, ValueError) as error: + self.send_error(400, str(error)) + return + except Exception as error: + self.send_error(500, str(error)) + return + + body = json.dumps({"text": text}).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + server = HTTPServer(("0.0.0.0", 8000), GenerateHandler) + CONTROL_DIR.joinpath("sglang-restore-ready").write_text( + restored_text + "\n", + encoding="utf-8", + ) + print("SGLang API listening on port 8000", flush=True) + server.serve_forever() + + +def main() -> None: + if os.environ.get("DYN_SNAPSHOT_RESTORE_STANDBY") == "1": + while True: + time.sleep(3600) + + args = parse_args() + snapshot_mode = args.mode == "snapshot" + if snapshot_mode: + configure_capture_environment() + CONTROL_DIR.joinpath("ready-for-snapshot").unlink(missing_ok=True) + + engine = create_engine(snapshot_mode) + try: + text = generate_text(engine, "The capital city of France is") + print(f"SGLang pre-checkpoint output={text!r}", flush=True) + + if not snapshot_mode: + return + + pause_generation(engine) + try: + engine.release_memory_occupation() + except BaseException: + continue_generation(engine) + raise + + CONTROL_DIR.joinpath("ready-for-snapshot").write_text( + "ready\n", + encoding="utf-8", + ) + + while not CONTROL_DIR.joinpath("restore-complete").exists(): + time.sleep(1) + + engine.resume_memory_occupation() + continue_generation(engine) + os.environ.pop("HF_HUB_OFFLINE", None) + + text = generate_text(engine, "The capital city of Germany is") + print(f"SGLang restored output={text!r}", flush=True) + serve_api(engine, text) + finally: + engine.shutdown() + + +if __name__ == "__main__": + main() diff --git a/docs/guides/sglang/deployment.yaml b/docs/guides/sglang/deployment.yaml new file mode 100644 index 0000000..a716c26 --- /dev/null +++ b/docs/guides/sglang/deployment.yaml @@ -0,0 +1,118 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: sglang-source +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: sglang-source + template: + metadata: + labels: + app: sglang-source + nvidia.com/snapshot-is-checkpoint-source: "true" + spec: + terminationGracePeriodSeconds: 1 + runtimeClassName: nvidia + securityContext: + seccompProfile: + type: Localhost + localhostProfile: profiles/block-iouring.json + nodeSelector: + nvidia.com/gpu.present: "true" + initContainers: + - name: model-cache + image: sglang-snapshot:replace-me + imagePullPolicy: Always + command: + - python3 + - -c + args: + - | + import os + from pathlib import Path + from huggingface_hub import snapshot_download + model = os.environ["SNAPSHOT_MODEL"] + marker = Path("/hf-cache") / (".snapshot-model-" + model.replace("/", "--")) + if not marker.exists(): + snapshot_download(repo_id=model) + marker.touch() + env: + - name: SNAPSHOT_MODEL + value: Qwen/Qwen3-0.6B + - name: HF_HOME + value: /hf-cache + - name: HF_HUB_DISABLE_XET + value: "1" + volumeMounts: + - name: model-cache + mountPath: /hf-cache + containers: + - name: main + image: sglang-snapshot:replace-me + imagePullPolicy: Always + args: + - --mode + - snapshot + env: + - name: SNAPSHOT_MODEL + value: Qwen/Qwen3-0.6B + - name: SGLANG_CONTEXT_LENGTH + value: "10240" + - name: SNAPSHOT_CONTROL_DIR + value: /snapshot-control + - name: HF_HOME + value: /hf-cache + - name: HF_HUB_OFFLINE + value: "1" + - name: HF_HUB_DISABLE_XET + value: "1" + - name: NCCL_CUMEM_ENABLE + value: "0" + - name: NCCL_NVLS_ENABLE + value: "0" + - name: NCCL_IB_DISABLE + value: "1" + - name: NCCL_RAS_ENABLE + value: "0" + - name: TORCH_NCCL_ENABLE_MONITORING + value: "0" + - name: TORCH_NCCL_DUMP_ON_TIMEOUT + value: "0" + ports: + - name: api + containerPort: 8000 + resources: + limits: + nvidia.com/gpu: "1" + readinessProbe: + exec: + command: + - cat + - /snapshot-control/ready-for-snapshot + periodSeconds: 1 + failureThreshold: 1800 + volumeMounts: + - name: model-cache + mountPath: /hf-cache + - name: snapshot-control + mountPath: /snapshot-control + subPath: main + - name: tun + mountPath: /dev/net/tun + volumes: + - name: model-cache + persistentVolumeClaim: + claimName: sglang-model-cache + - name: snapshot-control + emptyDir: {} + - name: tun + hostPath: + path: /dev/net/tun + type: CharDevice diff --git a/docs/guides/sglang/model-cache-pvc.yaml b/docs/guides/sglang/model-cache-pvc.yaml new file mode 100644 index 0000000..c19230b --- /dev/null +++ b/docs/guides/sglang/model-cache-pvc.yaml @@ -0,0 +1,13 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +apiVersion: v1 +kind: PersistentVolumeClaim +metadata: + name: sglang-model-cache +spec: + accessModes: + - ReadWriteOnce + resources: + requests: + storage: 20Gi diff --git a/docs/guides/tensorrt-llm.md b/docs/guides/tensorrt-llm.md new file mode 100644 index 0000000..946b97b --- /dev/null +++ b/docs/guides/tensorrt-llm.md @@ -0,0 +1,167 @@ +# Build and deploy a TensorRT-LLM replica + +Snapshot restores a replica by injecting its checkpointed state into a +snapshot-ready image: a TensorRT-LLM runtime image prepared with the application and container +layout Snapshot expects. The Snapshot agent injects the restore tooling at +runtime. + +> [!NOTE] +> TensorRT-LLM support is experimental and currently limited to a single GPU. + +## Build + +Start with the TensorRT-LLM runtime image, which includes TensorRT-LLM and its +runtime dependencies. Add one program that prepares TensorRT-LLM for checkpoint +and validates it after restore. Select the model when deploying the source pod. + +### 1. Download the example files + +Download [`app.py`](tensorrt-llm/app.py), +[`Dockerfile.tensorrt-llm`](tensorrt-llm/Dockerfile.tensorrt-llm), and +[`deployment.yaml`](tensorrt-llm/deployment.yaml) from the repository: + +```bash +mkdir -p tensorrt-llm-snapshot-image +cd tensorrt-llm-snapshot-image + +curl --fail --location \ + --output app.py \ + https://raw.githubusercontent.com/ai-dynamo/snapshot/main/docs/guides/tensorrt-llm/app.py + +curl --fail --location \ + --output Dockerfile.tensorrt-llm \ + https://raw.githubusercontent.com/ai-dynamo/snapshot/main/docs/guides/tensorrt-llm/Dockerfile.tensorrt-llm + +curl --fail --location \ + --output deployment.yaml \ + https://raw.githubusercontent.com/ai-dynamo/snapshot/main/docs/guides/tensorrt-llm/deployment.yaml +``` + +The program loads the model selected in `deployment.yaml` and calls +`LLM.generate()` to initialize TensorRT-LLM. The synchronous call returns only +after generation finishes, so no request remains in flight. The program runs +`gc.collect()` and writes `ready-for-snapshot` when it reaches the safe checkpoint +point. + +TensorRT-LLM does not use a framework pause or sleep call in this example. The +model and initialized CUDA state remain resident. After restore, the checkpointed +process calls `LLM.generate()` again and starts an API on port 8000. It writes +`trtllm-restore-ready` only after the generation succeeds and the API is +listening. To validate the restored replica, send a `POST` request to +`/generate` with a JSON body such as +`{"prompt":"What is the capital of Italy?"}`. + +The Dockerfile starts from the tested TensorRT-LLM 1.3.0 release candidate +image, creates `/snapshot-control`, and adds `app.py`. +`TLLM_NCCL_SYMMETRIC_ZERO_COPY=0` disables NCCL registered windows that CUDA +checkpoint does not support. `UCX_TLS=tcp,self` avoids RDMA mappings that CRIU +cannot restore. + +The source and restore pods must use the same immutable image and mount the +Snapshot control volume at `/snapshot-control`. + +### 2. Build the image + +```bash +export TENSORRT_LLM_RUNTIME_IMAGE=nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc23 +export TENSORRT_LLM_SNAPSHOT_IMAGE=/tensorrt-llm-snapshot: + +docker build \ + --platform linux/amd64 \ + --build-arg TENSORRT_LLM_RUNTIME_IMAGE="$TENSORRT_LLM_RUNTIME_IMAGE" \ + -f Dockerfile.tensorrt-llm \ + -t "$TENSORRT_LLM_SNAPSHOT_IMAGE" . + +docker push "$TENSORRT_LLM_SNAPSHOT_IMAGE" +``` + +The `docker push` command uploads the newly built image to the registry named +in `$TENSORRT_LLM_SNAPSHOT_IMAGE`. Step 3 deploys that image as the source pod. +Use the same full image name and tag for restored pods. + +Verify that the packaged image contains TensorRT-LLM and `app.py`: + +```bash +docker run --rm \ + --platform linux/amd64 \ + --entrypoint python3 \ + "$TENSORRT_LLM_SNAPSHOT_IMAGE" \ + -c 'import pathlib; import tensorrt_llm; assert pathlib.Path("/app/app.py").is_file()' +``` + +The command produces no output when both TensorRT-LLM and `/app/app.py` are +present. Any failure prints an error and returns a non-zero exit status. + +### 3. Deploy TensorRT-LLM + +Set the namespace where the TensorRT-LLM pod will run: + +```bash +export TENSORRT_LLM_NAMESPACE= +kubectl get namespace "$TENSORRT_LLM_NAMESPACE" +``` + +Set a model supported by the selected TensorRT-LLM image through the +`SNAPSHOT_MODEL` environment variable in +[`deployment.yaml`](tensorrt-llm/deployment.yaml): + +```yaml +env: + - name: SNAPSHOT_MODEL + value: Qwen/Qwen3-0.6B +``` + +The example uses one GPU, the PyTorch backend, and a maximum sequence length of +512 tokens. Revalidate checkpoint and restore before changing the model, +TensorRT-LLM image, GPU count, backend, or engine settings. + +> [!NOTE] +> This example runs TensorRT-LLM through the `LLM` API rather than `trtllm-serve`, +> so the standard `trtllm-serve` command-line arguments do not apply. The model is +> selected with `SNAPSHOT_MODEL`, and other engine settings are configured on the +> [`LLM` API](https://nvidia.github.io/TensorRT-LLM/llm-api/reference.html) in +> `app.py`. + +Use [`deployment.yaml`](tensorrt-llm/deployment.yaml) to deploy the image built +in step 2: + +```bash +kubectl set image \ + --local \ + --filename deployment.yaml \ + main="$TENSORRT_LLM_SNAPSHOT_IMAGE" \ + --output yaml | + kubectl apply \ + --namespace "$TENSORRT_LLM_NAMESPACE" \ + --filename - +``` + +The command replaces the example image value in `deployment.yaml` with +`$TENSORRT_LLM_SNAPSHOT_IMAGE` before creating the Deployment. It does not +modify the local file. + +Wait until the TensorRT-LLM replica finishes initialization and becomes safe to +checkpoint: + +```bash +kubectl rollout status \ + --namespace "$TENSORRT_LLM_NAMESPACE" \ + deployment/tensorrt-llm-source \ + --timeout=30m +``` + +List the generated Pod: + +```bash +kubectl get pods \ + --namespace "$TENSORRT_LLM_NAMESPACE" \ + --selector app=tensorrt-llm-source +``` + +Use that Pod name in the `PodSnapshot` created during the next step. The +readiness probe succeeds after `app.py` writes `ready-for-snapshot`. + +## Next steps + +- [Checkpoint a replica](checkpoint.md) +- [Restore a replica](restore.md) diff --git a/docs/guides/tensorrt-llm/Dockerfile.tensorrt-llm b/docs/guides/tensorrt-llm/Dockerfile.tensorrt-llm new file mode 100644 index 0000000..1d36939 --- /dev/null +++ b/docs/guides/tensorrt-llm/Dockerfile.tensorrt-llm @@ -0,0 +1,24 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +ARG TENSORRT_LLM_RUNTIME_IMAGE=nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc23 +FROM ${TENSORRT_LLM_RUNTIME_IMAGE} + +ARG TARGETARCH=amd64 + +ENV HF_HUB_DISABLE_XET=1 \ + TLLM_NCCL_SYMMETRIC_ZERO_COPY=0 \ + UCX_TLS=tcp,self + +USER root + +RUN set -eux; \ + if [ "${TARGETARCH}" != "amd64" ]; then \ + echo "Snapshot requires x86_64" >&2; \ + exit 1; \ + fi; \ + mkdir -p /snapshot-control + +WORKDIR /app +COPY app.py ./ +ENTRYPOINT ["python3", "/app/app.py"] diff --git a/docs/guides/tensorrt-llm/app.py b/docs/guides/tensorrt-llm/app.py new file mode 100644 index 0000000..6eb008b --- /dev/null +++ b/docs/guides/tensorrt-llm/app.py @@ -0,0 +1,115 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +import gc +import json +import os +import time +from http.server import BaseHTTPRequestHandler, HTTPServer +from pathlib import Path + +from tensorrt_llm import LLM, SamplingParams + +CONTROL_DIR = Path(os.environ.get("SNAPSHOT_CONTROL_DIR", "/snapshot-control")) +MODEL = os.environ["SNAPSHOT_MODEL"] + + +def generate_text(llm: LLM, prompts: list[str]) -> list[str]: + outputs = llm.generate( + prompts, + SamplingParams(temperature=0.0, max_tokens=16), + use_tqdm=False, + ) + texts = [] + for output in outputs: + if not output.outputs: + raise RuntimeError("TensorRT-LLM produced no output") + text = output.outputs[0].text.strip() + if not text: + raise RuntimeError("TensorRT-LLM produced empty output") + texts.append(text) + return texts + + +def serve_api(llm: LLM, restored_text: str) -> None: + class GenerateHandler(BaseHTTPRequestHandler): + def do_POST(self) -> None: + if self.path != "/generate": + self.send_error(404) + return + + try: + length = int(self.headers.get("Content-Length", "0")) + payload = json.loads(self.rfile.read(length)) + prompt = payload["prompt"] + if not isinstance(prompt, str) or not prompt.strip(): + raise ValueError("prompt must be a non-empty string") + text = generate_text(llm, [prompt])[0] + except (json.JSONDecodeError, KeyError, TypeError, ValueError) as error: + self.send_error(400, str(error)) + return + except Exception as error: + self.send_error(500, str(error)) + return + + body = json.dumps({"text": text}).encode() + self.send_response(200) + self.send_header("Content-Type", "application/json") + self.send_header("Content-Length", str(len(body))) + self.end_headers() + self.wfile.write(body) + + server = HTTPServer(("0.0.0.0", 8000), GenerateHandler) + CONTROL_DIR.joinpath("trtllm-restore-ready").write_text( + restored_text + "\n", + encoding="utf-8", + ) + print("TensorRT-LLM API listening on port 8000", flush=True) + server.serve_forever() + + +def main() -> None: + if os.environ.get("DYN_SNAPSHOT_RESTORE_STANDBY") == "1": + while True: + time.sleep(3600) + + CONTROL_DIR.joinpath("ready-for-snapshot").unlink(missing_ok=True) + + llm = LLM( + model=MODEL, + backend="pytorch", + dtype="float16", + trust_remote_code=True, + tensor_parallel_size=1, + max_num_tokens=1024, + max_seq_len=512, + max_batch_size=1, + enable_chunked_prefill=False, + kv_cache_config={"free_gpu_memory_fraction": 0.10}, + ) + + for text in generate_text( + llm, + [ + "Summarize why checkpoint and restore testing matters.", + "Continue this sequence with four numbers: 1, 2, 3, 4,", + ], + ): + print(f"TensorRT-LLM pre-checkpoint output={text!r}", flush=True) + + gc.collect() + CONTROL_DIR.joinpath("ready-for-snapshot").write_text( + "ready\n", + encoding="utf-8", + ) + + while True: + if CONTROL_DIR.joinpath("restore-complete").exists(): + text = generate_text(llm, ["Reply with one word: restored"])[0] + print(f"TensorRT-LLM restored output={text!r}", flush=True) + serve_api(llm, text) + time.sleep(1) + + +if __name__ == "__main__": + main() diff --git a/docs/guides/tensorrt-llm/deployment.yaml b/docs/guides/tensorrt-llm/deployment.yaml new file mode 100644 index 0000000..61f4cd7 --- /dev/null +++ b/docs/guides/tensorrt-llm/deployment.yaml @@ -0,0 +1,72 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: tensorrt-llm-source +spec: + replicas: 1 + progressDeadlineSeconds: 1800 + strategy: + type: Recreate + selector: + matchLabels: + app: tensorrt-llm-source + template: + metadata: + labels: + app: tensorrt-llm-source + nvidia.com/snapshot-is-checkpoint-source: "true" + spec: + terminationGracePeriodSeconds: 1 + runtimeClassName: nvidia + securityContext: + fsGroup: 1000 + fsGroupChangePolicy: OnRootMismatch + seccompProfile: + type: Localhost + localhostProfile: profiles/block-iouring.json + nodeSelector: + nvidia.com/gpu.present: "true" + containers: + - name: main + image: tensorrt-llm-snapshot:replace-me + imagePullPolicy: Always + env: + - name: SNAPSHOT_MODEL + value: Qwen/Qwen3-0.6B + - name: SNAPSHOT_CONTROL_DIR + value: /snapshot-control + - name: HF_HUB_DISABLE_XET + value: "1" + - name: TLLM_NCCL_SYMMETRIC_ZERO_COPY + value: "0" + - name: UCX_TLS + value: tcp,self + ports: + - name: api + containerPort: 8000 + resources: + limits: + nvidia.com/gpu: "1" + readinessProbe: + exec: + command: + - cat + - /snapshot-control/ready-for-snapshot + periodSeconds: 1 + failureThreshold: 1800 + volumeMounts: + - name: snapshot-control + mountPath: /snapshot-control + subPath: main + - name: tun + mountPath: /dev/net/tun + volumes: + - name: snapshot-control + emptyDir: {} + - name: tun + hostPath: + path: /dev/net/tun + type: CharDevice diff --git a/docs/guides/vllm.md b/docs/guides/vllm.md new file mode 100644 index 0000000..076c0a3 --- /dev/null +++ b/docs/guides/vllm.md @@ -0,0 +1,160 @@ +# Build and deploy a vLLM replica + +Snapshot restores a replica by injecting its checkpointed state into a +snapshot-ready image: a vLLM runtime image prepared with the application and +container layout Snapshot expects. The Snapshot agent injects the restore +tooling at runtime. + +## Build + +Start with the official vLLM image, which includes vLLM and its runtime +dependencies. Add one program that prepares vLLM for checkpoint and resumes it +after restore. Select the model when deploying the source pod. + +### 1. Download the example files + +Download [`app.py`](vllm/app.py), +[`Dockerfile.vllm`](vllm/Dockerfile.vllm), and +[`deployment.yaml`](vllm/deployment.yaml) from the repository: + +```bash +mkdir -p vllm-snapshot-image +cd vllm-snapshot-image + +curl --fail --location \ + --output app.py \ + https://raw.githubusercontent.com/ai-dynamo/snapshot/main/docs/guides/vllm/app.py + +curl --fail --location \ + --output Dockerfile.vllm \ + https://raw.githubusercontent.com/ai-dynamo/snapshot/main/docs/guides/vllm/Dockerfile.vllm + +curl --fail --location \ + --output deployment.yaml \ + https://raw.githubusercontent.com/ai-dynamo/snapshot/main/docs/guides/vllm/deployment.yaml +``` + +The program loads the model selected in `deployment.yaml`, runs one +generation to initialize vLLM, and then calls `pause_generation()` and +`sleep()`. It writes +`ready-for-snapshot` only when the process is safe to checkpoint. In a restore +container, it waits in standby until Snapshot injects the checkpointed process. +That process calls `wake_up()` and `resume_generation()`, runs another +generation, starts an API, and writes `vllm-restore-ready` when the API is +listening. To validate the restored replica, send a `POST` request to +`/generate` with a JSON body such as +`{"prompt":"What is the capital of Italy?"}`. + +The Dockerfile starts from the official vLLM 0.27.1 image and installs the +Ubuntu 24.04 glibc required by the current Snapshot restore bundle. It creates +`/snapshot-control` and adds `app.py`. +`HF_HUB_DISABLE_XET=1` prevents the model downloader from leaving an open cache +log that CRIU cannot reopen after restore. + +The source and restore pods must mount the Snapshot control volume at +`/snapshot-control`. + +### 2. Build the image + +```bash +export VLLM_RUNTIME_IMAGE=vllm/vllm-openai:v0.27.1 +export VLLM_SNAPSHOT_IMAGE=/vllm-snapshot: + +docker build \ + --platform linux/amd64 \ + --build-arg VLLM_RUNTIME_IMAGE="$VLLM_RUNTIME_IMAGE" \ + -f Dockerfile.vllm \ + -t "$VLLM_SNAPSHOT_IMAGE" . + +docker push "$VLLM_SNAPSHOT_IMAGE" +``` + +The `docker push` command uploads the newly built image to the registry named +in `$VLLM_SNAPSHOT_IMAGE`. Step 3 deploys that image as the source pod. Use the +same full image name and tag for restored pods. + +Verify that the packaged image contains vLLM and `app.py`: + +```bash +docker run --rm \ + --platform linux/amd64 \ + --entrypoint python3 \ + "$VLLM_SNAPSHOT_IMAGE" \ + -c 'import pathlib; import vllm; assert pathlib.Path("/app/app.py").is_file()' +``` + +The command produces no output when both vLLM and `/app/app.py` are present. +Any failure prints an error and returns a non-zero exit status. + +### 3. Deploy vLLM + +Set the namespace where the vLLM pod will run: + +```bash +export VLLM_NAMESPACE= +kubectl get namespace "$VLLM_NAMESPACE" +``` + +Set the model through the `SNAPSHOT_MODEL` environment variable in +[`deployment.yaml`](vllm/deployment.yaml): + +```yaml +env: + - name: SNAPSHOT_MODEL + value: Qwen/Qwen3-0.6B +``` + +Other values include `TinyLlama/TinyLlama-1.1B-Chat-v1.0` or a mounted model +path such as `/models/Qwen3-0.6B`. A mounted path must be available to both the +source and restored containers. + +> [!NOTE] +> This example runs vLLM directly through `AsyncLLM` rather than `vllm serve`, so +> the standard `vllm serve` command-line arguments do not apply. The model is +> selected with `SNAPSHOT_MODEL`, and other runtime settings are supplied through +> vLLM's [environment variables](https://docs.vllm.ai/en/v0.27.1/configuration/env_vars/) +> set in the Deployment's Pod template. + +Use [`deployment.yaml`](vllm/deployment.yaml) to deploy the image built in +step 2: + +```bash +kubectl set image \ + --local \ + --filename deployment.yaml \ + main="$VLLM_SNAPSHOT_IMAGE" \ + --output yaml | + kubectl apply \ + --namespace "$VLLM_NAMESPACE" \ + --filename - +``` + +The command replaces the example image value in `deployment.yaml` with +`$VLLM_SNAPSHOT_IMAGE` before creating the Deployment. It does not modify the +local file. + +Wait until the vLLM replica finishes initialization and becomes safe to +checkpoint: + +```bash +kubectl rollout status \ + --namespace "$VLLM_NAMESPACE" \ + deployment/vllm-source \ + --timeout=30m +``` + +List the generated Pod: + +```bash +kubectl get pods \ + --namespace "$VLLM_NAMESPACE" \ + --selector app=vllm-source +``` + +Use that Pod name in the `PodSnapshot` created during the next step. The +readiness probe succeeds after `app.py` writes `ready-for-snapshot`. + +## Next steps + +- [Checkpoint a replica](checkpoint.md) +- [Restore a replica](restore.md) diff --git a/docs/guides/vllm/Dockerfile.vllm b/docs/guides/vllm/Dockerfile.vllm new file mode 100644 index 0000000..11859f4 --- /dev/null +++ b/docs/guides/vllm/Dockerfile.vllm @@ -0,0 +1,28 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +ARG VLLM_RUNTIME_IMAGE=vllm/vllm-openai:v0.27.1 +FROM ${VLLM_RUNTIME_IMAGE} + +ARG TARGETARCH=amd64 + +ENV HF_HUB_DISABLE_XET=1 + +USER root + +RUN set -eux; \ + if [ "${TARGETARCH}" != "amd64" ]; then \ + echo "Snapshot requires x86_64" >&2; \ + exit 1; \ + fi; \ + printf 'deb http://archive.ubuntu.com/ubuntu noble main universe\n' \ + >/etc/apt/sources.list.d/snapshot-noble.list; \ + apt-get update; \ + apt-get install -y --no-install-recommends -t noble libc6 libc-bin; \ + rm -f /etc/apt/sources.list.d/snapshot-noble.list; \ + rm -rf /var/lib/apt/lists/*; \ + mkdir -p /snapshot-control + +WORKDIR /app +COPY app.py ./ +ENTRYPOINT ["python3", "/app/app.py"] diff --git a/docs/guides/vllm/app.py b/docs/guides/vllm/app.py new file mode 100644 index 0000000..604f336 --- /dev/null +++ b/docs/guides/vllm/app.py @@ -0,0 +1,124 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +import asyncio +import os +from pathlib import Path +from uuid import uuid4 + +import uvicorn +from fastapi import FastAPI +from pydantic import BaseModel, Field +from vllm import SamplingParams +from vllm.engine.arg_utils import AsyncEngineArgs +from vllm.usage.usage_lib import UsageContext +from vllm.v1.engine.async_llm import AsyncLLM + +CONTROL_DIR = Path(os.environ.get("SNAPSHOT_CONTROL_DIR", "/snapshot-control")) +MODEL = os.environ["SNAPSHOT_MODEL"] + + +class GenerateRequest(BaseModel): + prompt: str = Field(min_length=1) + + +async def generate_text( + engine: AsyncLLM, + prompt: str, + request_id: str, +) -> str: + result = None + async for output in engine.generate( + prompt, + SamplingParams(temperature=0.0, max_tokens=8), + request_id, + ): + result = output + if result is None or not result.outputs: + raise RuntimeError("vLLM produced no output") + text = result.outputs[0].text.strip() + if not text: + raise RuntimeError("vLLM produced empty output") + return text + + +async def serve_api(engine: AsyncLLM, restored_text: str) -> None: + app = FastAPI() + + @app.post("/generate") + async def generate(request: GenerateRequest) -> dict[str, str]: + text = await generate_text( + engine, + request.prompt, + f"request-{uuid4().hex}", + ) + return {"text": text} + + server = uvicorn.Server( + uvicorn.Config( + app, + host="0.0.0.0", + port=8000, + ) + ) + server_task = asyncio.create_task(server.serve()) + while not server.started: + if server_task.done(): + await server_task + raise RuntimeError("API stopped before startup") + await asyncio.sleep(0.1) + + CONTROL_DIR.joinpath("vllm-restore-ready").write_text( + restored_text + "\n", + encoding="utf-8", + ) + print("vLLM API listening on port 8000", flush=True) + await server_task + + +async def main() -> None: + if os.environ.get("DYN_SNAPSHOT_RESTORE_STANDBY") == "1": + await asyncio.Event().wait() + + CONTROL_DIR.joinpath("ready-for-snapshot").unlink(missing_ok=True) + + engine = AsyncLLM.from_engine_args( + AsyncEngineArgs( + model=MODEL, + enable_sleep_mode=True, + ), + usage_context=UsageContext.LLM_CLASS, + ) + + text = await generate_text( + engine, + "Reply with one word: ready", + "snapshot-preflight", + ) + print(f"vLLM pre-checkpoint output={text!r}", flush=True) + + await engine.pause_generation() + await engine.sleep() + CONTROL_DIR.joinpath("ready-for-snapshot").write_text( + "ready\n", + encoding="utf-8", + ) + + while True: + if CONTROL_DIR.joinpath("restore-complete").exists(): + await engine.wake_up() + await engine.resume_generation() + await engine.check_health() + text = await generate_text( + engine, + "Reply with one word: restored", + "snapshot-restore-check", + ) + print(f"vLLM restored output={text!r}", flush=True) + await serve_api(engine, text) + await asyncio.sleep(1) + + +if __name__ == "__main__": + asyncio.run(main()) + os._exit(0) diff --git a/docs/guides/vllm/deployment.yaml b/docs/guides/vllm/deployment.yaml new file mode 100644 index 0000000..d847f6b --- /dev/null +++ b/docs/guides/vllm/deployment.yaml @@ -0,0 +1,63 @@ +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +apiVersion: apps/v1 +kind: Deployment +metadata: + name: vllm-source +spec: + replicas: 1 + strategy: + type: Recreate + selector: + matchLabels: + app: vllm-source + template: + metadata: + labels: + app: vllm-source + nvidia.com/snapshot-is-checkpoint-source: "true" + spec: + terminationGracePeriodSeconds: 1 + runtimeClassName: nvidia + securityContext: + seccompProfile: + type: Localhost + localhostProfile: profiles/block-iouring.json + nodeSelector: + nvidia.com/gpu.present: "true" + containers: + - name: main + image: vllm-snapshot:replace-me + imagePullPolicy: Always + env: + - name: SNAPSHOT_MODEL + value: Qwen/Qwen3-0.6B + - name: SNAPSHOT_CONTROL_DIR + value: /snapshot-control + ports: + - name: api + containerPort: 8000 + resources: + limits: + nvidia.com/gpu: "1" + readinessProbe: + exec: + command: + - cat + - /snapshot-control/ready-for-snapshot + periodSeconds: 1 + failureThreshold: 1200 + volumeMounts: + - name: snapshot-control + mountPath: /snapshot-control + subPath: main + - name: tun + mountPath: /dev/net/tun + volumes: + - name: snapshot-control + emptyDir: {} + - name: tun + hostPath: + path: /dev/net/tun + type: CharDevice diff --git a/docs/limitations.md b/docs/limitations.md new file mode 100644 index 0000000..b529ec1 --- /dev/null +++ b/docs/limitations.md @@ -0,0 +1,13 @@ +# Limitations and known issues + +Snapshot currently focuses on inference cold-start; further use cases are on the +roadmap. + +## Current limitations + +- Single-GPU workloads only. +- x86_64 nodes only. +- vGPU is not supported. +- Runs only on NVIDIA GPUs supported by the required CUDA driver. + +Multi-GPU and Arm support are on the roadmap. diff --git a/docs/operations/install.md b/docs/operations/install.md new file mode 100644 index 0000000..e22431c --- /dev/null +++ b/docs/operations/install.md @@ -0,0 +1,42 @@ +# Installation + +Snapshot installs as a single per-cluster Helm release: a control-plane operator +and a privileged node agent (DaemonSet) on GPU nodes. Install it in its own +namespace, and run GPU workloads in separate namespaces. + +Review the [prerequisites](../../README.md#prerequisites) before installing. + +## Install from a release + +Find the latest version on the +[releases page](https://github.com/ai-dynamo/snapshot/releases), then install the +published chart, replacing ``: + +```bash +helm install snapshot oci://ghcr.io/ai-dynamo/snapshot/snapshot \ + --version \ + --namespace snapshot --create-namespace +``` + +By default the chart provisions its own `ReadWriteMany` checkpoint volume, shared +by every checkpoint. See [Storage](storage.md) for the volume model and options, +including reusing an existing claim. + +## Verify the installation + +```bash +kubectl get pods --namespace snapshot +kubectl rollout status daemonset/snapshot-agent --namespace snapshot +``` + +The operator and the `snapshot-agent` DaemonSet become ready once the node agent +is running on each GPU node. + +## Uninstall + +```bash +helm uninstall snapshot --namespace snapshot +``` + +Chart-created checkpoint volumes are retained on uninstall, so checkpoints survive +removal — see [Storage](storage.md#retention). diff --git a/docs/operations/security.md b/docs/operations/security.md new file mode 100644 index 0000000..b74c541 --- /dev/null +++ b/docs/operations/security.md @@ -0,0 +1,19 @@ +# Security + +Snapshot's node agent needs elevated privileges to checkpoint and restore +processes. This page describes what it requires and why. + +## Privileged node agent + +The `snapshot-agent` runs as a privileged DaemonSet with `hostPID`, `hostIPC`, and +`hostNetwork` so it can invoke CRIU and `cuda-checkpoint` against live processes on +the node. Workloads do not need to be privileged — only the agent does. + +Because of this, the agent's namespace must permit privileged pods. On clusters +that enforce [Pod Security Standards](https://kubernetes.io/docs/concepts/security/pod-security-standards/), +apply the `privileged` level (or an equivalent exception) to that namespace. + +## Seccomp + +CRIU requires a seccomp profile to perform checkpoint and restore. The Helm chart +installs the profile the agent needs. diff --git a/docs/operations/storage.md b/docs/operations/storage.md new file mode 100644 index 0000000..05bd6ab --- /dev/null +++ b/docs/operations/storage.md @@ -0,0 +1,56 @@ +# Storage + +Snapshot keeps every checkpoint in a single shared volume that all node agents +mount. Each agent reads and writes checkpoint artifacts there; workload pods never +mount checkpoint storage. + +## The checkpoint volume + +Today the checkpoint store is a Kubernetes PersistentVolumeClaim (PVC). Because +agents on multiple GPU nodes mount it concurrently, it must support +`ReadWriteMany` (RWX). The chart provisions one PVC per cluster by default and +mounts it at `/checkpoints` in every agent. + +## Configuration + +The chart's `storage.pvc` values control the PVC: + +| Value | Purpose | Default | +|-------|---------|---------| +| `storage.pvc.create` | Create the PVC (set `false` to use an existing one) | `true` | +| `storage.pvc.name` | Shared RWX PVC mounted by every agent | `snapshot-pvc` | +| `storage.pvc.size` | Requested size | `1Ti` | +| `storage.pvc.storageClass` | Storage class (empty = cluster default) | `""` | +| `storage.pvc.basePath` | Mount path inside the agent | `/checkpoints` | + +If the cluster has no default storage class that can provision RWX, set one: + +```bash +helm install snapshot oci://ghcr.io/ai-dynamo/snapshot/snapshot \ + --namespace snapshot --create-namespace \ + --set storage.pvc.storageClass= +``` + +### Use an existing PVC + +Point the chart at an existing RWX claim instead of creating one: + +```bash +helm install snapshot ... \ + --set storage.pvc.create=false \ + --set storage.pvc.name= +``` + +The named claim must support `ReadWriteMany`. Access modes are immutable, so a +`ReadWriteOnce` claim cannot be converted in place — create a new RWX claim and, if +the existing checkpoints are needed, copy them over once. + +## Retention + +Chart-created PVCs are retained when the Helm release is removed, so checkpoints +survive an uninstall. + +## Other backends + +`storage.type` currently supports `pvc`. Object-storage backends (`s3`, `oci`) are +reserved in the chart for future use and are not supported today. diff --git a/docs/operations/troubleshooting.md b/docs/operations/troubleshooting.md new file mode 100644 index 0000000..b2c16f2 --- /dev/null +++ b/docs/operations/troubleshooting.md @@ -0,0 +1,34 @@ +# Troubleshooting + +Common issues when running Snapshot, and where to look. + +## A checkpoint never becomes Ready + +A `PodSnapshot` becomes Ready only after the `snapshot-agent` confirms the +checkpoint contents — writing the checkpoint is not enough on its own. Check the +status and the agent logs: + +```bash +kubectl get podsnapshot -n +kubectl logs daemonset/snapshot-agent -n --all-containers +``` + +A common cause is a replica manifest that uses the raw runtime image instead of a +[snapshot-ready image](../guides/README.md), or that omits mounts or secrets the +replica needs to start. + +## Restore cannot find or mount checkpoint storage + +Restore discovers checkpoint storage through the `snapshot-agent` DaemonSet, which +must be ready and must have the checkpoint PVC available: + +```bash +kubectl rollout status daemonset/snapshot-agent -n +kubectl get pvc -n +``` + +## The agent or a restore pod will not start + +The `snapshot-agent` runs privileged with `hostPID`, `hostIPC`, and `hostNetwork`. +If the namespace enforces a restrictive Pod Security level, the agent — or a +restore pod — can be rejected. See [Security](security.md). diff --git a/docs/reference/api.md b/docs/reference/api.md new file mode 100644 index 0000000..48b8bee --- /dev/null +++ b/docs/reference/api.md @@ -0,0 +1,7 @@ +# API reference + +_Documentation for this page is in progress._ + +Snapshot is driven through four Kubernetes API objects. For usage-level +descriptions, see [How to use it](../../README.md#how-to-use-it) and the +[usage guides](../guides/README.md). diff --git a/docs/reference/architecture.md b/docs/reference/architecture.md new file mode 100644 index 0000000..d5349f0 --- /dev/null +++ b/docs/reference/architecture.md @@ -0,0 +1,3 @@ +# Architecture + +_Documentation for this page is in progress._ diff --git a/docs/reference/cli.md b/docs/reference/cli.md new file mode 100644 index 0000000..a70c322 --- /dev/null +++ b/docs/reference/cli.md @@ -0,0 +1,49 @@ +# `snapshotctl` CLI + +`snapshotctl` is a lower-level utility for checkpointing and restoring a pod +directly from a pod manifest. It is not the primary path — most users drive +Snapshot through the [Kubernetes resources](../guides/README.md) — but it is handy +for validation and debugging, and it is a quick way to try checkpoint/restore by +hand. + +## Requirements + +- The Snapshot Helm chart is installed in the target namespace, with the + `snapshot-agent` DaemonSet running and the checkpoint PVC mounted. +- `checkpoint` requires the operator (it resolves the `PodSnapshot` into a + checkpoint). `restore` is handled by the agent directly from pod annotations. + +## Checkpoint + +`snapshotctl checkpoint` creates a `PodSnapshot` from a pod manifest and waits for +the agent to checkpoint it: + +```bash +snapshotctl checkpoint \ + --manifest ./vllm-replica-pod.yaml \ + --snapshot vllm-snapshot \ + --container main \ + --namespace my-inference +``` + +The manifest must be a `Pod` (not a Deployment or Job) using a +[snapshot-ready image](../guides/README.md). + +## Restore + +`snapshotctl restore` creates a new pod from a manifest and restores it from a +named `PodSnapshot`: + +```bash +snapshotctl restore \ + --manifest ./vllm-replica-pod.yaml \ + --snapshot vllm-snapshot \ + --namespace my-inference +``` + +The restore manifest must contain a container with the same name checkpointed by that +`PodSnapshot`. `snapshotctl` returns once the restore is submitted — watch the +pod's `snapshot/Restored` status condition, readiness, and events for progress. + +The source README for the tool lives at +[`operator/cmd/snapshotctl/README.md`](../../operator/cmd/snapshotctl/README.md).