docs: document CUDA CustomStorage configuration

README.md

@@ -71,6 +71,8 @@ Before installing Snapshot, make sure you have:
 - The [Helm](https://helm.sh/docs/intro/install) CLI
 - A cluster that permits privileged pods for the node agent — see [Security](docs/operations/security.md)
 
+The default CUDA checkpoint path works with the general driver prerequisite above. The optional [CUDA CustomStorage path](docs/guides/custom-storage.md) uses the CUDA 13.4 Developer Preview API and additionally requires a driver that exports that API; CUDA 13.x compatibility alone is not a capability check.
+
 <!-- TODO(eng): open items for the team to validate (feed Support matrix / Limitations):
      - minimum Kubernetes version (any non-EOL release, or a specific floor?)
      - minimum GPU Operator version (e2e preflight uses 26.3.0+)

charts/snapshot/README.md

@@ -72,6 +72,19 @@ helm upgrade --install snapshot ./charts/snapshot \
   --set storage.pvc.name=my-snapshot-pvc
 ```
 
+## CUDA CustomStorage
+
+New checkpoints use the legacy CUDA driver-managed storage path by default. To externalize CUDA checkpoint state through the Snapshot-local NIXL POSIX adapter, set `config.cudaCheckpoint.storageMode=posix`. This requires a driver that exports the CUDA 13.4 CustomStorage API and is initially supported only for one-GPU workloads.
+
+```bash
+helm upgrade --install snapshot ./charts/snapshot \
+  --namespace ${NAMESPACE} \
+  --create-namespace \
+  --set config.cudaCheckpoint.storageMode=posix
+```
+
+The mode is recorded in each checkpoint manifest, and restore follows the artifact rather than the current creation setting. Snapshot does not silently fall back from an explicitly requested `posix` checkpoint to `legacy`. See [Configure CUDA CustomStorage with NIXL POSIX](../../docs/guides/custom-storage.md) before enabling it.
+
 ## CRD upgrades
 
 Helm creates the CRDs in [crds/](./crds) on a fresh install and then leaves them
@@ -137,6 +150,11 @@ kubectl get pods -n ${NAMESPACE} -l app.kubernetes.io/name=snapshot -o wide
 | `seccomp.deploy` | Deploy the CRIU seccomp profile ConfigMap and init container. Use this field name; `seccomp.enabled` is not a chart value | `true` |
 | `runtime.type` | CRI backend: `containerd` or `crio` | `containerd` |
 | `runtime.socketPath` | CRI socket (empty = default for `runtime.type`) | `""` |
+| `config.cudaCheckpoint.storageMode` | CUDA checkpoint storage mode for newly created checkpoints: `legacy` or `posix` | `legacy` |
+| `config.cudaCheckpoint.transferBufferCount` | Pinned transfer slots per active CUDA device | `4` |
+| `config.cudaCheckpoint.transferChunkBytes` | Bytes per pinned transfer slot | `67108864` |
+| `config.cudaCheckpoint.daemon.maxOperationSeconds` | Maximum time allowed for one CUDA checkpoint or restore helper request; does not forcibly interrupt a CUDA driver call already in progress | `3600` |
+| `config.cudaCheckpoint.daemon.resources` | CUDA helper CPU and memory requests and limits | See `values.yaml` |
 | `crdUpgrade.enabled` | Install and upgrade the CRDs from an operator init container (see below) | `true` |
 | `crdUpgrade.logLevel` | Init container log level | `info` |
 | `rbac.create` | Create agent and operator RBAC | `true` |

docs/guides/README.md

@@ -26,6 +26,7 @@ Using Snapshot is a two-step model:
 
 - [Checkpoint a replica](checkpoint.md)
 - [Restore a replica](restore.md)
+- [Configure CUDA CustomStorage with NIXL POSIX](custom-storage.md) — explicitly externalize CUDA checkpoint state through the Snapshot-local NIXL POSIX path.
 
 See [Installation](../operations/install.md) for cluster prerequisites and the
 [API reference](../reference/api.md) for full resource detail.

docs/guides/custom-storage.md

@@ -0,0 +1,146 @@
+# Configure CUDA CustomStorage with NIXL POSIX
+
+CUDA CustomStorage externalizes CUDA checkpoint state into Snapshot-managed files instead of relying on the CUDA driver's default checkpoint storage. Snapshot's `posix` mode moves those CUDA extents through its local NIXL POSIX adapter into the same checkpoint volume that stores the CRIU and root filesystem artifacts. CRIU remains the process-state checkpoint and restore engine in both storage modes; NIXL does not replace CRIU.
+
+```mermaid
+flowchart LR
+    A["Snapshot node agent"] --> R["CRIU"]
+    R --> P["Shared checkpoint PVC"]
+    A --> H["CUDA checkpoint helper"]
+    H --> C["CUDA 13.4 CustomStorage"]
+    C --> N["NIXL POSIX adapter"]
+    N --> P["Shared checkpoint PVC"]
+```
+
+CustomStorage is configured for the Snapshot node agents through Helm. It is not a workload flag, pod annotation, or `PodSnapshot` field.
+
+> [!IMPORTANT]
+> CustomStorage is an explicit opt-in for new checkpoints. The default remains `legacy`. Enable `posix` only on a release that includes the CustomStorage helper and NIXL POSIX adapter. CUDA 13.4 is currently a Developer Preview; follow the upstream release restrictions for preview software.
+
+## Prerequisites
+
+In addition to Snapshot's normal [prerequisites](../../README.md#prerequisites), `posix` mode requires:
+
+- A Linux NVIDIA driver that exports the CUDA 13.4 CustomStorage driver API, including `cuCheckpointOperationComplete`. CUDA 13.x minor-version compatibility alone does not make a new CUDA 13.4 API available. The CUDA 13.4 developer-preview release notes do not publish a numeric Linux driver floor for new 13.4 features, so the helper checks the capability at runtime. Use that result rather than only the installed toolkit or driver branch as the admission test. See the [CUDA 13.4 release notes](https://docs.nvidia.com/cuda/developer-preview/13.4/cuda-toolkit-release-notes/index.html#driver-compatibility).
+- A Snapshot agent image built with the NIXL POSIX transfer adapter. A transfer-neutral helper build does not support `posix` mode.
+- A shared RWX checkpoint PVC with enough capacity for the CRIU, root filesystem, and CUDA extent artifacts. Plan for approximately the checkpointed CUDA allocation bytes in addition to the existing process artifacts and temporary staging space.
+- A one-GPU workload. The initial `posix` implementation supports one or more CUDA-owning processes on one GPU. Snapshot rejects multi-GPU CustomStorage checkpoint creation before locking a target.
+
+The Snapshot agent image may be built against CUDA 13.0 headers because the helper dynamically resolves the public CUDA 13.4 entry point. The node driver, not the image's toolkit version, determines whether CustomStorage is available at runtime.
+
+## Select the CustomStorage path
+
+Set `config.cudaCheckpoint.storageMode` to `posix` in the Helm values used for the Snapshot release:
+
+```yaml
+config:
+  cudaCheckpoint:
+    storageMode: posix
+```
+
+Apply the values to the existing release:
+
+```bash
+helm upgrade snapshot oci://ghcr.io/ai-dynamo/snapshot/snapshot \
+  --version <VERSION> \
+  --namespace snapshot \
+  --reuse-values \
+  --values custom-storage-values.yaml
+```
+
+Wait for every node agent to use the new configuration before creating a `posix` checkpoint:
+
+```bash
+kubectl rollout status daemonset/snapshot-agent -n snapshot
+```
+
+The setting applies to every new CUDA checkpoint created by that agent deployment. There is no per-workload override in the initial release.
+
+## Configuration values
+
+| Value | Meaning | Default | Valid values |
+|---|---|---:|---|
+| `config.cudaCheckpoint.storageMode` | CUDA checkpoint storage mode for newly created checkpoints | `legacy` | `legacy`, `posix` |
+| `config.cudaCheckpoint.transferBufferCount` | Pinned transfer slots per active CUDA device; this controls pipeline depth, not the number of worker threads | `4` | `1`-`8` |
+| `config.cudaCheckpoint.transferChunkBytes` | Bytes in each pinned transfer slot | `67108864` (64 MiB) | 1-256 MiB, 4096-byte aligned |
+| `config.cudaCheckpoint.daemon.maxOperationSeconds` | Maximum time allowed for one CUDA checkpoint or restore helper request; a CUDA driver call already in progress cannot be forcibly interrupted | `3600` | `1`-`3600` |
+| `config.cudaCheckpoint.daemon.resources` | CPU and memory requests and limits for the helper container | See `values.yaml` | Kubernetes resource values |
+
+Snapshot rejects a transfer configuration that allocates more than 1 GiB of pinned host memory per active CUDA device. The pinned allocation is:
+
+```text
+transferBufferCount * transferChunkBytes * active CUDA devices
+```
+
+The qualified initial configuration uses four 64 MiB buffers, or 256 MiB of pinned host memory for the supported one-GPU operation. The helper memory limit must also leave room for CUDA, NIXL, and process overhead. Increase it if you increase the buffer allocation.
+
+`maxOperationSeconds` bounds one helper checkpoint or restore request through cooperative deadline checks. A CUDA driver call already in progress cannot be forcibly interrupted. Snapshot's general `config.restore.restoreTimeoutSeconds` setting must still leave enough time for CRIU, every CUDA-owning process, and the helper's response margin.
+
+## Checkpoint and restore behavior
+
+For a new checkpoint, Snapshot validates the one-GPU topology and the helper's CustomStorage capability before it locks or changes the CUDA target. `posix` is fail-closed: Snapshot does not silently fall back to `legacy` after an operator explicitly enables CustomStorage.
+
+Snapshot records the selected CUDA storage mode in the checkpoint manifest. Restore always follows that manifest rather than the current Helm creation policy:
+
+- A `legacy` checkpoint restores through the legacy driver-managed path, even when new checkpoint creation is configured for `posix`.
+- A `posix` checkpoint requires a CustomStorage-capable helper during every restore, even if new checkpoint creation has since been changed back to `legacy`.
+- Checkpoints created before the manifest recorded a storage mode are treated as `legacy` for compatibility.
+
+During restore, Snapshot starts reading the CUDA extent files into the node page cache while CRIU restores the process. After the restored CUDA-owning PIDs are known, the helper performs the authoritative transfer into CUDA-registered host buffers and completes the CUDA restore operation.
+
+## Verify the rollout
+
+Check the helper startup event on each node-agent pod:
+
+```bash
+kubectl logs -n snapshot \
+  -l app.kubernetes.io/instance=snapshot,app.kubernetes.io/component=snapshot-agent \
+  -c cuda-checkpoint-helper --prefix=true --tail=-1 --max-log-requests=100 \
+  | grep cuda_checkpoint_daemon_ready
+```
+
+A node that can create and restore `posix` artifacts reports all three capabilities as `true`:
+
+```text
+"custom_storage_driver_api_available":true
+"custom_storage_transfer_backend_available":true
+"custom_storage_available":true
+```
+
+After creating a checkpoint, check the agent log for the selected mode:
+
+```bash
+kubectl logs -n snapshot \
+  -l app.kubernetes.io/instance=snapshot,app.kubernetes.io/component=snapshot-agent \
+  -c agent --prefix=true --tail=-1 --max-log-requests=100 \
+  | grep 'CUDA CustomStorage explicitly enabled'
+```
+
+The log entry includes `cuda_storage_mode=posix`. A successful transfer also emits a `cuda_custom_storage_transfer` event with its byte count, duration, pinned-memory allocation, and phase timings.
+
+Run a complete checkpoint, restore, readiness, and inference smoke test before a wider rollout. Helper health proves capability, not workload correctness.
+
+## Return new checkpoints to legacy mode
+
+Set the creation policy back to `legacy` and roll out the agents:
+
+```bash
+helm upgrade snapshot oci://ghcr.io/ai-dynamo/snapshot/snapshot \
+  --version <VERSION> \
+  --namespace snapshot \
+  --reuse-values \
+  --set config.cudaCheckpoint.storageMode=legacy
+
+kubectl rollout status daemonset/snapshot-agent -n snapshot
+```
+
+This affects only new checkpoints. Keep CustomStorage-capable agents available for as long as any retained `posix` checkpoint may be restored.
+
+## Current limitations
+
+- `posix` checkpoint creation is qualified only for one GPU. A container may have more than one CUDA-owning process on that GPU.
+- The NIXL POSIX adapter uses the Snapshot checkpoint PVC. PageBroker and other external data planes are separate integrations.
+- Only one CUDA checkpoint or restore sequence runs at a time within one Snapshot agent. Multiple Snapshot agent DaemonSets on the same node are not coordinated and are unsupported.
+- An interrupted or timed-out CUDA driver call can leave the target outcome unknown. Snapshot does not automatically replay an operation that may have reached the driver.
+
+See [Troubleshooting](../operations/troubleshooting.md#cuda-customstorage) for common validation and runtime errors.

docs/operations/storage.md

@@ -50,6 +50,12 @@ you need the existing checkpoints, copy them over once.
 Chart-created PVCs are retained when the Helm release is removed, so checkpoints
 survive an uninstall.
 
+## CUDA CustomStorage capacity
+
+When CUDA CustomStorage is enabled, the checkpoint volume also stores the CUDA extent files. Plan for approximately the checkpointed CUDA allocation bytes in addition to CRIU images, the root filesystem diff, and temporary checkpoint staging. The default CustomStorage transfer pipeline uses 256 MiB of pinned host memory for the supported one-GPU operation; pinned memory is helper working memory and does not reduce the required PVC capacity.
+
+See [Configure CUDA CustomStorage with NIXL POSIX](../guides/custom-storage.md) for enablement, memory sizing, compatibility, and verification.
+
 ## Other backends
 
 `storage.type` currently supports `pvc`. Object-storage backends (`s3`, `oci`) are

docs/operations/troubleshooting.md

@@ -33,4 +33,29 @@ The `snapshot-agent` runs privileged with `hostPID`, `hostIPC`, and `hostNetwork
 If the namespace enforces a restrictive Pod Security level, the agent — or a
 restore pod — can be rejected. See [Security](security.md).
 
+## CUDA CustomStorage
+
+### A `posix` checkpoint is rejected before capture
+
+Snapshot deliberately fails before locking the CUDA target when the helper does not advertise both the CUDA 13.4 CustomStorage driver API and the NIXL POSIX transfer adapter. Check the helper startup event:
+
+```bash
+kubectl logs -n snapshot \
+  -l app.kubernetes.io/instance=snapshot,app.kubernetes.io/component=snapshot-agent \
+  -c cuda-checkpoint-helper --prefix=true --tail=-1 --max-log-requests=100 \
+  | grep cuda_checkpoint_daemon_ready
+```
+
+If `custom_storage_driver_api_available` is `false`, install a driver that exports the CUDA 13.4 CustomStorage API. A CUDA 13.x-compatible driver does not necessarily provide new CUDA 13.4 APIs. If `custom_storage_transfer_backend_available` is `false`, use a Snapshot agent image that includes the NIXL POSIX adapter.
+
+Snapshot also rejects `posix` creation when the workload uses more than one GPU. The initial implementation supports one or more CUDA-owning processes on one GPU.
+
+### A retained `posix` checkpoint no longer restores
+
+Restore follows the checkpoint manifest, not the current `storageMode` setting. Changing new checkpoint creation back to `legacy` does not convert existing `posix` artifacts. Keep a CustomStorage-capable driver and Snapshot image available until those artifacts are retired.
+
+### The helper times out or is unhealthy after an operation
+
+Check both agent and helper logs for the same target PID. An absent helper response after a state-changing CUDA request is an unknown outcome and is not replayed automatically. Do not resume the workload based only on storage cleanup. See [Configure CUDA CustomStorage with NIXL POSIX](../guides/custom-storage.md#configuration-values) before changing the helper deadline, restore timeout, or pinned-buffer settings.
+
 <!-- TODO(eng): expand with real failure modes and messages (CRIU / cuda-checkpoint errors, RWX PVC access, driver/runtime mismatches). The Dynamo snapshot doc's troubleshooting section is a good source. -->