docs: document CUDA CustomStorage configuration by hhzhang16 · Pull Request #120 · ai-dynamo/snapshot

hhzhang16 · 2026-08-26T19:09:28Z

Summary

Adds documentation for configuring and validating Snapshot's opt-in CUDA CustomStorage path (the behavior implemented in #109 + #96). The guide is linked from the main README, chart documentation, storage documentation, troubleshooting, and the guide index.

Review focus

Whether the CUDA capability requirements, Helm values, artifact compatibility, and current limitations are clear and accurate for someone enabling this for the first time.

Validation

Cross-checked the values, defaults, capability fields, and failure behavior against feat: add the gated local CustomStorage POSIX path #96.
Helm lint and render passed with the documented posix configuration.
All newly added links and anchors resolve.
git diff --check passes.

Signed-off-by: Hannah Zhang <hannahz@nvidia.com>

coderabbitai · 2026-08-26T19:09:37Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

🗂️ Base branches to auto review (1)

main

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: c857e048-938c-4796-8476-fa412efb28ef

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

_{Comment @coderabbitai help to get the list of available commands.}

ozbarshalom · 2026-08-27T09:18:15Z

 - The [Helm](https://helm.sh/docs/intro/install) CLI
 - A cluster that permits privileged pods for the node agent — see [Security](docs/operations/security.md)

+The default CUDA checkpoint path works with the general driver prerequisite above. Enabling [CUDA CustomStorage](docs/guides/custom-storage.md) additionally requires a driver that exports the CUDA 13.4 CustomStorage API; CUDA 13.x compatibility alone is not a capability check.


CUDA 13.4 is not GA

https://docs.nvidia.com/cuda/developer-preview/13.4/cuda-toolkit-release-notes/index.html#driver-compatibility which CUDA drivers supports CUDA 13.4?

The CustomStorage path is an explicit opt-in using the CUDA 13.4 Developer Preview API, while the default remains the existing driver-managed path. The guide calls this out

The helper checks cuCheckpointOperationComplete at runtime and rejects posix before mutating the workload when it isn’t available

Which CUDA driver supports CUDA 13.4? I believe there is a support matrix of GPU operator + CUDA driver + CUDA version

The missing piece for me is which CUDA driver does customers needs to install

ozbarshalom · 2026-08-27T09:35:04Z

 | `seccomp.deploy` | Deploy the CRIU seccomp profile ConfigMap and init container. Use this field name; `seccomp.enabled` is not a chart value | `true` |
 | `runtime.type` | CRI backend: `containerd` or `crio` | `containerd` |
 | `runtime.socketPath` | CRI socket (empty = default for `runtime.type`) | `""` |
+| `config.cudaCheckpoint.storageMode` | Storage policy for newly created CUDA checkpoints: `legacy` or `posix` | `legacy` |


move config.cudaCheckpoint -> storage.gpu.engine

storageMode -> backend [criu | nixl ] later we will also have

They are not interchangeable backend values imo -- CRIU is the process checkpoint engine, while NIXL POSIX is the CUDA artifact transfer backend.

I’m open to reorganizing the config hierarchy, but the accurate values would be closer to driver-managed | nixl-posix, not criu | nixl. Changing it would also require updating #96 rather than only renaming it in the docs

I do want to change the hierarchy, of course not only in the docs but that's my reference as PM for these interface changes.
regarding values: @dfeigin-nv WDYT?

ozbarshalom · 2026-08-27T09:36:21Z

+| `config.cudaCheckpoint.storageMode` | Storage policy for newly created CUDA checkpoints: `legacy` or `posix` | `legacy` |
+| `config.cudaCheckpoint.transferBufferCount` | Pinned transfer slots per active CUDA device | `4` |
+| `config.cudaCheckpoint.transferChunkBytes` | Bytes per pinned transfer slot | `67108864` |
+| `config.cudaCheckpoint.daemon.maxOperationSeconds` | Cooperative watchdog for one CUDA helper operation | `3600` |


@dfeigin-nv

I think "Cooperative watchdog for one CUDA helper operation" is unclear from a user perspective and also I don't get it.

It's the maximum time allowed for one CUDA checkpoint or restore helper request

ozbarshalom · 2026-08-27T09:37:19Z

+```yaml
+config:
+  cudaCheckpoint:
+    storageMode: posix


I'm not sure the full example is required here only to enable NIXL

I included the transfer and resource values to show the operational defaults but I can reduce this to the minimal setting so it's not as scary/complicated-looking

Updated so that the enablement example now contains only the required storageMode: posix setting. The tuning and resource options remain in the reference table.

ozbarshalom · 2026-08-27T09:39:44Z

+| `config.cudaCheckpoint.transferChunkBytes` | Bytes in each pinned transfer slot | `67108864` (64 MiB) | 1-256 MiB, 4096-byte aligned |
+| `config.cudaCheckpoint.daemon.maxOperationSeconds` | Cooperative transfer watchdog for one helper operation | `3600` | `1`-`3600` |
+| `config.cudaCheckpoint.daemon.resources` | CPU and memory requests and limits for the helper container | See the example above | Kubernetes resource values |
+| `config.restore.restoreTimeoutSeconds` | Overall restore budget in the Snapshot agent | `8100` | Positive integer |


why is it in the custom storage guide?

ozbarshalom · 2026-08-27T09:45:07Z

@@ -0,0 +1,156 @@
+# Use CUDA CustomStorage


custom storage engine

ozbarshalom · 2026-08-27T09:48:22Z

+
+The Snapshot agent image may be built against CUDA 13.0 headers because the helper dynamically resolves the public CUDA 13.4 entry point. The node driver, not the image's toolkit version, determines whether CustomStorage is available at runtime.
+
+## Enable CustomStorage


Customers don't "enable custom storage" they can modify the IO engine to use NIXL instead of CRIU.

I think this is the same layer distinction as one of the above comments. NIXL does not replace CRIU here — CRIU still captures and restores the process state. Selecting posix changes the CUDA path from driver-managed checkpoint storage to the CUDA 13.4 CustomStorage API, with NIXL POSIX moving the resulting extents.

From a user perspective, customStorage is another storage rather than the existing supported storage which is local PVC. This is not the case. I do understand that for the CUDA C/R it's a custom storage backend / engine but we should slightly abstract it from Snapshot users. (naming wise)

@dfeigin-nv WDYT?

I think the naming should be something clearer than custom storage. Custom Storage is more of an implementation detail for us, not something the user should actively choose

only after criu finishes the IO does the driver-managed checkpoint storage start, so legacy | criu are both correct. What is most clear to the user should be used.

Signed-off-by: Hannah Zhang <hannahz@nvidia.com>

docs: document CUDA CustomStorage configuration

9fd64a2

Signed-off-by: Hannah Zhang <hannahz@nvidia.com>

hhzhang16 requested review from Ronkahn21, danbar2, dfeigin-nv, galletas1712, hutm, julienmancuso, oleg-kushniriov, shayasoolin and shmuel-runai as code owners August 26, 2026 19:09

ozbarshalom reviewed Aug 27, 2026

View reviewed changes

ozbarshalom requested changes Aug 27, 2026

View reviewed changes

docs: clarify CustomStorage and NIXL configuration

bebceac

Signed-off-by: Hannah Zhang <hannahz@nvidia.com>


		The Snapshot agent image may be built against CUDA 13.0 headers because the helper dynamically resolves the public CUDA 13.4 entry point. The node driver, not the image's toolkit version, determines whether CustomStorage is available at runtime.

		## Enable CustomStorage

Uh oh!

Conversation

hhzhang16 commented Aug 26, 2026

Summary

Review focus

Validation

Uh oh!

coderabbitai Bot commented Aug 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ozbarshalom Aug 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

coderabbitai Bot commented Aug 26, 2026 •

edited

Loading

ozbarshalom Aug 27, 2026 •

edited

Loading