Reclaim Modal sandboxes whose create call was interrupted

[nicolaslara]

Symptom

Interrupt an eval that uses the Modal runtime while sandboxes are still being provisioned, and more sandboxes stay alive than the harness ever reports as up. They then run until their 24h timeout expires.

Cause

Modal commits the sandbox and hands it to the scheduler before Sandbox.create returns. A create interrupted in flight therefore still produces a running, billed sandbox. Timeline measured on one such sandbox:

t+0.00s   sandbox created and scheduled server-side
t+0.0xs   caller interrupted, inside `await Sandbox.create.aio(...)`
t+0.50s   task enqueued
t+1.50s   container RUNNING — nobody holds a handle to it

In ModalRuntime.start the handle is only assigned once the call returns:

self._sandbox = await modal.Sandbox.create.aio(...)   # interrupted here

So _sandbox stays None, and both cleanup paths key off exactly that attribute — teardown returns early, and the cleanup atexit backstop no-ops. The sandbox is unreachable from the process that made it.

Cancellation is the common trigger (Ctrl-C, or a rollout cancelled while its batch is still provisioning), and CancelledError is not an Exception, so it also slipped past except Exception in start and left no log line. That is why the "sandbox ... up" count doesn't match what's actually running. A connection dropped on the reply has the same shape.

Fix

The name passed to Sandbox.create is ours and unique per rollout, and Modal enforces name uniqueness per app, so an interrupted create is still addressable. Look it up on the way out and adopt it; the owner's stop then disposes of it like any other sandbox.

_adopt_orphan is a no-op when a handle is already held, so it is also safe on the existing except Exception path — which additionally covers a create that failed after the server created the sandbox.

Verification

Real Modal sandboxes, 12 rollouts started concurrently, batch cancelled mid-create, then every runtime stopped exactly as the owner's finally does. LEAKED is measured with Sandbox.list against the app, not inferred:

Before

interrupted=12 created_server_side=12 no_handle_after_start=3 adopted=0 LEAKED=3
interrupted=12 created_server_side=12 no_handle_after_start=2 adopted=0 LEAKED=2
interrupted=12 created_server_side=12 no_handle_after_start=7 adopted=0 LEAKED=7

After

interrupted=12 created_server_side=11 no_handle_after_start=1 adopted=6 LEAKED=0
interrupted=12 created_server_side=12 no_handle_after_start=0 adopted=9 LEAKED=0
interrupted=12 created_server_side=12 no_handle_after_start=0 adopted=4 LEAKED=0

In run 1 after the fix, one rollout's create never reached the server (created_server_side=11), so there was correctly nothing to adopt.

Cancelling outside the create window is unaffected: the handle is already held and _adopt_orphan returns immediately.

uv run ruff check, uv run ruff format --check, and uv run ty check verifiers all pass. No tests added, per AGENTS.md — the check above was a temporary script.

Two notes for maintainers, not changed here

• ModalConfig.creates_per_sec defaults to 40.0. Modal's default per-workspace sandbox creation limit is 5/s with a 150-token burst bucket, so this default is 8× over for a workspace that hasn't had its limit raised. Creates then retry with backoff, which keeps them in flight much longer and widens the window this PR closes. Worth either lowering the default or documenting that the limit needs raising.
• Sandbox.create accepts idle_timeout. It is the only cleanup that survives losing the client entirely — a hard kill, or a second Ctrl-C that kills the event loop before any recovery code runs. Currently only timeout=24h is set, so anything the process can't clean up lives for a day.

Disclosure: I work at Modal. This came out of debugging a leak with a user running verifiers on us.

🤖 Generated with Claude Code

[xeophon]

[P2] Keep the recovered handle across repeated cancellation

A bare asyncio.shield re-raises when this task is cancelled again. That second CancelledError is not caught by suppress(Exception), so the assignment is skipped even if the shielded lookup later completes; _sandbox remains None and teardown still cannot reclaim the sandbox. verifiers.v1.utils.aio.run_shielded exists specifically to absorb repeated task cancellations until the owned operation completes. The self._sandbox assignment needs to happen inside the coroutine passed to that helper so the handle is installed before cancellation is re-raised.

[xeophon]

[P1] Don't adopt a pre-existing sandbox after a duplicate-name error

Modal documents that Sandbox.create raises AlreadyExistsError when the requested name already exists. This broad except Exception then calls from_name, attaches to that already-running sandbox, and the owner's finally/abort path calls stop(), whose Modal teardown terminates the adopted handle. Because make_runtime and provision_runtime accept caller-supplied names, a duplicate-name create can therefore terminate a sandbox this runtime did not create. Please keep ownership of the in-flight create directly—for example, perform the create and self._sandbox assignment inside a coroutine passed to the existing run_shielded helper—instead of recovering by name after every exception.