Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions verifiers/v1/runtimes/modal.py
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,32 @@ async def start(self) -> None:
except (
Exception
) as e: # provisioning failure is one rollout's problem, not the eval's
await self._adopt_orphan()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Don't adopt a pre-existing sandbox after a duplicate-name error

Modal documents that Sandbox.create raises AlreadyExistsError when the requested name already exists. This broad except Exception then calls from_name, attaches to that already-running sandbox, and the owner's finally/abort path calls stop(), whose Modal teardown terminates the adopted handle. Because make_runtime and provision_runtime accept caller-supplied names, a duplicate-name create can therefore terminate a sandbox this runtime did not create. Please keep ownership of the in-flight create directly—for example, perform the create and self._sandbox assignment inside a coroutine passed to the existing run_shielded helper—instead of recovering by name after every exception.

raise SandboxError(f"modal sandbox provisioning failed: {e}") from e
except BaseException: # cancellation, which is not an Exception
await self._adopt_orphan()
raise

async def _adopt_orphan(self) -> None:
"""Reclaim a sandbox whose `create` did not live long enough to hand back a handle.

Modal commits the sandbox and schedules it before `Sandbox.create` responds, so a
create that does not return — Ctrl-C, a cancelled rollout, a connection dropped on
the reply — still boots and bills a sandbox, about a second after the caller stopped
waiting for it. `_sandbox` was never assigned, so neither `teardown` nor the atexit
backstop can see it and it runs to its 24h maximum lifetime. The name is ours and
unique per rollout, so the sandbox is still addressable: claim it here and the
owner's `stop` disposes of it like any other. Best effort — this runs while `start`
is already unwinding, usually from a cancellation, hence the shield.
"""
if self._sandbox is not None:
return
import modal

with contextlib.suppress(Exception):
self._sandbox = await asyncio.shield(

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Keep the recovered handle across repeated cancellation

A bare asyncio.shield re-raises when this task is cancelled again. That second CancelledError is not caught by suppress(Exception), so the assignment is skipped even if the shielded lookup later completes; _sandbox remains None and teardown still cannot reclaim the sandbox. verifiers.v1.utils.aio.run_shielded exists specifically to absorb repeated task cancellations until the owned operation completes. The self._sandbox assignment needs to happen inside the coroutine passed to that helper so the handle is installed before cancellation is re-raised.

modal.Sandbox.from_name.aio(_APP_NAME, self.name)
)

async def expose(self, port: int) -> str | None:
# Publish a server hosted IN the sandbox: Modal forwards `port` (named via
Expand Down