Repository navigation
fix(sandbox): Daytona DinD denylist overlay, gateway retries and leaked sandboxes - #1123
bingran-you wants to merge 3 commits into
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
main added _guard_daemon_from_containers to the DinD start path after this test was written; its fake VM exec does not answer the guard, so startup was refused before the uploads were checked. The guard has its own tests in test_sdk_lockdown.py.
96a0a80 to
7bd5489
Compare
…led create left The Daytona SDK creates a sandbox and then waits for it to start inside one create() call. A 502 while refreshing its state raised out of that wait, so benchflow got no handle: the sandbox kept running with nothing able to delete it, and the attempt was not retried because 502/503/504 were not transient (DaytonaBadGatewayError / DaytonaServiceUnavailableError in newer SDKs, a status-coded base DaytonaError in older ones). Treat gateway statuses (and DaytonaConnectionTimeoutError) as transient; the status-code fallback applies only to the base class. Every sandbox one DaytonaSandbox creates carries a benchflow.create_attempt label; the sandboxes carrying it, other than the one in use, are deleted after a failed or timed-out attempt (shielded and bounded when cancelled), after a retry succeeds (a sandbox Daytona committed after its error answer), and in stop(). The lookup and deletes retry gateway errors; only that exact label is matched, client-side as well as in the query, so other sandboxes on a shared API key are never touched. Adds CHANGELOG entries for this and the DinD overlay fix.
bingran-you
changed the title
|
Evidence for the gateway-retry part, from FrontierPhysics on benchflow 0.7.8: on 2026-10-07 21:30 UTC two GPT-6.1 Sol rollouts ( One ask: the two idempotent polls ( On the leak half: a sandbox leaked by a gateway error during create is one of the two leak paths we see. The other, a docker container left by a runner killed with SIGTERM, is #1153. |
Two Daytona fixes found while running every FrontierPhysics oracle (main
9787d079) on benchflow 0.7.11.1. Compose tasks with a URL denylist could not start on Daytona
Daytona DinD adds
/benchflow/compose/docker-compose-net-admin.yamlto every denylist compose invocation, but it did not upload that bundled overlay into the sandbox. Compose-based denylist tasks failed before their image build withopen /benchflow/compose/docker-compose-net-admin.yaml: no such file or directory(present in 0.7.8 through 0.7.11). The overlay is now uploaded with the other bundled compose files. The regression test drives the DinD startup path and checks, at build time, that every referenced compose file has been uploaded. Its second commit stubs the DinD daemon guard (_guard_daemon_from_containers, added to the start path after the test was written); the guard has its own tests intests/test_sdk_lockdown.py.2. A gateway error during sandbox creation was not retried and leaked the sandbox
The Daytona SDK creates a sandbox and then waits for it to start inside one
create()call. In the sweep, a 502 while refreshing the new sandbox's state raised out of that wait. benchflow got no handle, so the sandbox kept running with nothing able to delete it (found still running after the rollout ended), and the attempt was not retried: 502/503/504 map toDaytonaBadGatewayError/DaytonaServiceUnavailableErrorin SDK 0.225 and to a status-coded baseDaytonaErrorin 0.184, and neither was transient.DaytonaConnectionTimeoutError(0.225's transport timeout) are now transient. The status-code fallback applies only to the baseDaytonaError; typed classes go by name.DaytonaSandboxcreates carries abenchflow.create_attemptlabel. Sandboxes carrying it, other than the one in use, are deleted after a failed or timed-out attempt (shielded and bounded when cancelled), after a retry succeeds (catching a sandbox Daytona committed after its error answer), and instop(). The lookup and the deletes retry gateway errors. Only that exact label is matched, in the query and again client-side, so other sandboxes on a shared API key are never touched.Validation at
c5e673a3ruff checkandruff format --checkclean.tests/test_daytona_create_cleanup.py(25 tests) passes with the locked Daytona SDK 0.184.0 and with 0.225.0, including a contract test that runs each status through the installed SDK's own error mapping. Nine mutations of the fix (no label, no client-side filter, no post-success, stop, failure or timeout sweep, no lookup retry, no status fallback, no gateway class names) each fail at least one test; against main's code the new tests fail.tests/test_daytona_dind_compose_up.py,test_sandbox_multi_service.py,test_egress_denylist.py,test_network_denylist_config.py,test_sdk_lockdown.py: 234 passed; withdaytona_dind.pyreverted to main the overlay regression test fails.9787d079:ion-trap-3d-vs-2dandtwisted-slinky-wave-coupling, which failed on 0.7.11 with the overlay error, both completed with verifier reward 1.0 at7bd54896. Atc5e673a3the same two tasks andrixs-entanglement-witness(whose first 0.7.11 attempt hit the 502 above) all completed with verifier reward 1.0, and the run left no sandbox behind under its owner label. No gateway error occurred in that run, so the retry and cleanup paths are covered by the unit tests only.