docker sandbox: a runner killed by a signal leaves its container running (no SIGTERM/SIGHUP handler; prune only removes stopped containers) #1153

Status 2026-10-26

• PR fix(sandbox): Daytona DinD denylist overlay, gateway retries and leaked sandboxes #1123 (open) deletes a Daytona sandbox that a gateway error leaked during create; that is one leak path. The docker path here (SIGTERM/SIGHUP with no handler, prune removes stopped containers only) is unchanged on main 9c7a191 = PyPI 0.7.11 — benchflow
• release/0.8 handles SIGTERM and reaps stopped containers (partial); not in a PyPI release (Ship the release/0.8 fixes in a Python release, or cherry-pick them to main #1197).
• FrontierPhysics-Infra: operators still remove a leaked docker container by hand (its cleanup tooling covers Daytona sandboxes only) — Bo

Problem. bench eval run installs no SIGTERM or SIGHUP handler, so SIGTERM ends the process immediately. No finally block or atexit hook runs, and DockerSandbox.stop() is never called. The compose main service runs sleep infinity, so the container stays up and keeps its CPU/memory limits reserved until someone removes it by hand. SIGKILL, for example a supervisor's stall kill, has the same result and cannot be caught in-process.

Evidence

• 2026-10-08 04:11 UTC (13:11 KST), docker sandbox on a GCP host: an operator script stopped a live runner 4 h 48 min into its agent run (SIGTERM, then SIGKILL 10 s later if still alive). The bench process was gone, but the compose project's main container was still up and had to be removed with docker rm -f.
• Mechanism repro: an asyncio.run() program with a try/finally and an atexit hook, sent SIGTERM, exits 143 and runs neither.
• The leaked container is never collected. _prune_docker runs docker container prune, which removes only stopped containers. The relaunch gets a new compose project (<task>__<8 hex>), so its down --remove-orphans does not touch the old one.

Where (main e965eee, same as 0.7.8)

• src/benchflow: no signal.signal or add_signal_handler on the host side (the only SIGTERM use is inside the antigravity shim).
• src/benchflow/sandbox/docker.py:530-581 stop(): the only teardown path.
• src/benchflow/sandbox/_compose_files/docker-compose-prebuilt.yaml (and -build.yaml): command: sleep infinity.
• src/benchflow/evaluation.py:1139-1185 _prune_docker: stopped containers only. The benchflow.owned=true label carries no owner or run id.

Proposed fix

• Treat SIGTERM and SIGHUP like Ctrl-C: cancel the main task so each rollout's finally runs a bounded sandbox.stop(), then exit with 128+signal.
• Label each container with its owner (host, runner pid and start time, jobs dir). Add a docker reaper, run at eval start and as bench sandbox reap --docker, that removes running benchflow containers whose owner process is gone. This is the docker counterpart of the Daytona reaper (feat(sandbox): auto-reap orphaned Daytona sandboxes at eval start #663).
• Optionally have main exit when the runner's heartbeat file goes stale, so a SIGKILL cleans up too.

Workaround in use: no code fix. Operators run docker ps after stopping a runner and remove the container by hand. FrontierPhysics-Infra 652553d now stops a runner only when both its exact pass dir and its task match, so a live sibling is no longer hit.