Skip to content

fix: do not mark runs FAILED on transient sandbox API errors - #299

Open
nishant-uxs wants to merge 1 commit into
OpenHands:mainfrom
nishant-uxs:fix/sandbox-lookup-transient-vs-absent-285
Open

fix: do not mark runs FAILED on transient sandbox API errors#299
nishant-uxs wants to merge 1 commit into
OpenHands:mainfrom
nishant-uxs:fix/sandbox-lookup-transient-vs-absent-285

Conversation

@nishant-uxs

Copy link
Copy Markdown

Problem

get_sandbox_agent_url collapsed every sandbox-API exception into None ("absent"). The watchdog then marked the run FAILED with Timed out: Sandbox not available. HTTP 429, timeouts, and 5xx carry no information about sandbox liveness — during the #291 incident this false-failed live customer runs.

Root cause

Watchdog tick (past timeout_at) → backend.verify_runget_sandbox_agent_url caught Exception and returned None, so 429 / timeout / 5xx looked identical to "sandbox gone", then FAILED.

Broken invariant: "couldn't check" is not the same as "sandbox gone".

Solution

Minimal semantics fix (no watchdog redesign / no new retry policy):

  1. SandboxLookupTransientError for 429, 5xx, and transport failures (404 / empty list still mean absent).
  2. VerificationResult.retryable for transient lookup failures.
  3. Watchdog: if retryable, leave RUNNING for the next tick.

Failure-state diagram

lookup outcome
├── (url, key)      → verify bash → COMPLETED / FAILED as today
├── None (absent)   → FAILED "Sandbox not available" (unchanged)
└── transient error → leave RUNNING (retry next tick)  ← NEW

Test plan

  • pytest tests/test_sandbox_lookup_semantics.py — 11 passed
    • empty list / HTTP 404 → absent
    • 429 / 503 / timeout → transient (not absent)
    • successful lookup unchanged
    • watchdog: retryable leaves RUNNING; absent still FAILS
  • ruff + pyright on changed files

Fixes #285

Distinguish absent sandboxes from rate limits, timeouts, and 5xx so
the watchdog leaves RUNNING and retries instead of false-failing.

Fixes OpenHands#285
@github-actions github-actions Bot added the type: fix A bug fix label Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

type: fix A bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

A transient sandbox-API error (including HTTP 429) marks a running automation as FAILED

1 participant