fix(sandboxes): bound background output retrieval

[DamianB-BitFlipper]

Caps background-job output reads at 20 concurrent requests, retrieves stdout/stderr
sequentially, preserves completed exit codes when output retrieval fails, adds jittered retries and
errno diagnostics, and includes a 100-job concurrency test.

Verification:

• 82 focused tests passed
• Ruff checks passed
• Wheel build passed

---

Note

Medium Risk
Changes observable SDK behavior (timeout meaning, optional None stdout/stderr, new error fields) on a hot path used by batch VM job polling and gateway reads.

Overview
Hardens background job polling when many VM jobs finish at once: stdout/stderr tails are fetched sequentially under a client-wide cap of 20 concurrent gateway reads and a per-job output deadline (45s default; get_background_job/get_background_jobs timeout now bounds retrieval after the exit file is seen, not each read_file).

Once an exit code is known, completed and exit_code stay authoritative even if log reads fail or time out. BackgroundJobStatus gains optional stdout_error / stderr_error; failed streams are None instead of failing the whole status. Async output reads no longer run stdout/stderr in parallel.

read_file connection errors use _format_exception_diagnostic (nested errno, e.g. EMFILE) and jittered exponential retries. README and tests cover concurrency limits, deadline behavior, and partial output failures.

^{Reviewed by Cursor Bugbot for commit 8ca296c. Bugbot is set up for automated code reviews on this repo. Configure here.}

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8ca296c473

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

[chatgpt-codex-connector]

Enforce the deadline around synchronous output reads

When a synchronous gateway read stalls or repeatedly times out, this call is not bounded by deadline: read_file invokes the retry-decorated _gateway_read_file_get up to four times, and every attempt may consume the full request_timeout plus retry delays. With the default settings, one stream can therefore block for roughly two minutes despite the advertised 45-second output deadline, and successful output may even be returned after that deadline. Wrap the complete synchronous read/retry operation in an actual deadline mechanism or make its retries consume the remaining budget.

Useful? React with 👍 / 👎.

[cursor]

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

Want reviews to match your repository better? Bugbot Learning can learn team-specific rules from PR activity. A team admin can enable Learning in the Cursor dashboard.

^{Reviewed by Cursor Bugbot for commit 8ca296c. Configure here.}

[cursor]

Output timeout hits exit checks

Medium Severity

timeout is documented as an output-retrieval deadline after completion is known, but get_background_job still forwards it to the exit-file read_file call and get_background_jobs still forwards it to the platform batch request. A short or zero deadline can fail completion detection itself instead of returning completed=True with stdout_error / stderr_error.

Additional Locations (2)

• packages/prime-sandboxes/src/prime_sandboxes/sandbox.py#L3046-L3058
• packages/prime-sandboxes/src/prime_sandboxes/sandbox.py#L1554-L1555

 

^{Reviewed by Cursor Bugbot for commit 8ca296c. Configure here.}

[cursor]

Sync deadline skips retries

Medium Severity

The async output path bounds the whole fetch with asyncio.wait_for, but sync only checks the deadline before calling read_file. That call still runs multi-attempt tenacity retries, so sync retrieval can run far past the configured output deadline while holding a _background_job_output_semaphore slot.

 

^{Reviewed by Cursor Bugbot for commit 8ca296c. Configure here.}