Skip to content

fix(rollout): derive the hard deadline from budgets after --config-override - #1161

Merged
bingran-you merged 3 commits into
benchflow-ai:mainfrom
Wenbo11:fix/hard-deadline-config-override
Oct 30, 2026
Merged

bingran-you merged 3 commits into
benchflow-ai:mainfrom
Wenbo11:fix/hard-deadline-config-override

Conversation

@Wenbo11

@Wenbo11 Wenbo11 commented Oct 11, 2026 •

Copy link
Copy Markdown
Contributor

Problem

--config-override (or a config_override: block in a run-config YAML) is merged into the task config in Rollout.setup().
The host hard deadline (rollout/_deadline.py, added in #949) never sees that merge.
_computed_deadline_sec re-reads Task(cfg.task_path).config and sums the task file's agent.timeout_sec and sandbox.build_timeout_sec.
So if a run raises the agent budget with --config-override '{"agent":{"timeout_sec":...}}', the agent may legitimately run until the raised budget, but the backstop fires at the sum of the task file's budgets plus the fixed 1800 s margin.
The kill is reported as INFRA_ERROR ("transport or teardown wedged"), so the default retry policy reruns the attempt, and the rerun is killed at the same point.
This is the first item of #1134.

Change

_computed_deadline_sec passes the task config through apply_config_override(..., cfg.config_override) before reading budgets, the same call Rollout.setup() makes.
cfg.timeout still takes precedence, as it does in setup.
With no override, apply_config_override is a no-op.

Tests

tests/test_rollout_hard_deadline.py::TestDeadlineComputation::test_config_override_budgets_dominate_task_budget[agent-timeout|build-timeout] checks that the deadline grows by exactly the raised budget.
Both cases fail on main and pass here.

ruff check, ruff format --check and ty check are clean, and pytest tests/ passes (6502 passed, 69 skipped).

End-to-end repro (local Docker, oracle agent)

slow-oracle/task.md:

---
agent:
  timeout_sec: 60
verifier:
  timeout_sec: 30
sandbox:
  build_timeout_sec: 120
---

Write `done` to `/app/result.txt`.

environment/Dockerfile is FROM ubuntu:24.04 plus WORKDIR /app, oracle/solve.sh is sleep 2100; echo done > /app/result.txt, and verifier/test.sh writes reward 1 when /app/result.txt says done.

bench eval run --tasks-dir slow-oracle --agent oracle --sandbox docker \
  --config-override '{"agent":{"timeout_sec":3000}}' --retry-attempts 0
  • main: [HARD-DEADLINE] slow-oracle: rollout exceeded host deadline (2010s); abandoning sandbox (60 + 30 + 120 + 1800), 0/1 with errors=1, although the rollout's config.json records timeout_sec: 3000.
  • This branch: [PASS] slow-oracle (reward=1.00).

With the default retry policy, main reruns the attempt twice and each rerun is killed at the same point.

Notes

  • Only the first item of Host hard deadline ignores the agent.timeout_sec override and reports the kill as INFRA_ERROR #1134 is covered. Classifying a deadline kill as timeout rather than INFRA_ERROR, and the reviewer deadline, are separate changes.
  • Other places that also ignore the override, left for follow-ups:
    • bench continue (continue_run/orchestrator.py::build_rollout_config) rebuilds the RolloutConfig without the recorded config_override.
    • The resume mismatch check does not compare config_override.
    • Self-gen derives its default generated-skills root from the on-disk sandbox.workdir.

Refs #1134


Devin Review

…erride

The host hard deadline (benchflow-ai#949) re-read the agent and build budgets straight
from the task file, but Rollout.setup() enforces them after merging the
run's config override (0a64522). A run that raised agent.timeout_sec with
--config-override was therefore killed by the backstop at the task's
original budget plus margin, reported as an infra error, and retried.

Apply the same override before summing the budgets.

Refs benchflow-ai#1134

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@bingran-you bingran-you left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified against current main: both new cases fail on main and pass here; ruff, ty and the full suite pass, and release/0.8 carries the same line. Fresh CI test and fixture-scenarios pass; parity and pip-audit fail only on main's existing openclaw manifest drift and dnspython advisory.

@bingran-you
bingran-you merged commit a6bcebe into benchflow-ai:main Oct 30, 2026
10 of 12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants