Repository navigation
fix(rollout): derive the hard deadline from budgets after --config-override - #1161
Merged
bingran-you merged 3 commits intoOct 30, 2026
Merged
Conversation
…erride The host hard deadline (benchflow-ai#949) re-read the agent and build budgets straight from the task file, but Rollout.setup() enforces them after merging the run's config override (0a64522). A run that raised agent.timeout_sec with --config-override was therefore killed by the backstop at the task's original budget plus margin, reported as an infra error, and retried. Apply the same override before summing the budgets. Refs benchflow-ai#1134
Open
1 of 7 tasks
bingran-you
approved these changes
Oct 30, 2026
bingran-you
left a comment
Collaborator
There was a problem hiding this comment.
Verified against current main: both new cases fail on main and pass here; ruff, ty and the full suite pass, and release/0.8 carries the same line. Fresh CI test and fixture-scenarios pass; parity and pip-audit fail only on main's existing openclaw manifest drift and dnspython advisory.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
--config-override(or aconfig_override:block in a run-config YAML) is merged into the task config inRollout.setup().The host hard deadline (
rollout/_deadline.py, added in #949) never sees that merge._computed_deadline_secre-readsTask(cfg.task_path).configand sums the task file'sagent.timeout_secandsandbox.build_timeout_sec.So if a run raises the agent budget with
--config-override '{"agent":{"timeout_sec":...}}', the agent may legitimately run until the raised budget, but the backstop fires at the sum of the task file's budgets plus the fixed 1800 s margin.The kill is reported as
INFRA_ERROR("transport or teardown wedged"), so the default retry policy reruns the attempt, and the rerun is killed at the same point.This is the first item of #1134.
Change
_computed_deadline_secpasses the task config throughapply_config_override(..., cfg.config_override)before reading budgets, the same callRollout.setup()makes.cfg.timeoutstill takes precedence, as it does in setup.With no override,
apply_config_overrideis a no-op.Tests
tests/test_rollout_hard_deadline.py::TestDeadlineComputation::test_config_override_budgets_dominate_task_budget[agent-timeout|build-timeout]checks that the deadline grows by exactly the raised budget.Both cases fail on
mainand pass here.ruff check,ruff format --checkandty checkare clean, andpytest tests/passes (6502 passed, 69 skipped).End-to-end repro (local Docker, oracle agent)
slow-oracle/task.md:environment/DockerfileisFROM ubuntu:24.04plusWORKDIR /app,oracle/solve.shissleep 2100; echo done > /app/result.txt, andverifier/test.shwrites reward 1 when/app/result.txtsaysdone.main:[HARD-DEADLINE] slow-oracle: rollout exceeded host deadline (2010s); abandoning sandbox(60 + 30 + 120 + 1800), 0/1 with errors=1, although the rollout'sconfig.jsonrecordstimeout_sec: 3000.[PASS] slow-oracle (reward=1.00).With the default retry policy,
mainreruns the attempt twice and each rerun is killed at the same point.Notes
timeoutrather thanINFRA_ERROR, and the reviewer deadline, are separate changes.bench continue(continue_run/orchestrator.py::build_rollout_config) rebuilds theRolloutConfigwithout the recordedconfig_override.config_override.sandbox.workdir.Refs #1134