The optimizer maximized output_throughput and nothing else. Latency was
measured, reported and fed to the prompts, but no latency number could
block a KEEP. That is survivable for a lever that raises throughput
without touching per-request latency, and unsafe for any lever that
raises throughput *by* making each stream slower: against a
throughput-only gate such a lever does not merely tolerate a latency
regression, it selects for the largest one on offer.
--max-latency-ms names a ceiling on mean end-to-end latency. It is a
constraint rather than a target, so it sits outside the --target-*
mutually-exclusive group and combines with them.
Enforcement is at _lift_to_current_best, the single choke point that
writes current_best, so it holds for explore, kernel, framework,
specialist and integrate winners alike rather than only for the lane
wired first. Explore applies it a round earlier too, sparing an
over-budget variant a stack rebench it cannot survive. Where a rebench
supersedes the decision round the gate grades the rebench's latency,
because that is the measurement the headline throughput comes from --
which is why StackRebenchResult now carries it.
Off by default, leaving KEEP behaviour exactly as it was. When set the
gate fails closed: a candidate that reported no end-to-end latency is
refused, since an unmeasured constraint is not a satisfied one. The lift
dicts that lanes hand to the choke point now carry their measured
latency so that fail-closed rule refuses the untimed rather than the
unplumbed, and latency_from_result centralizes the three spellings of
the field in flight across the executors.
Refusals are recorded on SharedState and listed in the report: a
constrained session that ends near its baseline is otherwise
indistinguishable from one that found no headroom, and the two call for
opposite responses. A baseline already over budget warns rather than
failing, since it is the reference the run is measured against.
Co-authored-by: Cursor <cursoragent@cursor.com>
Why
The optimizer maximized
output_throughputand nothing else. Latency wasmeasured, reported, and fed to the prompts, but no latency number could block a
KEEP.
That is survivable for a lever that raises throughput without touching
per-request latency. It is unsafe for any lever that raises throughput by
making each stream slower — against a throughput-only gate, such a lever does
not merely tolerate a latency regression, it selects for the largest one on
offer.
What
--max-latency-msnames a ceiling on mean end-to-end latency. It is aconstraint, not a target, so it sits outside the
--target-*mutually-exclusivegroup and combines with them.
Enforcement lives at
_lift_to_current_best, the single choke point that writescurrent_best. That means it holds for explore, kernel, framework, specialistand integrate winners alike, rather than only for the lane that got wired first.
Explore applies it a round earlier as well, sparing an over-budget variant a
stack rebench it cannot survive. Where a rebench supersedes the decision round,
the gate grades the rebench's latency, because that is the measurement the
headline throughput comes from — which is why
StackRebenchResultnow carriesit.
Fail-closed
Off by default; KEEP behaviour is exactly as it was when unset.
When set, the gate fails closed: a candidate that reported no end-to-end latency
is refused, because an unmeasured constraint is not a satisfied one. This makes
latency part of the measurement contract for the promotion paths, so the lift
dicts each lane hands to the choke point now carry their measured latency — that
way the fail-closed rule refuses the untimed, not the unplumbed.
latency_from_resultcentralizes the three spellings of the field currently inflight across the executors (
e2el_mean_ms,mean_e2el_ms,e2el_ms), so alane reporting a different alias is not misread as unmeasured.
The baseline is the one exception: it is the reference point, so an over-budget
baseline warns rather than refuses. It warns loudly, because every subsequent
candidate will then fail the gate.
Refusals are recorded to
SharedState.latency_refusals(action, variant, tput,observed latency, budget, reason). Without that, a session that ends near
baseline is indistinguishable from one that exhausted its headroom.
Tests
56 new tests in
test_latency_budget.py: resolution precedence, the predicate,the field aliases, and call-site wiring for all seven promotion paths.
Review notes
This was split out of #1288, where the flag existed but only
ExploreExecutorhonoured it while the help text claimed otherwise. It is orthogonal to the
partition work: the constraint applies to any throughput-for-latency trade, so
it is reviewable and useful on its own.