Skip to content

feat(keep): --max-latency-ms as a constraint on every promotion - #1297

Open
rpoornac wants to merge 1 commit into
mainfrom
feat/max-latency-ms
Open

feat(keep): --max-latency-ms as a constraint on every promotion#1297
rpoornac wants to merge 1 commit into
mainfrom
feat/max-latency-ms

Conversation

@rpoornac

@rpoornac rpoornac commented Aug 26, 2026

Copy link
Copy Markdown
Collaborator

Why

The optimizer maximized output_throughput and nothing else. Latency was
measured, reported, and fed to the prompts, but no latency number could block a
KEEP.

That is survivable for a lever that raises throughput without touching
per-request latency. It is unsafe for any lever that raises throughput by
making each stream slower — against a throughput-only gate, such a lever does
not merely tolerate a latency regression, it selects for the largest one on
offer.

What

--max-latency-ms names a ceiling on mean end-to-end latency. It is a
constraint, not a target, so it sits outside the --target-* mutually-exclusive
group and combines with them.

Enforcement lives at _lift_to_current_best, the single choke point that writes
current_best. That means it holds for explore, kernel, framework, specialist
and integrate winners alike, rather than only for the lane that got wired first.
Explore applies it a round earlier as well, sparing an over-budget variant a
stack rebench it cannot survive. Where a rebench supersedes the decision round,
the gate grades the rebench's latency, because that is the measurement the
headline throughput comes from — which is why StackRebenchResult now carries
it.

Fail-closed

Off by default; KEEP behaviour is exactly as it was when unset.

When set, the gate fails closed: a candidate that reported no end-to-end latency
is refused, because an unmeasured constraint is not a satisfied one. This makes
latency part of the measurement contract for the promotion paths, so the lift
dicts each lane hands to the choke point now carry their measured latency — that
way the fail-closed rule refuses the untimed, not the unplumbed.
latency_from_result centralizes the three spellings of the field currently in
flight across the executors (e2el_mean_ms, mean_e2el_ms, e2el_ms), so a
lane reporting a different alias is not misread as unmeasured.

The baseline is the one exception: it is the reference point, so an over-budget
baseline warns rather than refuses. It warns loudly, because every subsequent
candidate will then fail the gate.

Refusals are recorded to SharedState.latency_refusals (action, variant, tput,
observed latency, budget, reason). Without that, a session that ends near
baseline is indistinguishable from one that exhausted its headroom.

Tests

56 new tests in test_latency_budget.py: resolution precedence, the predicate,
the field aliases, and call-site wiring for all seven promotion paths.

Review notes

This was split out of #1288, where the flag existed but only ExploreExecutor
honoured it while the help text claimed otherwise. It is orthogonal to the
partition work: the constraint applies to any throughput-for-latency trade, so
it is reviewable and useful on its own.

@rpoornac
rpoornac requested a review from a team as a code owner August 26, 2026 20:29
The optimizer maximized output_throughput and nothing else. Latency was
measured, reported and fed to the prompts, but no latency number could
block a KEEP. That is survivable for a lever that raises throughput
without touching per-request latency, and unsafe for any lever that
raises throughput *by* making each stream slower: against a
throughput-only gate such a lever does not merely tolerate a latency
regression, it selects for the largest one on offer.

--max-latency-ms names a ceiling on mean end-to-end latency. It is a
constraint rather than a target, so it sits outside the --target-*
mutually-exclusive group and combines with them.

Enforcement is at _lift_to_current_best, the single choke point that
writes current_best, so it holds for explore, kernel, framework,
specialist and integrate winners alike rather than only for the lane
wired first. Explore applies it a round earlier too, sparing an
over-budget variant a stack rebench it cannot survive. Where a rebench
supersedes the decision round the gate grades the rebench's latency,
because that is the measurement the headline throughput comes from --
which is why StackRebenchResult now carries it.

Off by default, leaving KEEP behaviour exactly as it was. When set the
gate fails closed: a candidate that reported no end-to-end latency is
refused, since an unmeasured constraint is not a satisfied one. The lift
dicts that lanes hand to the choke point now carry their measured
latency so that fail-closed rule refuses the untimed rather than the
unplumbed, and latency_from_result centralizes the three spellings of
the field in flight across the executors.

Refusals are recorded on SharedState and listed in the report: a
constrained session that ends near its baseline is otherwise
indistinguishable from one that found no headroom, and the two call for
opposite responses. A baseline already over budget warns rather than
failing, since it is the reference the run is measured against.

Co-authored-by: Cursor <cursoragent@cursor.com>
@github-actions

Copy link
Copy Markdown

CI E2E report — ✅ Succeeded

item value
result ✅ Succeeded
model Qwen/Qwen3-0.6B (dense)
resources 1× GPU, TP=1
PR branch feat/max-latency-ms
commit cfb242820d5875d4e4746f29c4af47862fa8070c
session_id 5b5b028e-3fab-4a59-a75b-57e8b7719892
queue → dispatch 0s
run time 170m 29s
total 170m 29s

details

@ZhengGong-amd

Copy link
Copy Markdown
Collaborator
  1. fail-closed silently kills promotion for integrate_patch / framework_agent / specialist / GEAK lanes. Their result dicts never populate e2el_mean_ms (verified: 0 occurrences of e2el in integrate_patch.py, framework_agent.py, specialists/rebench.py). Once any --max-latency-ms is set, latency_from_result returns None for these lanes and every KEEP is refused via latency_unmeasured_under_budget — regardless of actual latency. This only surfaces as a log.warning, so operators see "nothing KEEPs anymore" with no diagnosis. Fix: populate e2el_mean_ms from VariantResult.e2el_mean_ms in the three bench dicts (and fix the already-broken ttft_ms/itl_msttft_mean_ms/tpot_mean_ms while there), or reuse the existing disk-fallback reconstruction in sessions.py. Please also add a test that exercises _lift_to_current_best with each executor's real result shape — the current test hardcodes e2el_mean_ms on every lane and would not have caught this.

  2. The report claims "within budget" unconditionally. report.py's best-config line appends ", within budget" without ever comparing kept_e2el to the budget. Since baseline is allowed to exceed budget (warn-only) and still gets written to current_best, the report can state "1211.0 ms, within budget" against a 200 ms budget. Fix: compare against the budget and label accordingly.

  3. orchestration.md tells the model to read latency_refusals, but nothing renders it into any prompt section (render.py / conversation.py have zero references). Fix: add a Latency budget (constraint) prompt section when a budget is set, or drop the pointer to an invisible field.

  4. Resume path clears the env-layer budget before restoring it. _export_latency_budget(args.max_latency_ms) runs unconditionally before the if args.resume_from: branch, so on resume without the flag it pops HYPERLOOM_MAX_LATENCY_MS before _restore_latency_budget_from_state ever reads it — the documented "CLI > env > archived state" priority is dead code for the env layer. Fix: guard the :1772 export the same way _export_workload_envs_for_optimize already is (only when not args.resume_from).

  5. SKILL.md's operator→CLI flag table doesn't list --max-latency-ms. Since this table is documented as the source of truth for forwarding user-stated flags, a user's latency SLA stated in prompt form will be silently dropped. Please add it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants