feat(keep): --max-latency-ms as a constraint on every promotion

CHANGELOG.md

@@ -5,6 +5,38 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ## [Unreleased]
 
+### Added
+
+- **`--max-latency-ms`: a latency constraint on every KEEP.** The optimizer
+  maximized `output_throughput` and nothing else; latency was measured,
+  reported and fed to the prompts, but no latency number could block a
+  promotion. That is survivable for a lever that raises throughput without
+  touching per-request latency, and unsafe for any lever that raises throughput
+  *by* making each stream slower — against a throughput-only gate, such a lever
+  does not merely tolerate a latency regression, it selects for the largest one
+  available.<br/>
+  The flag names a ceiling on mean end-to-end latency in milliseconds. It is a
+  constraint rather than a target, so it sits outside the `--target-*`
+  mutually-exclusive group and combines with them. Enforcement is at
+  `_lift_to_current_best`, the single choke point that writes `current_best`, so
+  it holds for explore, kernel, framework, specialist and integrate winners
+  alike rather than only for the lane that happened to be wired first. Explore
+  also applies it a round earlier, which spares an over-budget variant a
+  stack-rebench it cannot survive. Where a rebench supersedes the decision
+  round, the gate grades the rebench's latency, because that is the measurement
+  the headline throughput comes from.<br/>
+  **Off by default**, which leaves KEEP behaviour exactly as it was. When set,
+  the gate **fails closed**: a candidate that reported no end-to-end latency is
+  refused, since an unmeasured constraint is not a satisfied one. Operator note:
+  this makes end-to-end latency effectively mandatory for promotion under a
+  budget — a lane that never times its candidates will not promote one. Refused
+  winners are recorded in `latency_refusals` and listed in the report, so a
+  constrained session that ends near its baseline can be told apart from one
+  that simply found no headroom. A baseline already over budget warns at
+  promotion time rather than failing, since it is the reference the run is
+  measured against, but it does mean nothing will be kept until a candidate
+  comes in under the ceiling.
+
 ## [v1.0.0] - 2026-08-26
 Current packaged version (`pyproject.toml`). See
 [release notes](docs/release-notes.md) and the

docs/reference/environment-variables.md

@@ -83,6 +83,14 @@ Set with CLI flags, not env vars. Pre-set `ISL` / `OSL` / `CONC` / `PRECISION` /
   `--conc`, `--max-model-len`, `--profile-osl`.
 - **Goal / budget:** `--target-gain`, `--max-hours`, `--target-summary`,
   `--target-tput`, `--compare-against-gpu`.
+- **Constraints:** `--max-latency-ms`. A ceiling on mean end-to-end latency, not
+  a target, so it combines with the `--target-*` flags instead of competing with
+  one. Off by default, leaving throughput the only gate on a KEEP. When set, it
+  is enforced on every KEEP whichever action produced it, and fails closed: a
+  candidate whose latency was never measured is refused, because an unmeasured
+  constraint is not a satisfied one. Set it whenever the search can buy
+  throughput by making each request slower — a throughput-only gate does not
+  merely tolerate that trade, it selects for the largest one available.
 - **Cluster topology & multi-node backend:** `--nodes`, `--gpus-per-node`,
   `--gpu-type`, `--mn-backend` (`rayjob` / `infera`), `--server-args` (rayjob).
   Per-pod sizing, the pod image and pod-side env are the provisioning
@@ -734,6 +742,9 @@ internal-only — do not set them by hand:
 
 * `HYPERLOOM_KERNEL_AGENT_ROOT`: internal CLI-only handoff to the
   kernel subprocess (Python constant `_KERNEL_AGENT_ROOT_ENV`).
+* `HYPERLOOM_MAX_LATENCY_MS`: internal projection of `--max-latency-ms`,
+  written by the CLI so the in-process executors and a resume read the
+  validated value rather than re-deriving it from argv. Use the flag.
 * Any `_INFERENCE_OPTIMIZER_*_INTERNAL_*` symbol: internal toggles for
   the test suite.
 

src/hyperloom/inference_optimizer/cli/__init__.py

@@ -14,6 +14,7 @@
 import asyncio
 import json
 import logging
+import math
 import os
 import shlex
 import sys
@@ -65,6 +66,7 @@
     resolve_model_display_name,
 )
 from hyperloom.orchestrator.actions.executors._aiter_jit import clean_stale_aiter_locks
+from hyperloom.orchestrator.actions.executors._latency_budget import LATENCY_BUDGET_ENV
 from hyperloom.orchestrator.actions.executors._workload_envs import (
     agentx_enabled as _agentx_enabled,
 )
@@ -1631,6 +1633,58 @@ def _export_operator_launch_shape(
         os.environ.pop("INFERENCE_OPTIMIZER_EXTRA_ENV", None)
 
 
+def _export_latency_budget(max_latency_ms: float | None) -> float:
+    """Validate ``--max-latency-ms`` and project it into env.
+
+    Validated here, at launch, rather than where the gate reads it: a budget
+    that cannot be parsed should be a usage error before the session starts, not
+    a constraint that silently evaluates to "off" three hours in. An empty input
+    clears the variable so a second session in the same shell cannot inherit a
+    budget the operator did not ask for this time.
+
+    Args:
+        max_latency_ms: The resolved flag value, if any.
+
+    Returns:
+        The canonical budget in ms, ``0.0`` when the gate is off.
+    """
+    budget = float(max_latency_ms or 0.0)
+    if budget and not math.isfinite(budget):
+        print("ERROR: --max-latency-ms must be a finite number", file=sys.stderr)
+        sys.exit(2)
+    if budget < 0:
+        print(f"ERROR: --max-latency-ms must be positive, got {budget:g}", file=sys.stderr)
+        sys.exit(2)
+    if budget > 0:
+        os.environ[LATENCY_BUDGET_ENV] = repr(budget)
+    else:
+        os.environ.pop(LATENCY_BUDGET_ENV, None)
+    return budget
+
+
+def _restore_latency_budget_from_state(args: Any, state: SharedState) -> None:
+    """Fill the budget from env or archive when this resume omitted the flag.
+
+    Priority is CLI flag > already-exported env > archived ``SharedState``, the
+    same chain :func:`_restore_operator_supplied_paths_from_state` applies to the
+    custom-workload paths. The resolved value is written back onto ``args`` so
+    :func:`_export_latency_budget` stays the only writer of the env it owns, and
+    so a restored budget is validated rather than trusted.
+
+    The archive tier is what lets a resume reproduce the session's measurement
+    contract: a budget is part of *which* candidates were admissible, so a
+    resume that silently dropped it would start keeping configurations the
+    original session had refused.
+
+    Args:
+        args: Parsed CLI namespace, updated in place.
+        state: Resumed session state.
+    """
+    if not (getattr(args, "max_latency_ms", None) or 0.0):
+        env_budget = os.environ.get(LATENCY_BUDGET_ENV, "").strip()
+        args.max_latency_ms = float(env_budget or state.latency_budget_ms or 0.0)
+
+
 # Terminal stop_reasons that represent a clean, successful optimizer run (exit 0).
 # Anything else (baseline / preflight failures, crashes, enablement stalls) exits
 # non-zero so CI surfaces genuine problems.
@@ -1715,6 +1769,7 @@ async def _run_optimize(args: argparse.Namespace) -> int:
         server_args=str(getattr(args, "server_args", "") or "").strip(),
         extra_env=parse_operator_extra_env(args),
     )
+    _export_latency_budget(getattr(args, "max_latency_ms", None))
     # Project resolved workload knobs into env for the fresh-launch path only.
     # A resume must NOT export here: ``args.tp``/etc. are still unresolved
     # (``None`` -> 1) because the persisted SharedState is loaded later; the
@@ -1952,6 +2007,12 @@ async def _run_optimize(args: argparse.Namespace) -> int:
             gpu_type=os.environ.get("GPU_TYPE") or state.gpu_type,
         )
         _persist_operator_supplied_paths(state)
+        # The budget is part of the measurement contract, so it resumes on the
+        # same restore / apply / persist path as the custom-workload paths above.
+        _restore_latency_budget_from_state(args, state)
+        state.latency_budget_ms = _export_latency_budget(getattr(args, "max_latency_ms", None))
+        if state.latency_budget_ms:
+            print(f"  re-exported max_latency_ms: {state.latency_budget_ms:g}")
         if state.framework_repo_path:
             print(f"  re-exported FRAMEWORK_REPO_PATH: {state.framework_repo_path}")
         if state.bypass_scripts_dir:

src/hyperloom/inference_optimizer/cli/bootstrap.py

@@ -22,6 +22,7 @@
 from hyperloom.common.coerce import to_unix
 from hyperloom.common.env import forge_explicitly_enabled
 from hyperloom.common.timeutil import now_iso
+from hyperloom.orchestrator.actions.executors._latency_budget import read_session_budget
 from hyperloom.orchestrator.actions.executors._workload_envs import (
     agentx_enabled as _agentx_enabled,
 )
@@ -355,6 +356,11 @@ def _resolve_framework_version(args_in: Any) -> str:
         bypass_scripts_dir=os.environ.get("HYPERLOOM_BYPASS_SCRIPTS_DIR", "").strip(),
         framework_repo_path=os.environ.get("FRAMEWORK_REPO_PATH", "").strip(),
         benchmark_backend=os.environ.get("HYPERLOOM_BENCHMARK_BACKEND", "").strip().lower(),
+        # Read back from the env the CLI published rather than re-derived from
+        # argv: that env is the validated, canonical form, and seeding from the
+        # raw flag would let the persisted manifest disagree with the budget the
+        # executors are actually gating on.
+        latency_budget_ms=read_session_budget(),
         nodes=max(1, int(getattr(args, "nodes", 1) or 1)),
         robustness_options=_build_robustness_options(args),
         warm_replay_enabled=not bool(getattr(args, "no_warm_replay", False)),

src/hyperloom/inference_optimizer/cli/parser.py

@@ -588,6 +588,23 @@ def _build_parser() -> argparse.ArgumentParser:
     grp.add_argument(
         "--target-baseline-dir", type=str, default=None, help="Stop when current best matches the baseline in DIR"
     )
+    opt.add_argument(
+        "--max-latency-ms",
+        type=float,
+        default=None,
+        metavar="MS",
+        help="Refuse any candidate whose mean end-to-end latency exceeds MS. A "
+        "constraint, not a target, so it is outside the --target-* group and "
+        "combines with them rather than replacing one. Off by default, which "
+        "leaves throughput the only gate on a KEEP. Set it whenever the "
+        "search can buy throughput by making each request slower, where a "
+        "throughput-only gate does not merely tolerate the regression but "
+        "selects for the largest one available. Enforced on every KEEP, "
+        "whichever action produced it, and in explore's decision round so an "
+        "over-budget variant never earns a rebench. The gate fails closed: a "
+        "candidate that reported no latency is refused, because an unmeasured "
+        "constraint is not a satisfied one.",
+    )
     opt.add_argument(
         "--resume-from",
         type=str,