fix(agentx): close AgentX round timeout gaps + overhead-cap visibility

CHANGELOG.md

@@ -5,6 +5,62 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ## [Unreleased]
 
+### Fixed
+
+- **The AgentX baseline overhead is derived from the warmup bound instead of a
+  flat constant.** `AGENTX_BASELINE_OVERHEAD_SEC` was a single measured number
+  (7200s, calibrated on GLM-5.2/Qwen3.8) covering setup, corpus load, warmup and
+  first-compile. Warmup is the share that actually varies by model, and it
+  already has an operator-visible bound in the client:
+  `AGENTX_WARMUP_GRACE_PERIOD`. A model whose warmup runs long is therefore a
+  model whose operator has already raised that knob — a raw aiperf run against
+  Kimi-K3 at concurrency 64 measured warmup alone at ~12075s, past the entire
+  flat cap. The overhead is now `5400s non-warmup + AGENTX_WARMUP_GRACE_PERIOD`,
+  and every input is logged at INFO so a field timeout can be read back to the
+  values that produced it.<br/>
+  **Operator note**: at canonical settings the cap is unchanged
+  (5400 + 1800 = 7200), so nothing moves for existing synthetic or GLM-5.2-class
+  runs. Raising `AGENTX_WARMUP_GRACE_PERIOD` now also raises the baseline
+  timeout by the same amount — which is the point, but it means the round's
+  worst-case wall clock grows with that knob. `AGENTX_BASELINE_OVERHEAD_SEC`
+  still overrides the derivation outright, and the "nothing has been tuned for
+  this model" warning now fires only when *neither* knob is set.
+
+- **Overriding `HYPERLOOM_PROFILE_MAX_ITERS` under AgentX no longer lifts the
+  host-RAM capture bound silently.** The AgentX branch clamps captured profile
+  steps to 8 because an agentic step carries orders of magnitude more profiler
+  events than the synthetic shape the normal cap is sized against — at the stock
+  cap a DeepSeek-V4 round was OOM-killed mid-capture three times in a row. The
+  operator override is applied afterwards and wins, which is intended, but the
+  two existing warnings could not report it: `cap` defaults to 128, so the
+  obvious `HYPERLOOM_PROFILE_MAX_ITERS=128` was neither below the steady-state
+  floor nor above the cap and restored the full exposure without printing
+  anything. The override is still honoured verbatim; it now warns.
+
+- **`AIPERF_HTTP_TCP_USER_TIMEOUT` is re-stated after the `AIPERF_*` scrub.**
+  `TCP_USER_TIMEOUT` bounds how long Linux tolerates an established connection
+  making no progress, and an agentic turn against a long-context model makes
+  none for as long as the server is prefill-bound. aiperf's stock 30s therefore
+  aborts otherwise-live connections mid-prefill, surfacing as a warmup failure
+  with no server-side error to match it. Upstream's Kimi-K3 and DSv4 recipes all
+  export `900000` (15 min); Hyperloom scrubs every inherited `AIPERF_*` except
+  `AIPERF_BIN`, so an operator setting it had no effect and the client ran on
+  the stock bound. Now exported after the scrub, tunable via
+  `AGENTX_HTTP_TCP_USER_TIMEOUT`.
+
+- **A loosened `AGENTX_FAILED_REQUEST_THRESHOLD` is flagged as a non-canonical
+  workload.** Raising the abort ratio keeps alive a run that upstream's 0.10
+  would have aborted, and the surviving requests are then mapped as an ordinary
+  measurement. aiperf stamps no scenario marker for it — the threshold is the
+  client's own safety net, not part of the scenario — so the round came back
+  `submission_valid=true`. Only a *larger* ratio is flagged; tightening it
+  measures a strictly cleaner run.<br/>
+  **Operator note**: a run that raises this knob is now stamped
+  `submission_valid=false` with `failed_request_threshold=<v>(canonical 0.10)`
+  in `submission_invalid_reasons`, and `benchmark_result.py` will refuse the
+  measurement. Rounds that previously passed on a raised threshold will now be
+  rejected — which is the intended correction, not a regression.
+
 ## [v1.0.0] - 2026-08-26
 Current packaged version (`pyproject.toml`). See
 [release notes](docs/release-notes.md) and the

src/hyperloom/inference_optimizer/assets/agentx/aiperf_client.sh

@@ -29,6 +29,9 @@
 #     non-submittable -- see the smoke note below),
 #   AGENTX_REALTIME_METRICS (rolling stats block; default true),
 #   AGENTX_DATASET_CONFIG_TIMEOUT (default 1800), AGENTX_LIVE_ASSISTANT,
+#   AGENTX_HTTP_TCP_USER_TIMEOUT (no-TCP-progress bound in ms; default 900000,
+#     matching upstream's long-context recipes -- aiperf's stock 30s aborts
+#     live connections while the server is prefill-bound),
 #   AGENTX_MMAP_CACHE_DIR (dataset mmap cache; defaults under $HF_HUB_CACHE),
 #   AGENTX_MAX_CTX (explicit opt-in client-side context cap; NEVER inferred
 #     from $MAX_MODEL_LEN -- see the replay-context note below),
@@ -185,8 +188,10 @@ DURATION="${AGENTX_DURATION:-3600}"
 # period for them to drain before profiling starts. This replaces the old
 # --warmup-duration / --num-warmup-sessions pair, which the scenario does not
 # use and which measured a different thing entirely.
-WARMLANE="${AGENTX_WARMUP_REQUESTS_PER_LANE:-10}"
-WARMGRACE="${AGENTX_WARMUP_GRACE_PERIOD:-1800}"
+CANON_WARMUP_PER_LANE=10
+CANON_WARMUP_GRACE=1800
+WARMLANE="${AGENTX_WARMUP_REQUESTS_PER_LANE:-$CANON_WARMUP_PER_LANE}"
+WARMGRACE="${AGENTX_WARMUP_GRACE_PERIOD:-$CANON_WARMUP_GRACE}"
 
 # Per-trajectory-tree idle cap. NOT the same thing as the scenario's 10s
 # whole-system cap, and NOT scenario-locked -- upstream passes it explicitly
@@ -212,6 +217,15 @@ done < <(env)
 # aiperf validates SERVICE_PROFILE_CONFIGURE_TIMEOUT >= DATASET_CONFIGURATION_TIMEOUT.
 export AIPERF_DATASET_CONFIGURATION_TIMEOUT="${AGENTX_DATASET_CONFIG_TIMEOUT:-1800}"
 export AIPERF_SERVICE_PROFILE_CONFIGURE_TIMEOUT="${AGENTX_DATASET_CONFIG_TIMEOUT:-1800}"
+# TCP_USER_TIMEOUT bounds how long Linux tolerates an established connection
+# making no progress -- and an agentic turn against a long-context model makes
+# no TCP progress for as long as the server is prefill-bound. aiperf's stock
+# 30s therefore aborts otherwise-live connections mid-prefill, which surfaces
+# as a warmup failure with no server-side error to match it. Upstream's
+# Kimi-K3 and DSv4 recipes all export 900000 (15 min) for exactly this, and the
+# scrub above would drop an inherited copy, so it has to be re-stated here or
+# the request timeout is left to a bound two orders of magnitude too small.
+export AIPERF_HTTP_TCP_USER_TIMEOUT="${AGENTX_HTTP_TCP_USER_TIMEOUT:-900000}"
 # Pre-canned assistant replay (recorded responses drive later turns).
 export AIPERF_DATASET_WEKA_LIVE_ASSISTANT_RESPONSES="${AGENTX_LIVE_ASSISTANT:-0}"
 # Headless realtime metrics are opt-in on current aiperf, and the scrub above
@@ -234,7 +248,10 @@ AIPERF="${AIPERF_BIN:-aiperf}"
 # map_aiperf.py carries no error counters. This is the safety net that turns a
 # server/client context mismatch into an honest failure instead of a fabricated
 # win on the surviving short sessions. Matches upstream's 0.10.
-FRT="${AGENTX_FAILED_REQUEST_THRESHOLD:-0.10}"
+# Declared as one value, like CANON_WARMUP_*/WARMLANE below, so the canonical
+# ratio and the default cannot drift apart.
+CANON_FRT=0.10
+FRT="${AGENTX_FAILED_REQUEST_THRESHOLD:-$CANON_FRT}"
 
 # ── Non-canonical workloads may run, but may never be submittable ────────────
 # The scenario enforces a 900s duration floor, so a shortened AGENTX_DURATION is
@@ -255,6 +272,16 @@ FRT="${AGENTX_FAILED_REQUEST_THRESHOLD:-0.10}"
 # leaderboard measurement -- by construction rather than by promise.
 CANON_ENTRIES=393
 CANON_DURATION=3600
+# Warmup is measurement-defining and was missing from this list until a measured
+# run exposed the gap: the agentic warmup is what puts the KV/radix cache under
+# realistic pressure before the window opens, so replaying at 1 request/lane
+# instead of 10 measures a materially emptier cache. It carries no scenario
+# marker either -- aiperf has no concept of "how much warmup is enough" -- so a
+# reduced-warmup round came back submission_valid=true and looked publishable.
+# On a 743B model the canonical 10/lane is a ~2h warmup, which is exactly when
+# an operator reaches for this knob, so the hole was reachable in practice.
+# (CANON_WARMUP_PER_LANE/CANON_WARMUP_GRACE are declared above, alongside
+# WARMLANE/WARMGRACE, so the canonical value and the default can't drift apart.)
 # The corpus this model family canonically replays, before any operator pin.
 # CANON_DS is resolved with the corpus above. The family whitelist behind it is
 # a derivation, not a registry -- a model upstream runs on the full corpus but
@@ -274,6 +301,26 @@ NONCANON=()
 [ "$DURATION" != "$CANON_DURATION" ] && NONCANON+=("duration=${DURATION}s(canonical ${CANON_DURATION}s)")
 [ -n "${AGENTX_MAX_CTX:-}" ] && NONCANON+=("client_context_cap=${AGENTX_MAX_CTX}")
 [ "${AGENTX_UNSAFE_OVERRIDE:-false}" = "true" ] && NONCANON+=("unsafe_override_forced")
+# `-lt`, not `!=`: only a *smaller* value under-pressures the cache or risks
+# truncating the drain before it finishes. A larger value is strictly more
+# warmup than canonical -- e.g. an operator raising the grace period so a
+# large model's warmup has room to drain -- and does not change what gets
+# replayed, so it must not be flagged as a deviation.
+[ "$WARMLANE" -lt "$CANON_WARMUP_PER_LANE" ] && \
+  NONCANON+=("warmup_per_lane=${WARMLANE}(canonical ${CANON_WARMUP_PER_LANE})")
+[ "$WARMGRACE" -lt "$CANON_WARMUP_GRACE" ] && \
+  NONCANON+=("warmup_grace=${WARMGRACE}s(canonical ${CANON_WARMUP_GRACE}s)")
+# The abort threshold is measurement-defining for the same reason warmup is:
+# raising it keeps a run alive that upstream's 0.10 would have aborted, and the
+# surviving requests are then mapped as a normal measurement. aiperf stamps no
+# scenario marker for it -- the threshold is the client's own safety net, not
+# something the scenario knows about -- so a loosened round comes back
+# submission_valid=true. Only a *larger* ratio deviates; tightening it below
+# canonical measures a strictly cleaner run. Compared with awk because the
+# ratio is a decimal, which `-lt` cannot handle.
+if awk "BEGIN{exit !(($FRT) > ($CANON_FRT))}" 2>/dev/null; then
+  NONCANON+=("failed_request_threshold=${FRT}(canonical ${CANON_FRT})")
+fi
 
 SMOKE_ARGS=()
 if [ "$DURATION" -lt "$CANON_DURATION" ] || [ "${AGENTX_UNSAFE_OVERRIDE:-false}" = "true" ]; then
@@ -348,6 +395,29 @@ if [ "${PROFILE:-0}" = "1" ]; then
   # the upstream profile it lands squarely inside setup and captures nothing.
   PWARM="${AGENTX_PROFILE_WARMUP_S:-2700}"
   PWIN="${AGENTX_PROFILE_WINDOW_S:-20}"
+  # The delay is a blind wall clock: it does not know which phase aiperf is in,
+  # and the two ways to get it wrong are NOT symmetric. Opening late is fatal --
+  # aiperf exits, the branch below only logs a warning, and the round produces no
+  # trace at all. Opening early merely captures a still-loaded system slightly
+  # before steady state, which TraceLens can still use. Measured: a 743B model
+  # spends ~2.5h in the agentic warmup, so a delay tuned on a 35B round lands
+  # either mid-warmup or past the end depending on which way the estimate erred.
+  #
+  # So clamp toward "early". The round cannot outlast the warmup drain plus the
+  # measurement window that follows it -- WARMGRACE bounds the former, DURATION
+  # the latter -- so cap the delay at WARMGRACE + DURATION - PWIN - margin and
+  # say when the cap bites. Omitting WARMGRACE would treat DURATION as if it
+  # were the whole round's clock instead of just the measurement phase, and
+  # clamp an operator-tuned PWARM (e.g. ~2.5h for a 743B model's warmup) down to
+  # a fraction of that -- forcing the capture to fire mid-warmup, the exact
+  # failure this self-bracketing exists to avoid. A capture inside warmup is a
+  # usable trace; a capture that never happens is not.
+  _pmax=$(( WARMGRACE + DURATION - PWIN - 60 ))
+  [ "$_pmax" -lt 0 ] && _pmax=0
+  if [ "$PWARM" -gt "$_pmax" ]; then
+    log "WARN profile delay ${PWARM}s exceeds the safe bound for a ${DURATION}s window; clamping to ${_pmax}s so the capture cannot land after the round ends"
+    PWARM="$_pmax"
+  fi
   log "PROFILE=1: self-bracketing profile window (delay=${PWARM}s window=${PWIN}s)"
   run_aiperf & APID=$!
   sleep "$PWARM"

src/hyperloom/inference_optimizer/tests/test_agentx_baseline_timeout.py

@@ -26,6 +26,7 @@
 
 from hyperloom.orchestrator.actions.executors.baseline import (
     AGENTX_BASELINE_OVERHEAD_SEC,
+    AGENTX_CANON_WARMUP_GRACE_SEC,
     AGENTX_DEFAULT_DURATION_SEC,
     BASELINE_DEFAULT_TIMEOUT_SEC,
     BaselineExecutor,
@@ -43,6 +44,7 @@ def _clear(monkeypatch):
         "AGENTX_DURATION",
         "AGENTX_BASELINE_TIMEOUT_SEC",
         "AGENTX_BASELINE_OVERHEAD_SEC",
+        "AGENTX_WARMUP_GRACE_PERIOD",
     ):
         monkeypatch.delenv(k, raising=False)
 
@@ -83,6 +85,73 @@ def test_overhead_budget_is_tunable(monkeypatch):
     assert agentx_baseline_timeout_sec() == AGENTX_DEFAULT_DURATION_SEC + 3600
 
 
+def test_default_overhead_warns_it_may_not_fit_every_model(monkeypatch, caplog):
+    """A raw aiperf run against Kimi-K3 (conc=64) measured warmup alone taking
+    ~12075s -- longer than this whole default cap. Nothing here can tell a
+    long-context/slow-prefill model apart from GLM-5.2/Qwen3.8, the models this
+    constant was measured on, so the gap must be surfaced instead of silently
+    assumed to fit every model.
+    """
+    _clear(monkeypatch)
+    with caplog.at_level("WARNING"):
+        agentx_baseline_timeout_sec()
+    assert any("AGENTX_BASELINE_OVERHEAD_SEC" in r.message for r in caplog.records)
+
+
+def test_explicit_overhead_override_suppresses_the_warning(monkeypatch, caplog):
+    _clear(monkeypatch)
+    monkeypatch.setenv("AGENTX_BASELINE_OVERHEAD_SEC", "20000")
+    with caplog.at_level("WARNING"):
+        agentx_baseline_timeout_sec()
+    assert not any("AGENTX_BASELINE_OVERHEAD_SEC" in r.message for r in caplog.records)
+
+
+def test_overhead_tracks_the_warmup_grace_the_operator_set(monkeypatch):
+    """The knob that bounds the warmup must also size the cap that has to cover it.
+
+    A model whose warmup runs long is a model whose operator has already had to
+    raise AGENTX_WARMUP_GRACE_PERIOD for the round to finish -- the Kimi-K3
+    case, where the flat overhead was smaller than the warmup itself. Deriving
+    from that same knob is what stops the two numbers disagreeing.
+    """
+    _clear(monkeypatch)
+    monkeypatch.setenv("AGENTX_WARMUP_GRACE_PERIOD", "14400")
+    grown = 14400 - AGENTX_CANON_WARMUP_GRACE_SEC
+    assert agentx_baseline_timeout_sec() == (AGENTX_DEFAULT_DURATION_SEC + AGENTX_BASELINE_OVERHEAD_SEC + grown)
+
+
+def test_canonical_grace_reproduces_the_measured_constant(monkeypatch):
+    """Splitting the constant must not move it: same inputs, same number."""
+    _clear(monkeypatch)
+    monkeypatch.setenv("AGENTX_WARMUP_GRACE_PERIOD", str(AGENTX_CANON_WARMUP_GRACE_SEC))
+    assert agentx_baseline_timeout_sec() == (AGENTX_DEFAULT_DURATION_SEC + AGENTX_BASELINE_OVERHEAD_SEC)
+
+
+def test_explicit_overhead_outranks_the_derivation(monkeypatch):
+    """A pinned overhead is an answer, not an input: the grace must not add to it."""
+    _clear(monkeypatch)
+    monkeypatch.setenv("AGENTX_WARMUP_GRACE_PERIOD", "14400")
+    monkeypatch.setenv("AGENTX_BASELINE_OVERHEAD_SEC", "3600")
+    assert agentx_baseline_timeout_sec() == AGENTX_DEFAULT_DURATION_SEC + 3600
+
+
+def test_a_tuned_grace_suppresses_the_uncalibrated_warning(monkeypatch, caplog):
+    """The warning is about nobody having sized this model, not about the default."""
+    _clear(monkeypatch)
+    monkeypatch.setenv("AGENTX_WARMUP_GRACE_PERIOD", "14400")
+    with caplog.at_level("WARNING"):
+        agentx_baseline_timeout_sec()
+    assert not any("AGENTX_BASELINE_OVERHEAD_SEC" in r.message for r in caplog.records)
+
+
+@pytest.mark.parametrize("bad", ["", "  ", "abc", "0", "-1"])
+def test_unparseable_grace_falls_back_to_canonical(monkeypatch, bad):
+    """A typo in the grace must not shrink the cap below the measured default."""
+    _clear(monkeypatch)
+    monkeypatch.setenv("AGENTX_WARMUP_GRACE_PERIOD", bad)
+    assert agentx_baseline_timeout_sec() == (AGENTX_DEFAULT_DURATION_SEC + AGENTX_BASELINE_OVERHEAD_SEC)
+
+
 def test_explicit_cap_wins_outright(monkeypatch):
     _clear(monkeypatch)
     monkeypatch.setenv("AGENTX_DURATION", "7200")