fix: guard None raw tokens in logprobs detokenization - #3304
Draft
mikasenghaas wants to merge 1 commit into
Draft
Conversation
tokenizer.convert_ids_to_tokens returns None for ids outside the tokenizer vocabulary. Models with embedding padding past the tokenizer (e.g. R1-Distill-Qwen) can sample such ids; _restore_leading_spaces then iterates None and the TypeError kills the AsyncLLM output loop and the whole API server with it. Observed as recurring random 'ApiServer died with exit code None' crashes under sustained sampling.
mikasenghaas
added a commit
that referenced
this pull request
Aug 19, 2026
The detokenizer patch ships separately in #3304; the pydantic-config pointer bump was an accidental sweep of local submodule state.
mikasenghaas
added a commit
that referenced
this pull request
Aug 19, 2026
* docs: add adaptive concurrency design Feedforward token budget (n_max = kappa * C / G) from learned per-env episode costs, trimmed by a binary engine-overload signal (AIMD on kappa). Covers eval/train cost weighting, the step-1 freeze, and KV offloading. Supersedes the reactive KV-target controller from #2908. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: clock adaptive-concurrency updates to steps Re-evaluate n_max once per training step (G reweigh + at most one kappa growth) - aligned with the per-step eval gate and naturally plant-clocked, since step time tracks mean episode time. HARD cuts stay on the 5s poll: waiting for a step boundary would sustain thrash. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: spec ConcurrencyController abstraction + config Self-contained controller with [orchestrator.concurrency] config: max_inflight ceiling (replaces orchestrator.max_inflight_episodes), optional initial_inflight to skip the ramp, adaptive kill switch. Kappa initialized by continuity at freeze exit. Pure state machine driven by collector pushes, dispatcher completions, and the step loop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: adaptive concurrency always on Drop the adaptive flag from the design - the controller is not optional. max_inflight is the only escape hatch (hard ceiling). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: adaptive rollout concurrency controller ConcurrencyController sizes the dispatcher's in-flight cap as n_max = clamp(kappa * C / G, group_size, max_inflight): a feedforward token budget from engine KV capacity and learned per-env episode costs, trimmed by a binary per-engine overload signal (HARD preemption cut on the 5s poll, per-step growth otherwise). Always on. - [orchestrator.concurrency]: max_inflight ceiling + initial_inflight; replaces max_inflight_episodes and oversampling_factor - InferenceMetricsCollector always polls (wandb mirroring stays gated), parses cache_config_info labels for KV capacity, and pushes EngineLoadSample facts to the controller; the old waiting-queue overload warning is subsumed by the SOFT signal - Dispatcher: dynamic set_limit, burst-capped refill, episode completion hook (env, kind, tokens, duration) - configs/concurrency/: three e2e configs (low start, overload start, GLM-Air SWE with 10-turn eval) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: backoff naming + config comment fixes Rename cut factors to BACKOFF_FACTOR / ESCALATED_BACKOFF_FACTOR, describe kappa as the over-commit factor, add wandb project + max_num_seqs to the hendrycks e2e configs (the vLLM default of 128 would queue instead of building KV pressure), and point verification at the W&B gauges (the periodic logger does not feed metrics.jsonl). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: add no-eval concurrency e2e config Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: frozen_steps knob + 50-step e2e configs concurrency.frozen_steps (default 0) pins the cap for the first k steps; the first re-evaluation lands at the k -> k+1 boundary. HARD cuts stay live inside the freeze. The e2e configs run 50 steps at default gpu_memory_utilization with AIME avg@4 evals every 10 steps, 8k max_model_len, and run names matching the config file names. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: motivate higher frozen_steps values Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: rename concurrency e2e configs hendrycks-low-no-eval -> hendrycks-initial-low, hendrycks-low -> hendrycks-initial-low-eval, hendrycks-high -> hendrycks-initial-high-eval. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: migrate example configs to initial_inflight The old max_inflight_episodes values become concurrency.initial_inflight starting caps - a hard ceiling is usually not wanted with the adaptive controller. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: use config blocks for concurrency in examples Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: raise max_off_policy_steps in hendrycks configs The adaptive cap runs hundreds of episodes in flight against a 64-sample batch, so episodes span more steps than the default off-policy budget of 8. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: slash-namespace dispatcher gauge keys dispatcher/inflight/{train,eval,groups} and dispatcher/off_policy_level/{max,mean}, mirroring the cancelled/errored/queued key scheme. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: drop groups-in-flight gauge Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: derive initial cap from engine max context With initial_inflight unset, the starting cap is the pessimistic bound C / max_model_len: an episode cannot exceed the engine context, so that many episodes always fit. max_model_len comes from /v1/models (set explicitly or taken from the model config) and also replaces seq_len as the cost-estimate bootstrap once known. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: richer concurrency logging + wandb inflight mirror - Active-voice change lines with state snapshots: '{Increased,Decreased} concurrency {x} -> {y} at step {n} ({reason}) - kappa=.. cost=.. capacity=.. signal=..' plus per-env EWMA snapshots at debug - One-time capacity report after the first poll ('Inference reports 3.0M tokens of KV cache capacity - max model len 8.2K') and a derivation line when the initial cap is runtime-derived - Collector accepts extra_metrics and mirrors inference/dispatcher/inflight/{train,eval} into the per-poll W&B payload so inflight reads on the engine time axis Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: show inflight vs cap in the wandb overview Drop the inference/-prefixed mirror; the dispatcher gauges keep their own keys and join the overview's inference section as one panel (dispatcher/inflight/{train,eval} + concurrency/max_inflight), on the same wall-time axis as the engine metrics. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: survive out-of-vocab ids in logprobs detokenization Padded-vocab models (Qwen: logits dim > tokenizer vocab) can sample out-of-vocab ids once RL updates shift the untrained tail logits. vLLM's _restore_leading_spaces then iterates a None raw piece and the TypeError kills the async output processor and the engine (hit twice in the 50-step concurrency verification runs, at steps 37 and 12). Guard the raw piece; the id still detokenizes via decode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: apply concurrency bootstrap only once The bootstrap guard in observe() held true on every metrics poll until the first train step set kappa. The cap was recomputed from the moving cost EWMA every ~5s, so it flapped by hundreds of episodes before step 1. Latch it: one feedforward raise on the first capacity observation, then updates only at step boundaries (HARD cuts stay live). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: say episodes for in-flight units The dispatcher's permit unit is one episode (one env.run, possibly several traces), so in-flight logs and identifiers now say episodes: InflightEpisode, cancel_inflight_{train_,}episodes, and the console lines (pipeline view, off-policy cancel warning, drain notice, mode switch). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: drop cap series from overview inflight panel Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: fix stale overview comment Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: step-clock the cost EWMA folding The estimates updated per completed episode with alpha 0.1 - a ~10-episode memory over a heavy-tailed length distribution, which swung G (and the cap) by 20-30% between steps. Completions now accumulate between step boundaries and fold in as one interval mean (alpha 0.3 per step), so a step's worth of episodes averages out before the estimate moves. Simulated max step-to-step cap swing drops to ~5% under a Pareto length mix at 150 completions/step. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: interpolate eval and train cost mixes The eval census was a hard 0 -> 1 -> 0 gate: while any eval work was in flight, G priced the whole pool from the eval mix, then snapped back to the train mix at the drain boundary. Now the census decrements per completed eval episode and G blends the two mixes on the eval share of the cap, min(1, remaining / n_max): a full jump only when the epoch actually fills the pool, and a continuous glide back to the train price as it drains. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: 512-episode bias-corrected cost EWMA Replace the step-folded estimator with a per-episode EWMA at alpha = 1/512: the smoothing horizon is fixed in episodes, so it no longer scales with batch size or step cadence. Bias-corrected (decayed sum over decayed weight) so early estimates equal the plain mean instead of anchoring on the first episode - an eval env at 120 episodes per epoch prices correctly from its first epoch. Simulated step-to-step cap swing stays ~5% under a Pareto length mix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: default frozen_steps to 1 At the step-1 boundary the cost EWMA has only seen the episodes that completed fastest — the short ones — so the first re-evaluation overshoots the cap (observed 256 -> 2139 with a 2K estimate on SWE episodes that average far longer). Holding the cap for one step lets the first full batch complete before the estimate is trusted. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: make frozen_steps=1 actually freeze the first boundary on_step incremented completed_steps before checking frozen, so the step-1 boundary compared 1 < frozen_steps and values 0 and 1 behaved identically (observed: frozen_steps=1 still re-evaluated at step 1 off a 1.8K biased cost estimate). Snapshot frozen before the increment: k now freezes the first k boundaries. Also exempt the one-shot bootstrap derivation from the freeze — it sets the starting value the freeze pins, so blocking it under the new default of 1 would strand no-initial_inflight runs at the floor. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: simplify frozen_steps docstring Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: cap hendrycks e2e configs at 1024 inflight The duty-cycle gap (episodes spend ~35% of wall time outside an open engine request) lets kappa compound the cap well past useful engine load on the sanity envs - ceiling it at 1024. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: allow SignSGD with full optimizer offload SignSGD is stateless, so it runs through the wrapper's generic per-chunk CPU step path (plain GradientOffloadManager + _step_chunk) with no AdamW-specific machinery. Halves per-rank CPU optimizer state vs AdamW full offload by dropping both moment buffers. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: queue-overload HARD cut for agentic rollouts Agentic rollouts overload inference by queueing, not preempting: vLLM's admission control parks excess load in the waiting queue, so the preemption HARD gate never fires (observed: 213 capacity-queued vs 192 running, prefix-cache hit rate 0.06, num_preemptions_total 0, and a 5x step-time blowup while the controller raised the cap on a SOFT signal). Classify HARD when capacity-queued requests exceed QUEUE_RATIO of running for QUEUE_PERSISTENCE_POLLS consecutive polls (persistence filters natural turn-completion bursts), and cut to just under what the engines actually serve (0.9x running) instead of backing off from inflated inflight. The metrics collector now ships running and the by-reason waiting breakdown in EngineLoadSample; engines without the breakdown fall back to total waiting. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: shed youngest episodes on overload cut A cut only blocks new admissions, so the over-admitted episodes keep thrashing the prefix cache until they finish at degraded throughput (observed: 690 -> 320 cut took 20+ min to drain, with 10-18 min steps and a 0.05 hit rate the whole way). Bind an on_overload hook from the controller to the dispatcher: cut() reports the episode excess and the dispatcher drops that many in-flight train episodes, youngest groups first (least inference spend), reusing the off-policy drop_group path so the sink finalizes partial groups. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: slew-limit cap growth to 1.25x per re-evaluation A raise floods in fresh episodes whose fast finishers drag the cost EWMA down, inviting a bigger raise (observed 256 -> 567 -> 690 while the estimate fell 7.7K -> 6.3K). Limiting upward moves to 1.25x per step turns the spiral into a ramp the queue-overload cut interrupts after one increment. Cuts stay unlimited. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: cuts never raise the cap; unwire shed-youngest for now The queue cut targets 0.9x running, but running can be inflated by work the dispatcher no longer tracks — observed live: 'Increased concurrency 231 -> 256 (queue overload)'. Clamp cut targets to the current cap. Unwire on_overload -> shed_youngest: cancelling the orchestrator task does not abort the env-side rollout (3195 starts vs 2616 done after shedding 489 episodes, none stopped Cancelled), so shedding leaves zombies hitting inference while the freed permits admit new episodes on top — total load rises. Keep the dispatcher method; rewire once rollout abort propagates to the env server. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: floor the cost estimate with measured request cost Completion-based estimates only see finished episodes; after a cap raise the fast short episodes finish first and drag the estimate down, inviting a bigger raise (observed 7.7K -> 6.3K -> 5.5K while the cap spiraled 256 -> 690). The live request stream has no such bias: every turn resends its episode's full context, so in-flight episodes are counted at their current size the whole time they run, and single-turn workloads are covered by the generation term. The collector derives mean prompt+generation tokens per request from the vLLM histogram deltas each poll; the controller folds them into a request-weighted decayed mean and takes the max of it and the completion mix. Drop frozen_steps in the e2e config to test that the cold-start freeze is no longer needed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: size-bias the measured request cost A plain per-request mean under-prices the in-flight pool: small contexts take turns far more often than the large residents that actually fill KV (observed: request mean 6.7K while resident cost was 13.2K per running request, so the feedforward equilibrium sat 2x above true capacity and every ramp ended in a queue cut). Weigh each request by its own token mass instead — E[X^2]/E[X] from the prompt-histogram bucket deltas, midpoint-approximated, +Inf bucket bounded by the engine max context. On the live GLM distribution this reads 16.8K vs the 13.2K resident measure: slightly conservative, and kappa growth learns back the slack. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: clock kappa growth on polls, not step boundaries Step-clocked growth adapts at wildly different speeds across workloads (steps range seconds to tens of minutes), and its binding check sampled inflight at the boundary instant — exactly when episode completions burst — so kappa stalled while the cap was binding all step long (observed: kappa stuck at 1.13 for three clear steps with the engine at 0.99 prefix hit). Grow kappa a small factor per clear-and-binding poll instead; SOFT vetoes become per-poll for free. Also fall back to the observed request cost (not the pessimistic engine max context) for envs without completions — the max-context fallback made kappa's continuity init read a user-set start as an intentional ~4x over-commit whenever step 1 completed before any episode did. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: early growth brake and kappa ceiling memory Thrash onset is a cliff, not a slope: prefix hit went 0.97 -> 0.06 within two minutes of kappa growing past the workload's true over-commit (job 748: clean climb to kappa 1.51 / cap 380, then a queue cut to 197). Two additions: - Veto kappa growth on any poll with capacity-queued requests. Unlike generic waiting, capacity-queuing means KV blocks are full right now; the 2-poll waiting filter let growth feed the cap through the cliff between bursts. - Remember the ceiling: an overload cut pins growth at 0.9x the kappa that overloaded, so the ceiling is rediscovered by a wobble instead of a fresh thrash episode every cycle (without memory the sawtooth re-pays the full discovery cost forever). A slow relief (~+2% per 10 min) lets a genuinely lightened workload re-probe. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * docs: update adaptive concurrency for the measured-cost controller Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: clock the controller on pipeline turnovers, not steps Step duration varies from seconds to tens of minutes across workloads, so anything step-clocked adapts at workload-dependent speed. Re-evaluate the cap every poll (with a 2% deadband against estimate noise) and bound raises by the pipeline turnover: each completion advances the clock by 1/inflight, so one turnover means the in-flight pool has been replaced once regardless of scale — 100 completions at concurrency 10 buy 10x the raise budget of 100 at concurrency 1000, matching how much less biased their estimates are. Raises compound at most 1.25x per turnover; cuts and decreases are never limited. The turnover slew subsumes frozen_steps (the cap can never outrun the pipeline's demonstrated stability), so the config field is removed. Kappa now initializes on the first poll with real cost data instead of the first step boundary; on_step shrinks to eval bookkeeping and backoff reset. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: SOFT signal on high KV usage Thrash onset is a cliff at full KV, and the queue-based signals only fire once it is already happening. Live-request KV usage rises ahead of it (measured ~0.53 at healthy load, ~1.0 at thrash), so treat any decode engine above 0.8 as SOFT: growth stops while there is still headroom to absorb in-flight context growth, cuts stay queue-driven. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: lower KV-usage SOFT threshold to 0.7 Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: shed on overload now that rollout abort propagates Bump verifiers to feat/rollout-abort (26eafc07): cancelling the orchestrator-side task now aborts the env-side rollout, so Dispatcher.shed_youngest no longer creates zombies — rewire it to the controller's on_overload hook. An overload cut now sheds the youngest in-flight episodes immediately instead of draining the excess for tens of minutes at thrashed throughput. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: proportional kv-headroom trim Cap moves alone cannot relieve pressure from episodes maturing in place — they only block admissions. Observed on the kv-soft gate: growth froze at usage 0.7 and the cap eased 371 -> 326, but the admitted pool's contexts grew ~15%/step under it and crossed the thrash cliff (prefix hit 0.05) within minutes anyway. With rollout abort now propagating, trim the pool itself: above KV_USAGE_TARGET (0.85), set the cap to inflight * target / usage and shed the excess youngest episodes, on a short cooldown so each trim propagates before the next is sized. Holds usage at the target continuously instead of riding to the cliff and paying a full queue-overload cut; the queue gate remains the backstop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: kv-headroom trims re-derive kappa A trim that lowers only the cap gets undone by the feedforward: the slew allowance regrows per completion and the cap creeps back into the trim zone, shedding ~8-16 young episodes every cooldown (observed: 4 trims in 2.5 min). Re-derive kappa from the trimmed cap the way cuts do; with growth already vetoed above KV_USAGE_SOFT this yields a grow (<0.7) / hold (0.7-0.85) / trim (>0.85) usage band instead of a shed loop. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: address PR review comments - Treat the admitted unit as a black box in the controller docstring; docstring-style constants describing meaning only. - Constants: ESTIMATE_ALPHA 1/1024, BACKOFF_FACTOR 0.8, KAPPA_MAX 8, QUEUE_PERSISTENCE_POLLS 6 (react within a minute). - Rename RolloutDispatcher -> Dispatcher, inflight_permits -> current_inflight, shed_youngest -> cancel_inflight. - Drop the out-of-vocab detokenization patch. - Configs: initial_inflight = batch_size with a 2x max_inflight ceiling across the example and nightly configs. - Docs: trim the adaptive-concurrency intro. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: weight the cost mix by live in-flight counts The sampling-ratio weighting predicted the standing mix and needed two corrections bolted on: duration weighting (admission mix vs standing mix) and the eval-census interpolation (predicting the mix through eval epochs). The dispatcher knows the standing mix exactly — weight the per-env estimates by its live per-(kind, env) in-flight counts instead. Train and eval need no separate treatment (eval units enter the mix as they are admitted), so the census, its interpolation, and the duration weighting are deleted; the configured ratios remain only as the empty-pool fallback. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: hysteresis for kv-headroom trims Trimming exactly to the trigger guarantees an immediate re-trigger under monotonic pool growth (observed: trims every cooldown, 16-40 units each, while survivors grew ~15%/step). Trigger at 0.85 but resize to 0.75, so each trim buys growth headroom before the next. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: lower kv-usage thresholds (soft 0.6, trigger 0.8, target 0.7) Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat!: reduce the controller to measured AIMD The feedforward kappa*C/G no longer controlled anything: the turnover slew was the binding constraint on every ramp (raw targets sat 2-5x above the cap), the usage trim overrides it on the way down (it was added precisely because G-tracking is too slow for in-place pool growth), and the gates own overload. Everything that computed G existed to produce a number the other mechanisms clipped. Replace it with AIMD on measured KV pressure, turnover-clocked: grow the cap x1.25 per pipeline turnover while engines are clear, usage < 0.6 and the cap binds; trim pool+cap to usage 0.7 above 0.8; cut on preemptions (0.8x inflight) or persistent capacity-queueing (0.9x running), cancel the excess, drain-latch, escalate to 0.5x if overload survives a drain. The starting cap is user-set or C / max_model_len — the only cost-like quantity left, read once. Deleted: kappa and its ceiling/relief, both cost estimators (per-env completion EWMAs and the size-biased request census with its histogram parsing), cost_estimate/mix_cost, the inflight-mix hook, the re-evaluation deadband, slew_allowance as a separate mechanism (the growth law absorbs it), and on_step. ~480 -> ~280 lines with no model of unit cost anywhere. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: align tests, configs and docs with the AIMD controller Fix the hendrycks configs (high-eval start needs its ceiling at the starting cap; freeze-era header comments), rewrite the adaptive concurrency docs for the measured-AIMD design, and revert the SignSGD full-offload allowance — SignSGD crashes with an async CUDA illegal memory access on this stack, so the guard stays until that is fixed. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: min_inflight escape hatch A hard floor on the adaptive cap (None auto-sets to 1). Setting min_inflight = max_inflight pins the cap, recovering fixed-concurrency behavior; not recommended in practice. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: address second review round - Rename Dispatcher param max_inflight_episodes -> initial_max_inflight and document its relation to max_inflight_ceiling (starting value of the dynamic cap vs the configured hard maximum bounding out_q). - Drop the dead n <= 0 guard in cancel_inflight (resize_down only calls it with a positive excess). - Public hook attributes on the controller (set_limit, get_inflight, on_overload); single float-returning clamp; unconditional apply_limit on growth (it no-ops when unchanged). - Gauges: capacity_tokens -> capacity; drop queue_overload_polls. - Remove the stale detokenization NOTE in the vLLM server. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: revamp the concurrency e2e config matrix Two workloads x four scenarios, all descriptively named and capped at max_inflight = 1024: - hendrycks (single node, 4 train + 4 infer GPUs, mirrors the hendrycks-sanity example; replaces the 2-GPU variants): initial-low (16, grow from near zero), initial-none (derived start), initial-high (1024, cut from overload), evals (256 + AIME avg@4 every 10 steps). gpu_memory_utilization = 0.2 shrinks the KV budget artificially so the controller operates below the ceiling instead of at huge concurrency. - glm-air (2 nodes, replaces glm-air-swe): the same three initial scenarios plus evals (256 + 10-turn-capped SWE-Bench Verified every 10 steps). Each config uses its own sandbox label so parallel runs cannot delete each other's sandboxes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: slurm blocks for the hendrycks e2e configs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: pin flash_attention_2 for the hendrycks e2e configs R1-Distill-Qwen-1.5B resolves to the HF model impl, and attn=auto resolves to flash_attention_3, which requires the custom impl — the trainer refused to start. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: drop the initial-low e2e variants The low start exercises the same turnover ramp as the derived start, just from a lower anchor — the none variant covers it. Final matrix: {none, high, evals} x {hendrycks, glm-air}. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: 4 API server processes for the hendrycks e2e configs The initial-high run (1024 concurrent clients) killed vLLM's single ApiServer process; the frontend needs to scale with the client count. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: pin the hendrycks KV budget via num_gpu_blocks_override Even at low gpu_memory_utilization the 1.5B model gets ~12.8M KV tokens, putting the usage band above the max_inflight ceiling — the derived start clamped to the ceiling and the none scenario degenerated into the high one. Pin ~2.1M tokens so the derived start (~256) and the band (~350-400) sit well below the ceiling. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * feat: min_inflight is the only cap floor Drop the implicit group_size floor: group episodes are admitted individually, so a group trickles through in waves below its size — nothing requires a group to fit in flight at once. min_inflight (default 1) is now the single lower bound, and recovery from tiny caps is fast in turnover terms (at inflight 1 every completion is a full turnover). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: start hendrycks-initial-high at 700 Two attempts at a 1024 start died to vLLM ApiServer processes being signal-killed under the client connection storm (with 1 and with 4 frontend processes). 700 is still decisively over the pinned ~350-400 usage band, so the overload-cut scenario is intact without stressing the frontend. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: prime VM sandboxes for hendrycks-initial-high The ApiServer deaths under the 1024 start were host thrash from 1024 subprocess-runtime episodes spawning local processes, not the client connections. Run the high scenario on prime VM sandboxes and restore the 1024 start. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: hold the drain latch until the engines settle Cuts cancel episodes, so dispatcher inflight drops below the cap almost immediately while the engines still churn through their own backlog — preemption deltas from that stale churn re-triggered escalated cuts every poll (observed cascade: 1024 -> 8, one halving per 5s). Release the drain only when inflight is under the cap AND the poll shows no preemptions or queue overload. Also fix the hendrycks KV pin for 64-token blocks (32768 blocks gave 8.4M tokens, not 2.1M — the derived start clamped to the ceiling and triggered the storm in the first place). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * chore: say episodes, not units, in console logs Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Refund admission burst budget on episode completion The burst window capped gross admissions at max(group_size, cap/10) per 5s. For fast single-turn episodes the completion rate exceeds that admission rate, so inflight equilibrates at admission_rate x duration (observed: 24 at cap 58) and the cap never binds, locking out growth. Completions now refund the window budget: the window meters net pool growth, so ramps stay smoothed while steady-state replacement is never throttled. Also switch the hendrycks e2e configs to prime VM sandboxes and fix num_gpu_blocks_override for 16-token blocks (FA2 backend), restoring the intended 2.1M-token KV budget. * Fix four controller/dispatcher bugs from review - Growth was dead below cap 10: the permit is released before the completion hook fires, so the binding check read inflight <= cap - 1 and 0.9 * cap was unsatisfiable for small caps. Count the completing episode back in. - Queue cuts targeted 0.9 x engine running requests but the cap counts episodes; agentic episodes idle between turns, so this over-cut by the duty cycle. Cut in pool units like the preemption path. - Escalation latched forever: it only reset on a CLEAR-poll drain release, but steady state post-cut sits in the SOFT usage band, so every later cut halved. Reset via a grace window of polls after the drain instead. - Overload sheds count only live cancellations toward the excess; never-dispatched markers free no permits and made sheds undershoot. Also expire the growth gate 15s after the poll that set it: if the metrics path stalls (the very state runaway growth creates), growth freezes instead of compounding blind. And skip growth on error-marker completions (tokens <= 0) — they consumed no engine capacity, and a slow-failing error storm otherwise grows the cap against idle engines. * Drop cluster-specific node exclude from e2e configs * Budget hendrycks e2e concurrency for the sandbox gateway VM-sandbox runtimes hold one sandbox per in-flight episode, so peak concurrency across concurrent runs must stay under ~1k or the gateway 429s. Cap the three runs at 400/300/256 and shrink the KV budget to 786K tokens (12288 blocks x 16-token FA2 blocks x 4 engines) so the optimal band (~175) sits below every ceiling and each run still exercises its intended path (cut down / grow up / absorb eval waves). * Read KV capacity from kv_cache_size_tokens The num_gpu_blocks label in vllm:cache_config_info does not reflect num_gpu_blocks_override, so the derived capacity was 4x too high on runs that pin the block count. kv_cache_size_tokens reports the actual budget; keep the block product as a fallback for engines that do not expose it. * Adapt GLM-4.5-Air SWE example to a 4-node FP8 recipe 1 trainer node (sign-SGD full CPU offload, cp=4) + 3 FP8 inference replicas with online quantization. The concurrency section is removed: the controller runs on defaults, deriving its start from the measured KV budget. * Remove concurrency debug configs The hendrycks/glm-air matrix configs were development scaffolding for the e2e verification runs; the results live in the PR description. The GLM-4.5-Air SWE example is the maintained recipe. * Use fp8_per_block online quantization in the GLM-Air example Matches the research-prod GLM recipe: the bf16 checkpoint is loaded and quantized online, since NCCL weight broadcasts push bf16 trainer weights into the engines on every update. * Guard None raw tokens in logprobs detokenization tokenizer.convert_ids_to_tokens returns None for ids outside the tokenizer vocabulary. Models with embedding padding past the tokenizer (e.g. R1-Distill-Qwen) can sample such ids; _restore_leading_spaces then iterates None and the TypeError kills the AsyncLLM output loop and the whole API server with it. Observed as recurring random 'ApiServer died with exit code None' crashes under sustained sampling. * Revert "Guard None raw tokens in logprobs detokenization" This reverts commit 0c1b5d4. * Order concurrency fields initial, min, max * Route GLM-Air FP8 through DeepGEMM kernels Without a block-wise FP8 GEMM backend the engine falls back to torch._scaled_mm, which asserts on block-scale shapes during engine init. Enable DeepGEMM and disable the FlashInfer block-scale GEMM, matching the research-prod GLM recipe. * Serve GLM-Air FP8 checkpoint with kernel-format weight transfer The online fp8_per_block path crashes for glm4_moe on this stack (the block-scale GEMM falls back to torch._scaled_mm and asserts), so the example serves the native FP8 checkpoint instead. Weight updates then need trainer-side quantization: implement convert_layer_to_vllm_kernel for glm4_moe (standard GQA qkv fusion; the MoE handling mirrors GLM-MoE-DSA, whose MoE layout is identical) and enable quantize_in_weight_transfer. The kernel copy path has no TP shard handling, so the engines run EP experts + DP attention (data_parallel_size=8), matching the GLM prod deployment shape. Trainer keeps the bf16 checkpoint; the orchestrator addresses the served FP8 name. * Disable MoE DP chunking for the FP8 GLM-Air engines The cutlass FP8 MoE DP-chunking path under-sizes its permute scratch for large profile/prefill batches (moe_permute asserts n_token <= max_num_tokens), killing the engine at init. Same mitigation as the GLM prod recipe. * Serve GLM-Air FP8 via per-tensor online quantization Both block-scale FP8 attempts crash for glm4_moe on this stack (the blockscale GEMM asserts at engine init; the cutlass MoE DP-chunk scratch under-sizes). Per-tensor online quant uses the standard scaled_mm and fused-MoE FP8 kernels, keeps plain TP=8, and re-uses the same quantize-on-load path for weight broadcasts. * fix: stop metrics collector before finalizing monitors The collector's poll loop outlived monitors.finalize() on clean exit, so its next wandb.log hit the finished run and surfaced as a spurious 'Inference metrics poll failed: UsageError' warning. Stop it first; the teardown's second stop() is a no-op. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Bump verifiers to a3188c06 Rollout-abort cleanup hardening: shielded sandbox create, strongly referenced cancel tasks, and cancel drain on client close. * Fix four review findings; drop unused glm4_moe kernel converter - Overload signals read decode engines only: in P/D deployments a prefill queue or prefill preemption is normal flow and must not trigger cuts. - Batch/group divisibility is checked after the batch-size default lands, so a config setting only group_size cannot pass with a non-divisible default. - A group's age for youngest-first shedding is its oldest member's start; taking the newest made long-running groups look young and cancelled the most sunk cost. - Admission-window refunds apply only to natural completions: refunding cancelled episodes handed a mass shed's worth of burst budget to the refill while the overload was still draining. The glm4_moe kernel-format converter served the abandoned quantize_in_weight_transfer experiment; the example now uses plain online quantization, which needs no trainer-side support. * Explain adaptive concurrency by intuition in the docs Drop thresholds, constants, and the config listing from the docs section; describe grow/trim/cut as probe-and-back-off against the engines. Bump verifiers for the cancel-path race fixes. * Drop unrelated changes from the branch The detokenizer patch ships separately in #3304; the pydantic-config pointer bump was an accidental sweep of local submodule state. * Address review: config defaults and small cleanups - max_inflight defaults to 1024: an unbounded default made the growth path's failure modes existential; None still removes the ceiling. - min_inflight defaults to 1 instead of None-meaning-1, simplifying the bounds validator. - Signal is a Literal instead of an Enum. - Reword the admission-smoothing comment. * Raise max_inflight in examples exceeding the new default ceiling * a few minor tweaks * Bump verifiers to 6bcedc76 (merge main into rollout-abort) * Update lockfile for verifiers 0.3.1.dev57 * chore: cap CI RL runs at 512 inflight Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix: dedupe concurrency block in wiki-search CI config Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Bump verifiers to main (rollout-abort merged) feat/rollout-abort landed as PrimeIntellect-ai/verifiers#2396; the submodule now tracks verifiers main. --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
tokenizer.convert_ids_to_tokensreturnsNonefor token ids outside the tokenizer vocabulary. Models whose embedding matrix is padded past the tokenizer (e.g.R1-Distill-Qwen) can sample such ids.convert_ids_list_to_tokensguards the decoded string (or "") but not the raw piece:_restore_leading_spacesiteratesNone, and theTypeErrorkills the AsyncLLM output loop — the API server shuts down and every in-flight request on it fails.Noneguard viaapply_shared_vllm_patches(thevllm.general_pluginsentry point), returning the decoded string unchanged.Verification
Observed as recurring
Process ApiServer_N died with exit code Nonecrashes on three different nodes during sustained R1-Distill-Qwen-1.5B sampling (adaptive-concurrency e2e runs), each preceded by this traceback ininference.log. The same workload is running under the patch; no recurrence so far (crashes previously recurred within tens of minutes).🤖 Generated with Claude Code