feat: add bash context compaction by mikasenghaas · Pull Request #2448 · PrimeIntellect-ai/verifiers

mikasenghaas · 2026-08-26T19:23:16Z

Summary

add the same optional CompactionConfig to Bash and RLM harnesses
use an explicit threshold or 90% of an advertised model context window
compact and retry once after context-bound decoding or an overlong request
replace only the recent tool outputs needed to make a checkpoint fit
return OverlongPromptError to the harness without stopping the trace
pin the RLM harness to nano-rlm 1e45450
keep context detection and retry policy outside the main Bash loop

Supersedes #1983 with automatic context-limit detection and the current Bash harness API.
Companion: nano-rlm #147.

Breaking

Interception no longer stops a trace on OverlongPromptError. Harnesses must handle the returned context_length error or apply their own rollout limit.
RLMHarnessConfig.summarize_at_tokens moves to RLMHarnessConfig.compaction.summarize_at_tokens.
Leave compaction unset to disable proactive and reactive compaction.

Verification

Before this change, interception stopped the trace on an overlong prompt. After this change, the harness receives the provider error and can recover.

uv run pytest -q — passed; live E2E tests skipped without PRIME_API_KEY
uv run ruff check .
uv run ruff format --check .
nano-rlm: uv run pytest -q — 136 passed

The one-off taskset and configs lived under /tmp/vf-context-compaction. They are not part of this PR. The model server started with:

CUDA_VISIBLE_DEVICES=0 uv run inference --vllm.model Qwen/Qwen3-4B-Instruct-2507 --vllm.max-model-len 4096 --vllm.gpu-memory-utilization 0.25 --vllm.enforce-eager --server.port 8010 --backend-port 8110

Each case used this command, with the named case config:

VLLM_API_KEY=EMPTY PYTHONPATH=/tmp/vf-context-compaction uv run eval @ /tmp/vf-context-compaction/base.toml @ /tmp/vf-context-compaction/<case>.toml --no-serve --no-rich --no-push --clean

Case config	Expected behavior	Verified behavior
`bash-decode.toml`	Detect a context-bound decode, compact, and continue.	Passed with reward `1.0`; the first call ended at 4,096 tokens and the trace opened a compacted branch.
`bash-tool-result.toml`	Pass the overlong checkpoint error to Bash, drop the tool result, compact, and continue.	Passed with reward `1.0`; the trace recorded `OverlongPromptError` and the dropped-output marker.
`rlm-decode.toml`	Apply the same decode recovery in Nano RLM.	Passed with reward `1.0` and `num_compactions=1`.
`rlm-tool-result.toml`	Apply the same overlong tool-result recovery in Nano RLM.	Passed with reward `1.0`, recorded `OverlongPromptError`, and `num_compactions=2`.

Note

Add context compaction to Bash harness program

Adds compaction field to BashHarnessConfig (default True), which passes --compaction to the underlying program
The program detects context-length errors from the API, progressively drops latest tool outputs, summarizes the conversation, rebuilds the message history, and resumes the run loop
InterceptionServer checks X-Verifiers-Context-Compaction=1 on model calls. If present and the prompt is overlong, it returns a 400 context_length error without stopping the rollout, allowing the client to compact and retry
Risk: BashHarnessConfig.compaction defaults to True, changing default behavior for all Bash harnesses to attempt compaction instead of terminating on overlong prompts

^{Macroscope summarized cf0ad84. (Automatic summaries will resume when PR exits draft mode or review begins).}

Keep long Bash harness runs alive when the provider rejects an overlong\nprompt. The harness requests a nano-rlm-style checkpoint summary and\ncontinues from a fresh branch.\n\nMake compaction the default while preserving clean context-length\ntruncation when users disable it.

mikasenghaas · 2026-08-26T19:25:42Z

 MCP_TIMEOUT = 600.0
+CONTEXT_COMPACTION_HEADER = "X-Verifiers-Context-Compaction"
+
+CHECKPOINT_COMPACTION_PROMPT = (


use """ """ here

mikasenghaas · 2026-08-26T19:25:46Z

+    "seamlessly continue the work."
+)
+
+POST_COMPACTION_FRAMING = (


mikasenghaas · 2026-08-26T19:26:20Z

+)
+
+COMPACTED_TOOL_RESULT = (
+    "[tool output dropped because it exceeded the model context limit]"


just say "context limit" here

mikasenghaas · 2026-08-26T21:09:03Z

+        else:
+            prompt = (
+                "Call the `overflow_context` tool exactly once, then answer `recovered`. "
+                "In an RLM IPython session, call it with "


dont mention rlm here

bc we use this test for bash as well no?

mikasenghaas · 2026-08-26T21:41:27Z

drop the context_

mikasenghaas · 2026-08-26T21:42:40Z

+    base_url = os.environ.get("VF_COMPACTION_E2E_BASE_URL")
+    model = os.environ.get("VF_COMPACTION_E2E_MODEL")
+    context_window = int(os.environ.get("VF_COMPACTION_E2E_CONTEXT_WINDOW", "4096"))


hmm no env vars wtf

mikasenghaas · 2026-08-26T21:57:45Z

wtf is this?

mikasenghaas · 2026-08-26T21:58:15Z

 from verifiers.v1.types import ID


+class CompactionConfig(BaseConfig):


duplicate this in bash and rlm, not in top-level config

mikasenghaas · 2026-08-26T21:59:15Z

@@ -715,14 +715,10 @@ async def sample() -> web.Response:
                            status=400,
                        )
                except OverlongPromptError as e:


why even catch this at all? isn't this doing rewriting of sorts?return web.json_response( dialect.error_body("context_length"), status=400, )

Review: compaction is a per-loop policy, so each in-house harness owns its CompactionConfig and threshold discovery instead of sharing a top-level config class and a clients module. The duplicated resolver caches per (base_url, model) and queries the upstream /models card host-side, since the interception server serves no /models route for agent-side discovery. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review: interception rewrote overlong prompt failures into a synthesized "context_length" body. Let them fall through to the generic RolloutError relay instead: the harness receives the provider's original message (from which it can learn the context window) with the error's 400 status. A rollout whose final call overflowed without recovery now fails like any other provider error; "context_length" is gone as a stop condition, so drop it from Trace.is_truncated. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mikasenghaas · 2026-08-26T22:18:34Z

+        return value
+
+
+async def resolve_compaction_threshold(ctx: ModelContext) -> int | None:


wait this should not be vf side... but harness side

mikasenghaas · 2026-08-26T22:19:00Z

@@ -43,10 +43,10 @@ def __init__(self, message: str = "", *, status_code: int = 502) -> None:


 class OverlongPromptError(ProviderError):


do we still use this? if not, remove?

Review: threshold discovery belongs to the agent loops, not the vf host. The bash program reads the provider's /models card itself (like nano-rlm's engine already does) and the vf harnesses only forward the explicitly configured threshold. The RLM policy crosses ACP flat (compaction toggle + summarize_at_tokens) to match nano-rlm's flattened ExecutionPolicy; pin bumped to f452d52. Behind the interception server (which serves no /models route) discovery yields nothing and both loops fall back to reactive compaction, learning the threshold from the relayed provider error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review: nothing catches or type-checks OverlongPromptError since interception relays it like any other provider failure. Raise a plain ProviderError with a deterministic 400 at the two sites that produced it (renderer pre-flight overflow, Responses context_length_exceeded) and drop the phrase-sniffing from model_error. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Add GET /v1/models to the interception server, relaying the upstream listing through the session's shared client (cached per endpoint, one upstream fetch per process). Agent loops can now discover their compaction threshold in training and eval alike. The path is dialect-universal: OpenAI and Anthropic SDKs both list models at /v1/models, so one route serves every dialect and only the auth carrier differs (tried per dialect). The response schema stays the upstream's; only OpenAI-compatible engines advertise a context window (vLLM's max_model_len), others fall back to reactive compaction. Also parse the listing as a parameterized mapping in the bash program - the OpenAI SDK cannot construct a bare dict, so discovery raised ValueError instead of returning None. Pin nano-rlm da9f4a4 with the same fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

summarize_at_tokens is a plain token count now - the (lo, hi) task-seeded draw is no longer used. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Review: the models listing is a proxy concern, not a client one. handle_models relays GET /v1/models straight from the session's endpoint config (the matched dialect contributes bearer extraction and provider auth), the upstream body and status pass through verbatim, and the Client.models method and its per-endpoint cache go away - every request hits the provider, like any other relay. Also order compaction last in BashHarnessConfig. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The model can ignore tool_choice="none" on the checkpoint turn and reply with a tool call and no text (observed on ~6% of compactions with deepseek-v4-flash on Terminal-Bench 2) - the rebuilt branch then starts with no context on the work done so far. Resample the checkpoint up to three times until it yields a text summary; pin nano-rlm ac8fdb0 with the same fix. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

The failed checkpoints were not disobedience: the model understood the request and chose to run one more state-gathering tool call before summarizing, which the loop never grants it. Say explicitly that the summary must come from the conversation as it stands; pin nano-rlm b1b4140 with the same wording. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

One checkpoint call per compaction - the hardened prompt is the guard against tool-call replies. Also name estimated_tokens' input for what it counts. Pin nano-rlm f5c14aa with the same changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mikasenghaas · 2026-08-27T17:44:49Z

Superseded by the stack #2453 (relay overlong prompt errors) + #2454 (context compaction), split for review.

## Summary - add an optional `CompactionConfig` to the bash harness: compact into a handoff summary at `summarize_at_tokens`, or when 16k tokens remain below the model context window - the program discovers the window itself from the provider's `/models` card (via `models.list()`), so the interception server gains a stateless `GET /v1/models` relay (one route serves every dialect) - truncate a tool result over 20KB to its head and tail with a warning naming the original size - compact reactively on an attributed 400/413 overflow error, from the current state; a rejected checkpoint falls back to the last state that passed a threshold check (which holds a full reserve of room by definition), and an empty or tool-calling reply is resampled - after three failed checkpoint attempts the program ends the run cleanly — still a trainable sample - an overflow with no history beyond the task propagates: nothing to reclaim The RLM harness gets the same policy in #2459, stacked on this PR so this one can merge before nano-rlm does. Together with #2453 (merged) and #2459 this supersedes #2448. ## Breaking - None on its own: `compaction` is a new optional field, unset by default. ## Verification - `uv run pytest -q tests/v1` — passed; live E2E tests skipped without `PRIME_API_KEY` - Tool truncation checked by exact match: a synthetic 55KB output produces the byte-identical expected head/tail + warning header; short outputs pass through untouched. Terminal-Bench 2 e2e: 8 tasks, local vLLM `poolside/Laguna-XS-2.1` at 32k (glm45 reasoning + glm47 tool parsers), `compaction = {}` so the threshold auto-discovers to `32768 − 16384 = 16384`. Trace analysis: - 8/8 rollouts end `ok=true` with scoring run (1 solved); no harness errors, no truncation stops. - The threshold triggers exactly where designed: 7/8 episodes compacted with peak contexts of 17.0k-23.4k tokens; the one episode that stayed at 15.4k never compacted. - 20KB tool truncation visible in 3 episodes' traces (`Warning: truncated output` with original size). - Laguna puts its entire checkpoint reply in the reasoning channel, so under the summaries-are-content-only rule every compaction on this model exercises the failure path end to end: three resampled checkpoint attempts, then the program ends the run cleanly - the rollout stays a trainable sample. Summary carry-over itself was demonstrated on models that answer in the content channel (Qwen3-0.6B at 4k/8k, deepseek-v4-flash on tb2): every non-final branch ends with the checkpoint prompt followed by the summary, and the next branch opens with `[system, framed summary]`. - A final combined verification run on a content-channel model is pending. 🤖 Generated with [Claude Code](https://claude.com/claude-code)  --- > [!NOTE] > **Medium Risk** > The shared agent loop now permanently truncates large tool outputs and, when compaction is enabled, replaces long histories with model-generated summaries—both can change task outcomes; the new models relay is ancillary and failures only disable auto-thresholds. > > **Overview** > Adds an **opt-in** context compaction path for the bash agent: `BashHarnessConfig` gains `CompactionConfig` (`summarize_at_tokens` optional), which forwards `--compaction` / `--summarize-at-tokens` into the shared minimal/bash chat program. > > When compaction is on, the program uses a new **`Compactor`** to compact before context blows up: it can trigger on usage crossing a threshold (explicit token limit, or auto from the model card with a 16k reserve), on `finish_reason=length`, on post-tool estimated token growth, or reactively on provider 400/413 overflow. Compaction asks the model for a plain-text handoff summary (`tool_choice=none`), rebuilds history as system + framed summary user message, falls back to the last “good” checkpoint on overflow, and **ends the loop cleanly** on `CompactionFailed` instead of crashing. Threshold discovery calls `models.list()` against the intercepted base URL. > > **Tool results are always middle-truncated** at ~20KB (head/tail + warning) via `bound_tool_message` before they enter the transcript, including interception rewrites. > > The interception server adds **`GET /v1/models`** as an authenticated upstream relay (30s timeout) so the in-container SDK can read provider context-window fields without recording a model turn. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit e1c80ac. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>     > [!NOTE] > ### Add tool truncation and auto-compaction to `BashHarness` agent loop > - Adds `CompactionConfig` to `BashHarnessConfig` and passes `--compaction` and `--summarize-at-tokens` flags to the spawned program in [harness.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2454/files#diff-1544e2ee3426241d8c97940a3e8daf25b2eeeeb95f8abb27e88bed4af420e609) > - Introduces the `Compactor` class in [program.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2454/files#diff-cfa9a2dff501160af71f3c07beb4c6ca40926e64e93b43f217548811dcc78a36) that detects context exhaustion, builds a summary checkpoint, and rebuilds the conversation; auto-discovers a token threshold from the provider's `/v1/models` endpoint when none is given > - Truncates large tool outputs to `TOOL_OUTPUT_MAX_BYTES` (20,000) with head/tail preservation via `truncate_tool_output` before they enter the conversation > - Proxies `GET /v1/models` through the interception server in [server.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2454/files#diff-84e2f8d027d609e8760ef6d533535ec9d99dfc6384d5ddb6111b4001b2010f35) so the program can query the provider for context window size > - `chat()` now returns the full completion object instead of the first message, and the main loop ends cleanly on `CompactionFailed` > - Risk: `chat()` signature and return type changed in [program.py](https://github.com/PrimeIntellect-ai/verifiers/pull/2454/files#diff-cfa9a2dff501160af71f3c07beb4c6ca40926e64e93b43f217548811dcc78a36); any callers expecting the first message instead of the full completion will break. Tool outputs over 20,000 bytes are silently truncated mid-conversation. > >  > > <sup><a href="https://app.macroscope.com">Macroscope</a> summarized 27e1c91.</sup> >  >  --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

mikasenghaas added 2 commits August 26, 2026 19:22

recover from context overflow

b60b651

mikasenghaas mentioned this pull request Aug 26, 2026

feat: tool truncation + auto-compaction PrimeIntellect-ai/nano-rlm#147

Merged

mikasenghaas commented Aug 26, 2026

View reviewed changes

mikasenghaas changed the title ~~feat: add Bash context compaction~~ feat: add bash context compaction Aug 26, 2026

mikasenghaas added 2 commits August 26, 2026 21:54

simplify compaction flow

c1a13ee

propagate context limit errors

4bb99de

mikasenghaas commented Aug 26, 2026

View reviewed changes

mikasenghaas and others added 2 commits August 26, 2026 22:12

mikasenghaas commented Aug 26, 2026

View reviewed changes

mikasenghaas and others added 8 commits August 26, 2026 22:27

drop ranged compaction thresholds

5b48a26

summarize_at_tokens is a plain token count now - the (lo, hi) task-seeded draw is no longer used. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

simplify checkpoint to one attempt

daec369

One checkpoint call per compaction - the hardened prompt is the guard against tool-call replies. Also name estimated_tokens' input for what it counts. Pin nano-rlm f5c14aa with the same changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

mikasenghaas mentioned this pull request Aug 27, 2026

feat: tool truncation + auto-compaction to bash harness #2454

Merged

mikasenghaas closed this Aug 27, 2026

Navigation Menu

Uh oh!

feat: add bash context compaction - #2448

feat: add bash context compaction#2448
mikasenghaas wants to merge 14 commits into
mainfrom
feat/bash-compaction

mikasenghaas commented Aug 26, 2026 •

edited by macroscopeapp Bot

Loading

Uh oh!

mikasenghaas Aug 26, 2026

Uh oh!

mikasenghaas Aug 26, 2026

Uh oh!

mikasenghaas Aug 26, 2026

Uh oh!

mikasenghaas Aug 26, 2026

Uh oh!

mikasenghaas Aug 26, 2026

Uh oh!

mikasenghaas Aug 26, 2026

Uh oh!

mikasenghaas Aug 26, 2026

Uh oh!

mikasenghaas Aug 26, 2026

Uh oh!

mikasenghaas Aug 26, 2026

Uh oh!

mikasenghaas Aug 26, 2026

Uh oh!

mikasenghaas Aug 26, 2026

Uh oh!

mikasenghaas Aug 26, 2026

Uh oh!

mikasenghaas commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		from verifiers.v1.types import ID


		class CompactionConfig(BaseConfig):

		return value


		async def resolve_compaction_threshold(ctx: ModelContext) -> int \| None:

		@@ -43,10 +43,10 @@ def __init__(self, message: str = "", *, status_code: int = 502) -> None:


		class OverlongPromptError(ProviderError):

Uh oh!

Conversation

mikasenghaas commented Aug 26, 2026 • edited by macroscopeapp Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Breaking

Verification

Add context compaction to Bash harness program

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mikasenghaas commented Aug 27, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

mikasenghaas commented Aug 26, 2026 •

edited by macroscopeapp Bot

Loading