Skip to content

fix(interception): per-digest retry-replay cache - #2461

Draft
faresobeid wants to merge 1 commit into
mainfrom
fix/replay-cache-per-digest
Draft

fix(interception): per-digest retry-replay cache#2461
faresobeid wants to merge 1 commit into
mainfrom
fix/replay-cache-per-digest

Conversation

@faresobeid

@faresobeid faresobeid commented Aug 29, 2026

Copy link
Copy Markdown
Collaborator

The single last_request/last_response slot assumes one outstanding model request per rollout session. A harness running concurrent conversation branches (e.g. rho's new parallel() subagent fan-out) breaks that assumption: completing any turn evicts every other branch's replay entry, so a marked SDK retry that should replay re-samples instead and forks the graph.

This keys the cache by request-body digest (OrderedDict, bounded at 32, FIFO eviction), preserving the exact replay-on-marked-retry and fresh-attempt-supersedes semantics per body while making them immune to interleaving. The streaming path is untouched (it never wrote the slots). tests/v1 green locally (e2e skipped, needs PRIME_API_KEY).

🤖 Generated with Claude Code

Note

Replace single-entry retry replay with per-digest cache in InterceptionServer

  • Replaces RolloutSession.last_request and last_response with an OrderedDict cache bounded to REPLAY_CACHE_SIZE (32). Marked SDK retries now replay the recorded response for up to 32 distinct request bodies per session.
  • A fresh attempt with a matching request body clears its prior replay entry.
  • Risk: Removes RolloutSession.last_request and RolloutSession.last_response. Code accessing these must use RolloutSession.replays.

Macroscope summarized 3bd913d.

The single last_request/last_response slot assumed one outstanding model request
per rollout session: with concurrent conversation branches (a harness fanning out
subagents), completing any turn evicted every other branch's replay entry, so a
marked SDK retry re-sampled and forked the graph. Key the cache by request digest
(bounded, FIFO eviction) — same replay and supersede semantics per body, immune
to interleaving.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants