fix(interception): per-digest retry-replay cache - #2461
Draft
faresobeid wants to merge 1 commit into
Draft
Conversation
The single last_request/last_response slot assumed one outstanding model request per rollout session: with concurrent conversation branches (a harness fanning out subagents), completing any turn evicted every other branch's replay entry, so a marked SDK retry re-sampled and forked the graph. Key the cache by request digest (bounded, FIFO eviction) — same replay and supersede semantics per body, immune to interleaving. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The single
last_request/last_responseslot assumes one outstanding model request per rollout session. A harness running concurrent conversation branches (e.g. rho's newparallel()subagent fan-out) breaks that assumption: completing any turn evicts every other branch's replay entry, so a marked SDK retry that should replay re-samples instead and forks the graph.This keys the cache by request-body digest (
OrderedDict, bounded at 32, FIFO eviction), preserving the exact replay-on-marked-retry and fresh-attempt-supersedes semantics per body while making them immune to interleaving. The streaming path is untouched (it never wrote the slots).tests/v1green locally (e2e skipped, needs PRIME_API_KEY).🤖 Generated with Claude Code
Note
Replace single-entry retry replay with per-digest cache in
InterceptionServerRolloutSession.last_requestandlast_responsewith anOrderedDictcache bounded toREPLAY_CACHE_SIZE(32). Marked SDK retries now replay the recorded response for up to 32 distinct request bodies per session.RolloutSession.last_requestandRolloutSession.last_response. Code accessing these must useRolloutSession.replays.Macroscope summarized 3bd913d.