Problem
Streaming responses are fully buffered before relay (documented in the README limits): the client's time-to-first-token becomes the whole upstream response time. Correct for guarding, but it degrades interactive streaming UX.
Proposal
State-machine relay instead of buffer-everything:
- Pass text deltas through immediately.
- Hold tool-call deltas (and everything after the first one) until the reassembled call is judged; then release or replace with an error event.
Notes
- The can't-un-send tension is real: once bytes are relayed they can't be recalled, so the hold-point must begin at the first tool-call fragment, not at judgment time.
- Provider framing differs (OpenAI
delta.tool_calls, Anthropic content_block_*, Gemini per-chunk) — the reassembly logic in stream.rs already models this; the relay needs the same per-provider awareness.
- Fail-closed on malformed streams: garbled frames after a tool-call start → terminate, don't pass through.
Problem
Streaming responses are fully buffered before relay (documented in the README limits): the client's time-to-first-token becomes the whole upstream response time. Correct for guarding, but it degrades interactive streaming UX.
Proposal
State-machine relay instead of buffer-everything:
Notes
delta.tool_calls, Anthropiccontent_block_*, Gemini per-chunk) — the reassembly logic instream.rsalready models this; the relay needs the same per-provider awareness.