fix(responses): add aggregate memory budget - #1179
Draft
eoinfennessy wants to merge 9 commits into
Draft
eoinfennessy wants to merge 9 commits into
eoinfennessy wants to merge 9 commits into
Conversation
Signed-off-by: Eoin Fennessy <efenness@redhat.com>
Signed-off-by: Eoin Fennessy <efenness@redhat.com>
mkoushni
added a commit
to mkoushni/ai-fork
that referenced
this pull request
Sep 16, 2026
Mixing git-pinned core/filter with crates.io protocol/tls pulled two praxis_core copies into the graph and broke server health-check wiring. Pin the full praxis stack to the same rev until praxis-proxy#1179 ships on crates.io. Signed-off-by: mkoushni <mkoushni@redhat.com>
…-retention # Conflicts: # apis/src/openai/responses/agentic_loop/mod.rs # apis/src/openai/responses/mcp_dispatch/mod.rs # apis/src/openai/responses/mcp_dispatch/tests.rs # apis/src/openai/responses/store/filter.rs
Signed-off-by: Eoin Fennessy <efenness@redhat.com>
Signed-off-by: Eoin Fennessy <efenness@redhat.com>
Signed-off-by: Eoin Fennessy <efenness@redhat.com>
Signed-off-by: Eoin Fennessy <efenness@redhat.com>
szedan-rh
pushed a commit
to szedan-rh/ai
that referenced
this pull request
Sep 17, 2026
…is-proxy#1030) * fix(stream-events): enforce timeout_secs after a silent backend timeout_secs was only checked on incoming SSE chunks and EOS. Cap the upstream read timeout and pair examples with cluster read_timeout_ms so an idle connection after the first event is torn down. Closes praxis-proxy#938 Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(stream-events): keep completed streams persistable after idle close An idle or deadline abort after a terminal SSE event must not set skip_persist; the parser already observed a completed lifecycle. Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(tests): keep agentic-loop read_timeout unique and shrink chunked helper The example now ships cluster read_timeout_ms, so the vLLM harness must not inject a second copy. Split the chunked backend writer to satisfy clippy's line limit. Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(stream-events): skip idle-timeout errors after a terminal SSE event Only publish stream_error_code and skip_persist when the parser is still Open, so a slow HTTP close after response.completed does not suppress persistence. mark_stream_termination_handled stays unconditional. Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(stream-events): keep idle-timeout tests valid after IRR wrap The idle-backend example test injected timeout_secs at pre-IRR indent and failed YAML parse. Treat peer read_timeout as Io so IRR does not discard the stream, and isolate OpenResponses streaming-response from parallel CPU load. Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(tests): allow logical SSE error after a committed stream late_upstream_failure_does_not_replace_committed_sse forbade any "error" substring. After Io terminations are marked handled, the logical stream appends an SSE error terminator instead of replacing the 200 event-stream response. Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(stream-events): apply timeout_secs after load balancing IRR runs the buffered request-body phase before load_balancer, so capping ctx.upstream earlier was a no-op. Place the documented filter after load_balancer, cap the remaining budget on the selected peer, and treat Io as a timeout only after the first SSE chunk exhausts timeout_secs. Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(docs): sync stream_events body-access table Removing the unused request-body hook made the generated Responses pipeline README stale, which failed lint. Signed-off-by: mkoushni <mkoushni@redhat.com> * chore(deps): bump rustls to 0.23.45 cargo audit fails on RUSTSEC-2026-0285; 0.23.45 rejects TLS 1.3 handshake messages sent at the wrong encryption level. Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(stream-events): recap leftover timeout on restored IRR peer IRR reconstructs response-body contexts with upstream: None after snapshotting the original read_timeout into the live body. Remember the selected peer at arm time and restore it before remaining-budget recaps so a chunk near the deadline cannot restart the full per-read timer. Signed-off-by: mkoushni <mkoushni@redhat.com> * test(openresponses): serialize live templates to avoid CPU starvation The suite runs --filter ids via Promise.all. Concurrent Qwen CPU generations starved basic-response until the 300s client timeout. Run each template alone so buffered and streaming cases no longer contend for the same backend. Signed-off-by: mkoushni <mkoushni@redhat.com> * docs(stream-events): leftover recap is copied onto the live body Restoring the selected peer publishes leftover timeout_secs on ctx.upstream. The streaming executor copies that cap onto the live SubResponseBody; mutating the restored peer alone would leave the original per-read timer running. Signed-off-by: mkoushni <mkoushni@redhat.com> * ci(integration): harden pinned Codex CLI download GitHub release assets intermittently return HTTP 500 during curl fetch. Retry transient failures longer, verify the archive before caching it, and fall back to gh release download when curl fails. Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(stream_events): start timeout at first chunk and recap live body Do not cap upstream read_timeout at arm time; timeout_secs is an absolute deadline from the first SSE chunk only. Each chunk publishes leftover budget through HttpFilterContext::cap_stream_read_timeout so Praxis can copy it onto the live SubResponseBody instead of restarting the dispatch snapshot (requires praxis-proxy/praxis#1172). Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(stream_events): pin Praxis 1172 and close timeout recap gaps Git-pin praxis-proxy/praxis#1172 so cap_stream_read_timeout is available on CI; crates.io 0.5.5 predates that merge. Recap leftover budget on every chunk after the first SSE event (including zero at the deadline), align check_timeout with io_exceeded_stream_deadline, and fix the idle integration test comment syntax. Signed-off-by: mkoushni <mkoushni@redhat.com> * Fix broken rustdoc link for cap_stream_read_timeout Use the fully qualified praxis_filter path so config.rs doc builds with -D warnings. Signed-off-by: mkoushni <mkoushni@redhat.com> * Regenerate openai_stream_events filter docs Sync generated docs with the fully qualified cap_stream_read_timeout link. Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(stream_events): cap absolute deadline on live body after first chunk Relative read-timeout recaps restart when downstream backpressure delays the next upstream poll, so timeout_secs could overrun. Publish an absolute cutoff through cap_stream_deadline instead. Pin mkoushni/praxis at 21be732 until the Praxis change lands upstream. Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(ci): allow temporary mkoushni/praxis git pin in deny.toml dependency-check rejects git sources not listed in allow-git. The cap_stream_deadline fork pin needs the same temporary exemption as the earlier praxis-proxy/praxis git deps. Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(deps): pin praxis core/filter via upstream git, not a personal fork Use praxis-proxy/praxis for the temporary cap_stream_deadline git pin and restore protocol/tls/praxis to crates.io 0.5.5 like main. Drop the fork allow-list entry from deny.toml. Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(deps): git-pin all praxis crates to avoid duplicate core types Mixing git-pinned core/filter with crates.io protocol/tls pulled two praxis_core copies into the graph and broke server health-check wiring. Pin the full praxis stack to the same rev until praxis-proxy#1179 ships on crates.io. Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(deps): pin Praxis to v0.5.6 Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(tests): import json_post for stream events Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(stream-events): adapt timeout tests to Praxis 0.5.6 Signed-off-by: mkoushni <mkoushni@redhat.com> * docs: regenerate filter documentation Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(xtask): make filter doc discovery deterministic Signed-off-by: mkoushni <mkoushni@redhat.com> * fix(docs): use resolved Praxis source for filter docs Signed-off-by: mkoushni <mkoushni@redhat.com> --------- Signed-off-by: mkoushni <mkoushni@redhat.com> Co-authored-by: Sébastien Han <seb@redhat.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #953
Summary
Add request-wide retained-payload accounting for OpenAI Responses agentic executions. This bounds memory retained across loop state, response history, SSE parsing, tool dispatch, response-store persistence, and terminal serialization while preserving canonical ownership of response data.
Changes
openai_agentic_loop.max_retained_bytes:max_infer_itersagainst Praxis’ global iteration ceiling.ResponsesState, including independently owned JSON values, strings, buffers, parser state, approval records, and sibling-filter state.errorevent withoutresponse.completedor[DONE].max_tool_callsaccounting across web search and file search while keeping MCP on its independent limit.Tests
response.createdpluserror, suppresses[DONE], and does not persist the response.