Skip to content

fix(responses): add aggregate memory budget - #1179

Draft
eoinfennessy wants to merge 9 commits into
praxis-proxy:mainfrom
eoinfennessy:openai-agentic-loop-retention
Draft

eoinfennessy wants to merge 9 commits into
praxis-proxy:mainfrom
eoinfennessy:openai-agentic-loop-retention

Conversation

@eoinfennessy

@eoinfennessy eoinfennessy commented Sep 15, 2026

Copy link
Copy Markdown
Member

Closes #953

Summary

Add request-wide retained-payload accounting for OpenAI Responses agentic executions. This bounds memory retained across loop state, response history, SSE parsing, tool dispatch, response-store persistence, and terminal serialization while preserving canonical ownership of response data.

Changes

  • Add openai_agentic_loop.max_retained_bytes:
    • Default: 64 MiB
    • Valid range: 4 KiB–256 MiB
    • The smallest configured value wins across loop instances.
  • Enforce max_infer_iters against Praxis’ global iteration ceiling.
  • Add aggregate payload accounting to ResponsesState, including independently owned JSON values, strings, buffers, parser state, approval records, and sibling-filter state.
  • Replace cloned dispatcher payloads with indexed assignments and borrowed views where possible.
  • Make response finalization transactional:
    • Initial overflow returns HTTP 413.
    • Buffered continuation overflow returns HTTP 502.
    • Already-committed streams emit one SSE error event without response.completed or [DONE].
    • Failed requests skip persistence and release retained payload state.
  • Bound transient SSE parser/event/frame ownership, tool-call argument accumulation, local lifecycle synthesis, terminal canonicalization, usage merging, and citation annotation staging.
  • Coordinate the loop owner and dispatchers so request-side mutations occur only after all budget admissions and each provider response is processed once.
  • Apply aggregate budgeting to:
    • File-search response decoding, model-context formatting, public results, citations, and in-place output reconciliation.
    • MCP result batches, approval resumption, deferred discovery, and approval output generation.
    • Web-search response bodies, result formatting, bridge messages, and indexed output updates.
  • Add response-store payload accounting and size-only pending-approval queries for SQLite and PostgreSQL. Oversized approval resumptions remain retryable and are not consumed.
  • Preserve ordered built-in max_tool_calls accounting across web search and file search while keeping MCP on its independent limit.
  • Document the new configuration and add an overflow example configuration.

Tests

  • Expanded unit coverage for retained-payload accounting, ownership transitions, streaming overflow, terminal construction, file search, web search, MCP approvals, persistence, citations, and usage merging.
  • Added OpenAI SDK integration coverage for buffered and streaming requests using the explicit retained-payload budget.
  • Added a synthetic inference fixture proving that committed-stream overflow emits exactly response.created plus error, suppresses [DONE], and does not persist the response.
  • Updated inference fixture coverage and generated documentation.

mkoushni added a commit to mkoushni/ai-fork that referenced this pull request Sep 16, 2026
Mixing git-pinned core/filter with crates.io protocol/tls pulled two
praxis_core copies into the graph and broke server health-check wiring.
Pin the full praxis stack to the same rev until praxis-proxy#1179 ships on crates.io.

Signed-off-by: mkoushni <mkoushni@redhat.com>
…-retention

# Conflicts:
#	apis/src/openai/responses/agentic_loop/mod.rs
#	apis/src/openai/responses/mcp_dispatch/mod.rs
#	apis/src/openai/responses/mcp_dispatch/tests.rs
#	apis/src/openai/responses/store/filter.rs
Signed-off-by: Eoin Fennessy <efenness@redhat.com>
Signed-off-by: Eoin Fennessy <efenness@redhat.com>
Signed-off-by: Eoin Fennessy <efenness@redhat.com>
Signed-off-by: Eoin Fennessy <efenness@redhat.com>
szedan-rh pushed a commit to szedan-rh/ai that referenced this pull request Sep 17, 2026
…is-proxy#1030)

* fix(stream-events): enforce timeout_secs after a silent backend

timeout_secs was only checked on incoming SSE chunks and EOS. Cap the
upstream read timeout and pair examples with cluster read_timeout_ms so
an idle connection after the first event is torn down.

Closes praxis-proxy#938

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(stream-events): keep completed streams persistable after idle close

An idle or deadline abort after a terminal SSE event must not set
skip_persist; the parser already observed a completed lifecycle.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(tests): keep agentic-loop read_timeout unique and shrink chunked helper

The example now ships cluster read_timeout_ms, so the vLLM harness must
not inject a second copy. Split the chunked backend writer to satisfy
clippy's line limit.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(stream-events): skip idle-timeout errors after a terminal SSE event

Only publish stream_error_code and skip_persist when the parser is still
Open, so a slow HTTP close after response.completed does not suppress
persistence. mark_stream_termination_handled stays unconditional.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(stream-events): keep idle-timeout tests valid after IRR wrap

The idle-backend example test injected timeout_secs at pre-IRR indent
and failed YAML parse. Treat peer read_timeout as Io so IRR does not
discard the stream, and isolate OpenResponses streaming-response from
parallel CPU load.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(tests): allow logical SSE error after a committed stream

late_upstream_failure_does_not_replace_committed_sse forbade any
"error" substring. After Io terminations are marked handled, the
logical stream appends an SSE error terminator instead of replacing
the 200 event-stream response.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(stream-events): apply timeout_secs after load balancing

IRR runs the buffered request-body phase before load_balancer, so
capping ctx.upstream earlier was a no-op. Place the documented filter
after load_balancer, cap the remaining budget on the selected peer, and
treat Io as a timeout only after the first SSE chunk exhausts
timeout_secs.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(docs): sync stream_events body-access table

Removing the unused request-body hook made the generated Responses
pipeline README stale, which failed lint.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* chore(deps): bump rustls to 0.23.45

cargo audit fails on RUSTSEC-2026-0285; 0.23.45 rejects TLS 1.3
handshake messages sent at the wrong encryption level.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(stream-events): recap leftover timeout on restored IRR peer

IRR reconstructs response-body contexts with upstream: None after
snapshotting the original read_timeout into the live body. Remember
the selected peer at arm time and restore it before remaining-budget
recaps so a chunk near the deadline cannot restart the full per-read
timer.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* test(openresponses): serialize live templates to avoid CPU starvation

The suite runs --filter ids via Promise.all. Concurrent Qwen CPU
generations starved basic-response until the 300s client timeout.
Run each template alone so buffered and streaming cases no longer
contend for the same backend.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* docs(stream-events): leftover recap is copied onto the live body

Restoring the selected peer publishes leftover timeout_secs on
ctx.upstream. The streaming executor copies that cap onto the live
SubResponseBody; mutating the restored peer alone would leave the
original per-read timer running.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* ci(integration): harden pinned Codex CLI download

GitHub release assets intermittently return HTTP 500 during curl
fetch. Retry transient failures longer, verify the archive before
caching it, and fall back to gh release download when curl fails.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(stream_events): start timeout at first chunk and recap live body

Do not cap upstream read_timeout at arm time; timeout_secs is an
absolute deadline from the first SSE chunk only. Each chunk publishes
leftover budget through HttpFilterContext::cap_stream_read_timeout so
Praxis can copy it onto the live SubResponseBody instead of restarting
the dispatch snapshot (requires praxis-proxy/praxis#1172).

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(stream_events): pin Praxis 1172 and close timeout recap gaps

Git-pin praxis-proxy/praxis#1172 so cap_stream_read_timeout is available
on CI; crates.io 0.5.5 predates that merge. Recap leftover budget on
every chunk after the first SSE event (including zero at the deadline),
align check_timeout with io_exceeded_stream_deadline, and fix the idle
integration test comment syntax.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* Fix broken rustdoc link for cap_stream_read_timeout

Use the fully qualified praxis_filter path so config.rs doc builds with -D warnings.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* Regenerate openai_stream_events filter docs

Sync generated docs with the fully qualified cap_stream_read_timeout link.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(stream_events): cap absolute deadline on live body after first chunk

Relative read-timeout recaps restart when downstream backpressure delays
the next upstream poll, so timeout_secs could overrun. Publish an absolute
cutoff through cap_stream_deadline instead. Pin mkoushni/praxis at 21be732
until the Praxis change lands upstream.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(ci): allow temporary mkoushni/praxis git pin in deny.toml

dependency-check rejects git sources not listed in allow-git. The
cap_stream_deadline fork pin needs the same temporary exemption as the
earlier praxis-proxy/praxis git deps.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(deps): pin praxis core/filter via upstream git, not a personal fork

Use praxis-proxy/praxis for the temporary cap_stream_deadline git pin and
restore protocol/tls/praxis to crates.io 0.5.5 like main. Drop the fork
allow-list entry from deny.toml.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(deps): git-pin all praxis crates to avoid duplicate core types

Mixing git-pinned core/filter with crates.io protocol/tls pulled two
praxis_core copies into the graph and broke server health-check wiring.
Pin the full praxis stack to the same rev until praxis-proxy#1179 ships on crates.io.

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(deps): pin Praxis to v0.5.6

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(tests): import json_post for stream events

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(stream-events): adapt timeout tests to Praxis 0.5.6

Signed-off-by: mkoushni <mkoushni@redhat.com>

* docs: regenerate filter documentation

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(xtask): make filter doc discovery deterministic

Signed-off-by: mkoushni <mkoushni@redhat.com>

* fix(docs): use resolved Praxis source for filter docs

Signed-off-by: mkoushni <mkoushni@redhat.com>

---------

Signed-off-by: mkoushni <mkoushni@redhat.com>
Co-authored-by: Sébastien Han <seb@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Agentic output accumulation has no aggregate memory budget

1 participant