Skip to content

fix(acp): wake quiet hosts for respawns and queued retries - #7459

Open
loganj wants to merge 1 commit into
mainfrom
fix/acp-quiet-retry-79d180db
Open

fix(acp): wake quiet hosts for respawns and queued retries#7459
loganj wants to merge 1 commit into
mainfrom
fix/acp-quiet-retry-79d180db

Conversation

@loganj

@loganj loganj commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator

Problem

With --no-presence --no-typing --heartbeat-interval 0, a crashed ACP transport can leave work queued indefinitely. Background respawn results were only drained before select!; neither their arrival nor queue retry eligibility woke the host. A respawn-only wake is insufficient: replacement initialization normally precedes the independent queue backoff deadline.

Change

  • Select on respawn completion, reusing the existing return-to-pool and dispatch paths.
  • Select on the earliest queued retry throttle, including already-expired deadlines, only while the pool is ready and has idle capacity.
  • Give the existing 30-second maintenance/refill check its own wake source. Circuit cooldown/refill policy is unchanged; it no longer depends on presence/typing traffic.
  • Disable closed respawn receivers without spinning or disabling timers; prioritize shutdown.
  • Keep retry backoff/budget, batching, timestamps, dedup, delivery state, and intentional host max-turn cap unchanged. Held/no-slot dispatch releases clear expired retry throttles through the existing mark_complete path.

No reconnect/gateway changes or adapter retry engine. Based directly on main, independent of other ACP plumbing work. Closest related open PR found: #7317 (usage-limit retry policy); this is a scheduling fix, not that policy change.

Validation

  • cargo test -p buzz-acp: 920 library tests + 9 integration tests pass.
  • Five new Rust tests exercise the production wake helper and queue boundary: respawn before/after retry eligibility, backoff preservation, one-shot redelivery, empty/removed/held/in-flight scopes, closed receivers, and shutdown cancellation.
  • cargo clippy -p buzz-acp --all-targets --all-features -- -D warnings, cargo fmt --all -- --check, just file-size-check, and git diff --check: pass.
  • Actual standalone stock host binary built from this tree + installed/generated Pi adapter fixture, pinned adapter source 801d23374a588fd1154bb3da8783f6dcc9084e15 and upstream pi-acp 0.0.33. Loopback-only synthetic relay/provider; no live agent/config changes.
    • All optional presence/typing/heartbeat wakes disabled; 2.5-second adapter inactivity budget, 45-second callback bound, host hard cap unchanged.
    • Fast replacement: initialization 1.347s after requeue, retry eligible at 4.447s. Automatic original-batch recovery, one saved worker terminal and one callback, five dispatch cycles.
    • Delayed replacement: initialization 9.118s after requeue, retry eligible at 4.157s. Same recovery invariants, four dispatch cycles.
    • Both use exactly one relay input; both settle within 2.84s of the later of replacement/eligibility (including a 1.5s duplicate-observation window). This also rejects waiting for the 30s maintenance tick.
    • Same fixture against freshly built unmodified main 3c7f288c60d67df78577b237e27c3dfc8831aaa1: expected failure at the 45-second callback bound.

Full workspace/Desktop/mobile just ci not run locally; no Tauri files touched. Relevant ACP package checks above are complete.

Review / integration gate

Draft for independent review. The fixture uses the pinned pre-repair adapter to isolate this host defect, not concurrent adapter shutdown-fencing changes. Composed testing with the final adapter candidate and independent review remain required before claiming the shared lifecycle fix complete. No activation, merge, direct main push, or incident replay performed. Existing all-agents-dead exit and max-turn safety policies remain unchanged.

Signed-off-by: Logan Johnson <loganj@squareup.com>
@github-actions

github-actions Bot commented Sep 7, 2026

Copy link
Copy Markdown

🔐 Codex Security Review

Note: This is an automated, security-focused review generated by Codex.
Use it as a supplement to human review; false positives are possible.

Scope

  • Exact PR diff: 3c7f288c60d67df78577b237e27c3dfc8831aaa1...e4b3cb9057b33554aae8a1769a17a34dd3eb96ce
  • Model: gpt-5.6-sol

💡 Click "edited" above to see earlier reviews for this PR.


Review Summary

Overall Risk: NONE

No concrete security, correctness, or reliability issues were found in the changed code. The retry, respawn, maintenance, shutdown, and lazy-pool paths preserve their existing invariants.

Findings

No concrete security, correctness, or reliability findings were identified.

Notes

  • No additional limitations were reported.

Generated by Codex Security Review |
Requested by: @loganj |
Workflow run

@loganj

loganj commented Sep 7, 2026

Copy link
Copy Markdown
Collaborator Author

@buzz-security-review e4b3cb9

@github-actions github-actions Bot added the codex-security-review-current The posted Codex security review matches its recorded range. label Sep 7, 2026
@loganj
loganj marked this pull request as ready for review September 8, 2026 02:23
@loganj
loganj requested a review from a team as a code owner September 8, 2026 02:23
@loganj

loganj commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator Author

Independent combined evidence (host + companion adapter) — exact heads, no source changes

This PR (host scheduling fix): independent host-only review of exact head e4b3cb9057b33554aae8a1769a17a34dd3eb96ce (base main @ 3c7f288c60d67df78577b237e27c3dfc8831aaa1) — APPROVED, no blocking defect in the changed quiet-host respawn / queued-retry wake paths. 5 independently reran focused checks (recovery-wake + retry-deadline seams) plus reused verified package receipts: 920 library + 9 integration passed, zero failures (clippy/fmt/size receipts and all manifest hashes independently recomputed).

Companion adapter PR (pic-suite): buzz://pr?id=7473c7f5bab6568886f33b56738613f3d4321ae22b727622f4859a4157ee9850&owner=efccd8ff4cab0cf2fc878d4ce288c336756c4fb2376ba98922002e6d5b7afd8a&d=pic-suite — repaired head 2e59ef5ff63912de2a5b70fec476b86a889dd0ee, base/main 7527ed8de78afb60085dc477a964fd1632b6d57a (unchanged; verified ancestor of the tip). Independent review APPROVED for the adapter repair and the exact composed integration: 24 independent checks — 16 lifecycle/watchdog boundary cases, real-process idle-death retirement (with and without prior cancellation), bounded startup-child retirement (including TERM-ignoring pre-RPC children), composed quiet recovery before and after retry eligibility, genuine circuit reopen with maintenance refill, ordinary typing, and an intentional hard-cap policy probe — with the reused full-suite receipt 367 passed, zero failures.

Composed quiet-recovery results: each case (before retry eligibility, after eligibility, circuit reopen, normal typing) redelivered the original interrupted input exactly once with one durable worker run, one worker write, one terminal, one callback; startup children retire within bounded grace; the hard turn cap is preserved (the new progress path does not bypass it).

Gate status at posting: exact-range security review SUCCESS, zero findings (run 34156878857); CI success (run 34156407833). This post is independent evidence metadata only — not a formal GitHub review approval and not merge/activation authorization. Activation additionally requires the effective max-turn/idle policy check for the two originally affected managed agents (an activation-only gate, not a source blocker).

@jedwards27 jedwards27 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:bot: Jude’s code review agent

Verdict: APPROVE

Reviewed: 3c7f288c60d67df78577b237e27c3dfc8831aaa1..e4b3cb9057b33554aae8a1769a17a34dd3eb96ce (exact head e4b3cb9057b33554aae8a1769a17a34dd3eb96ce)

Risk: high — this changes ACP host scheduling across respawn completion, retry eligibility, maintenance/refill, cancellation, and shutdown.

Behavior/contracts traced:

  • The biased production select! preserves shutdown priority while independently waking for respawn completion, retry eligibility, and maintenance (crates/buzz-acp/src/lib.rs:3106-3125).
  • Retry wakes arm only when the pool is ready and has idle capacity; next_retry_deadline excludes empty, removed, and in-flight scopes (crates/buzz-acp/src/queue.rs:497-513). Existing requeue/mark_complete ordering retains retry budget, backoff, and timestamps.
  • Consuming a respawn result clears the single-slot in-flight state, restores a successful replacement to its indexed slot, and immediately dispatches pending work (crates/buzz-acp/src/lib.rs:3713-3758). Maintenance compacts state, refills circuit-eligible empty slots, and dispatches flushable work (crates/buzz-acp/src/lib.rs:3049-3093).
  • Closed respawn receivers disable only that wake arm without spinning or suppressing timers; cancellation does not consume a later result (crates/buzz-acp/src/recovery_wake.rs:21-32,65-104). Shutdown remains first priority and aborts/drains respawns while explicitly shutting down returned children (crates/buzz-acp/src/lib.rs:3118-3125,4122-4135).
  • Adversarial review found no changed path that bypasses retry budgets or the hard turn cap, duplicates dispatch, strands an eligible quiet-host batch, or leaks a returned respawn child. Batching, dedup/delivery state, removed-scope handling, and circuit policy remain unchanged.

Findings: no blocking or non-blocking defect found in the reviewed range.

Author action: none.

Verification owner: integration/deployment owner for composed companion-adapter runtime replay and confirmation of effective max-turn/idle activation policy for affected managed agents.

Validation at matching exact head:

  • Clean detached worktree and git rev-parse HEAD confirmed e4b3cb9057b33554aae8a1769a17a34dd3eb96ce immediately before submission.
  • git diff --check 3c7f288c60d67df78577b237e27c3dfc8831aaa1..HEAD — pass.
  • Independent exact-head cargo test -p buzz-acp — 920 library tests + 9 integration tests, 0 failures.
  • Required exact-head CI results inspected: Rust lint/unit tests, Linux cross-compiles, Windows Rust, Desktop core/build/smoke/integration, relay/backend integration, PostgreSQL, security, Semgrep, zizmor, and DCO are successful. Path-inapplicable jobs are skipped.

Runtime evidence: the PR documents loopback stock-host/adaptor fixture results for fast and delayed replacement, with unmodified-base mutation evidence. A later exact-head composed report covers before/after retry eligibility, circuit reopen/refill, normal typing, bounded child retirement, exactly-once durable delivery, and preserved hard cap. These are supporting author-posted artifacts; I did not independently replay the original live incident or run the final composed process suite.

Residual risk: adapter/runtime configuration and deployment activation outside this three-file host patch. No conventional UI or native visual surface changed, so screenshot/native UI evidence is not applicable.

@jedwards27 jedwards27 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:bot: Jude’s code review agent

Verdict: APPROVE
Reviewed: 3c7f288c60d67df78577b237e27c3dfc8831aaa1..e4b3cb9057b33554aae8a1769a17a34dd3eb96ce (exact head e4b3cb9057b33554aae8a1769a17a34dd3eb96ce)
Risk: high — host scheduler wakeups, retry timing, respawn state, shutdown ordering, and child cleanup.

Behavior/contracts traced: Both lanes traced the production biased select!, retry-deadline eligibility, respawn result consumption and slot replacement, maintenance/refill/dispatch, closed-channel behavior, cancellation, and shutdown cleanup (crates/buzz-acp/src/lib.rs:3049-3125,3713-3758,4122-4135; queue.rs:497-513; recovery_wake.rs:21-32,65-104). Quiet-host recovery no longer depends on incidental relay/presence/typing traffic. Shutdown remains first priority; retry wakes only arm with a ready pool and idle capacity; removed/empty/in-flight scopes do not generate retry deadlines; returned children are explicitly shut down during teardown.

Findings: no concrete author-actionable defect found. The lanes found no path in the searched three-file host patch that bypasses retry budgets, duplicates dispatch, strands eligible quiet-host work, suppresses timers after a closed respawn receiver, or leaks a returned respawn child.

Author action: none.
Verification owner: integration/deployment owner for composed companion-adapter replay and activation-policy confirmation.

Validation

At matching exact head:

  • git diff --check 3c7f288...HEAD — pass.
  • cargo test -p buzz-acp — pass in shared exact-head evidence: 920 library + 9 integration tests, 0 failures.
  • Exact-head GitHub Rust lint/unit, Windows Rust, Linux cross-compile, security, relay/backend integration, DCO, and aggregate required gates — success.

Manual/native evidence: no live incident replay or composed companion-adapter process run.
Residual risk: activation/configuration outside this three-file patch—effective max-turn/idle policy and companion-adapter runtime behavior—was not independently exercised. This is an integration/deployment confidence gap, not author rework.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

codex-security-review-current The posted Codex security review matches its recorded range.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants