Remove reply promise from FIFO when a send is interrupted#13
Merged
Conversation
send_message pushes its reply promise onto the shared @reply_promises FIFO before writing the command. If the sending fiber was cancelled (Async::Stop) or errored while blocked inside send_data, the promise stayed queued with no command on the wire, so no reply would ever arrive for it. Because replies are matched purely positionally, one orphaned promise permanently shifted reply correlation by one for every later sender on the same connection: each reply resolved the previous sender's promise and the newest sender hung forever, silently. On a long-lived inbound socket this accumulated with every mid-send teardown (e.g. a caller hanging up while a command was being flushed) until every new call stalled on its first round-trip — recoverable only by reconnect or process restart. Outbound was immune since its FIFO dies with the per-call socket. Now send_message removes the promise from the FIFO and rejects it when send_data raises, keeping the invariant that a queued promise always corresponds to a command that fully reached the wire.
bgoRelatel
approved these changes
Jul 15, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a gradual, process-wide stall on long-lived inbound connections caused by reply-promise poisoning of the shared FIFO.
Base#send_messagepushes its reply promise onto@reply_promisesbefore writing the command. If the sending fiber was cancelled (Async::Stop, e.g. a call torn down on hangup while its command was still being flushed) or errored insidesend_data, nothing removed the promise — it stayed queued with no corresponding command on the wire, so no reply would ever arrive for it. Since replies are matched purely positionally (@reply_promises.shiftinreceive_message), one orphaned promise permanently shifts reply correlation by one for every later sender on the same connection: each reply resolves the previous sender's promise, and the newest sender hangs forever with no error.On an inbound socket (one connection shared by all calls for the process lifetime) this accumulates with every mid-send teardown until every new call stalls silently on its very first round-trip; only a reconnect or process restart clears it. Outbound is immune because its FIFO dies with the per-call socket.
Prior art: a3bb505 / 30a8306 already moved
@app_promisesoff FIFO matching onto Event-UUID correlation for the same fragility;command/reply/api/responsecarry no correlation token, so the FIFO must instead be kept strictly in sync with the wire.The fix
If
send_dataraises (includingAsync::Stop— hencerescue Exception), remove the promise from the FIFO and reject it, preserving the invariant that a queued promise always corresponds to a command that fully reached the wire. The lone partial-write case degrades to FreeSWITCH replying-ERRto one merged garbage frame, which lands on the sender whose command was swallowed — a truthful failure, with the FIFO staying in sync.Tests
New regression tests in
test/functional/librevox/listener/cancellation_poisoning_test.rb, using a fake connection that can park a sender insidesend_dataso it can be deterministically stopped in the vulnerable window:All three fail on master (starved promise / cross-talked reply / blocked FIFO slot) and pass with the fix. Full suite: 120 runs, 414 assertions, 0 failures.