Fix unbounded event-task leak on the per-connection barrier#15
Merged
Conversation
henrikbjorn
force-pushed
the
fix-event-barrier-task-leak
branch
5 times, most recently
from
July 19, 2026 10:59
177496c to
c0e5085
Compare
Listener::Base dispatched every ESL event through a single per-connection Async::Barrier (@event_barrier). In async, Barrier#async appends the task to an internal list and only ever removes it in #wait / #cancel. librevox calls @event_barrier.wait exactly once — in #connection_closed (socket drop) — and the read loop never drains it between messages. So on a long-lived inbound connection subscribed to events, every event received permanently leaks a task node (plus the finished Async::Task it references and a node in the barrier's @finished queue). On a busy FreeSWITCH that is thousands of nodes per second, freed only when the socket drops — a slow, unbounded memory leak (observed as ~8GB over ~2 days in the answering_machine daemon). Replace the barrier with a Set of in-flight event tasks: each task adds itself on entry and removes itself in its own ensure, so a finished task is never retained. #connection_closed still drains whatever is genuinely in flight by waiting on a snapshot of the set. Adds a regression test asserting completed event tasks are not retained.
henrikbjorn
force-pushed
the
fix-event-barrier-task-leak
branch
from
July 19, 2026 11:00
c0e5085 to
9223c8f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Librevox::Listener::Basedispatches every ESL event through a singleper-connection
Async::Barrier(@event_barrier):In
async,Barrier#asyncappends the task to an internal list and a task isremoved only by
#wait/#cancel. librevox calls@event_barrier.waitexactly once — in
#connection_closed(socket drop) — and the read loop neverdrains it between messages.
So on a long-lived inbound connection subscribed to events, every event
received permanently retains a task node (plus the finished
Async::Taskitreferences, plus a node in the barrier's
@finishedqueue). On a busyFreeSWITCH that is thousands of nodes/second, freed only when the socket drops —
a slow but unbounded memory leak. This is the cause of the ~8 GB-over-~2-days
growth we saw in the
answering_machinedaemon.A
Barrieris a scatter-gather join primitive — it retains its tasks onpurpose so a later
waitcan join them. It's the wrong tool for supervising anunbounded, fire-and-forget event stream where
waitonly happens at shutdown.Fix
Track in-flight event tasks in a
Setinstead. Each task adds itself on entryand removes itself in its own
ensure, so a finished task is never retained.#connection_closedstill drains genuinely in-flight hooks by waiting on asnapshot of the set.
The per-event task itself is still required (a hook may issue a command and
block on its reply, which arrives via the same reader loop — running it inline
would deadlock; see
test_on_event_with_api_does_not_block_handle_response).Only the retention of completed tasks changes.
Test
Adds
test_completed_event_tasks_are_not_retained: after 100 events thelistener holds zero in-flight tasks. Full suite green (121 runs, 0 failures).
Note
This is a minimal, targeted fix. The
spike/em-style-simplificationbranchreworks this dispatch path more broadly and may supersede it — happy to fold the
change into that direction instead.