Skip to content

Fix unbounded event-task leak on the per-connection barrier#15

Merged
henrikbjorn merged 1 commit into
masterfrom
fix-event-barrier-task-leak
Jul 19, 2026
Merged

Fix unbounded event-task leak on the per-connection barrier#15
henrikbjorn merged 1 commit into
masterfrom
fix-event-barrier-task-leak

Conversation

@henrikbjorn

Copy link
Copy Markdown
Member

Problem

Librevox::Listener::Base dispatches every ESL event through a single
per-connection Async::Barrier (@event_barrier):

@event_barrier.async do
  on_event(response)
  invoke_event_hooks(response)
end

In async, Barrier#async appends the task to an internal list and a task is
removed only by #wait / #cancel. librevox calls @event_barrier.wait
exactly once — in #connection_closed (socket drop) — and the read loop never
drains it between messages.

So on a long-lived inbound connection subscribed to events, every event
received permanently retains a task node (plus the finished Async::Task it
references, plus a node in the barrier's @finished queue). On a busy
FreeSWITCH that is thousands of nodes/second, freed only when the socket drops —
a slow but unbounded memory leak. This is the cause of the ~8 GB-over-~2-days
growth we saw in the answering_machine daemon.

A Barrier is a scatter-gather join primitive — it retains its tasks on
purpose so a later wait can join them. It's the wrong tool for supervising an
unbounded, fire-and-forget event stream where wait only happens at shutdown.

Fix

Track in-flight event tasks in a Set instead. Each task adds itself on entry
and removes itself in its own ensure, so a finished task is never retained.
#connection_closed still drains genuinely in-flight hooks by waiting on a
snapshot of the set.

The per-event task itself is still required (a hook may issue a command and
block on its reply, which arrives via the same reader loop — running it inline
would deadlock; see test_on_event_with_api_does_not_block_handle_response).
Only the retention of completed tasks changes.

Test

Adds test_completed_event_tasks_are_not_retained: after 100 events the
listener holds zero in-flight tasks. Full suite green (121 runs, 0 failures).

Note

This is a minimal, targeted fix. The spike/em-style-simplification branch
reworks this dispatch path more broadly and may supersede it — happy to fold the
change into that direction instead.

Listener::Base dispatched every ESL event through a single per-connection
Async::Barrier (@event_barrier). In async, Barrier#async appends the task to an
internal list and only ever removes it in #wait / #cancel. librevox calls
@event_barrier.wait exactly once — in #connection_closed (socket drop) — and the
read loop never drains it between messages.

So on a long-lived inbound connection subscribed to events, every event received
permanently leaks a task node (plus the finished Async::Task it references and a
node in the barrier's @finished queue). On a busy FreeSWITCH that is thousands of
nodes per second, freed only when the socket drops — a slow, unbounded memory
leak (observed as ~8GB over ~2 days in the answering_machine daemon).

Replace the barrier with a Set of in-flight event tasks: each task adds itself on
entry and removes itself in its own ensure, so a finished task is never retained.
#connection_closed still drains whatever is genuinely in flight by waiting on a
snapshot of the set.

Adds a regression test asserting completed event tasks are not retained.
@henrikbjorn
henrikbjorn force-pushed the fix-event-barrier-task-leak branch from c0e5085 to 9223c8f Compare July 19, 2026 11:00
@henrikbjorn
henrikbjorn merged commit 86e2f73 into master Jul 19, 2026
3 checks passed
@henrikbjorn
henrikbjorn deleted the fix-event-barrier-task-leak branch July 19, 2026 11:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant