Skip to content

fix(llminternal): guard RunLive errChan sends so abandoned producers exit on reconnect#1167

Open
chethanuk wants to merge 1 commit into
google:mainfrom
chethanuk:fix/1152-runlive-errchan-leak
Open

fix(llminternal): guard RunLive errChan sends so abandoned producers exit on reconnect#1167
chethanuk wants to merge 1 commit into
google:mainfrom
chethanuk:fix/1152-runlive-errchan-leak

Conversation

@chethanuk

Copy link
Copy Markdown

Fixes #1152

Long-lived /run_live servers leak one goroutine (plus its captured connection state) every time a live session reconnects — and, it turns out, on ordinary teardown too. The producers inside Flow.RunLive do bare sends on the unbuffered, per-attempt errChan; once the consumer loop has stopped reading that channel, the producer blocks on the send forever.

Root cause

internal/llminternal/base_flow.go spawns two producer goroutines per connection attempt (a reader and a sender) that report failures via errChan <- err at three sites (:389, :430, :439). The consumer reads errChan at most once per attempt: on a resumable error it sets reconnect = true, breaks out, and cleanup() cancels connCtx and closes the connection. The other producer's in-flight Recv/Send* call now fails against the closed connection, and its bare send on the abandoned channel blocks permanently:

sequenceDiagram
    participant R as reader
    participant C as consumer
    participant S as sender
    R->>C: errChan <- resumable err
    C->>C: reconnect = true, break
    C->>C: cleanup: cancelConn + Close conn
    Note over C: next attempt gets fresh errChan
    S--xS: SendContent fails (conn closed)
    S--xS: errChan <- err on abandoned channel
    Note over S: blocks forever - leaked
Loading

The same mechanics fire without any reconnect: cancel the invocation context while the reader is parked in Recv (the everyday session-teardown path), and the consumer exits via ctx.Done()cleanup() closes the connection → the reader's Recv fails → its bare send strands it. So every normal live-session teardown leaks the reader goroutine, which makes this slightly broader than the reconnect-only symptom in the issue.

One small correction to the issue text, respectfully: cleanup() does run on the reconnect path — it is called unconditionally after the consumer loop (:541), before the reconnect check. Only the send-site guards are missing, which makes the fix smaller than proposed.

The fix

Guard all three errChan sends with the exact idiom the same function already uses for eventsChan (:409-413):

select {
case errChan <- err:
case <-connCtx.Done():
}
return

Guarding on connCtx, not ctx, is load-bearing: cleanup() always cancels connCtx — before the next attempt and before every return — so a guarded send is guaranteed to unblock an abandoned producer. ctx outlives reconnects and would leave the producer blocked across attempts.

Deliberately untouched: the eventsChan send (already guarded), the consumer's double-break flow at :521-537, and cleanup() placement.

Why not a buffered make(chan error, 2)? It also fixes the current leak, but as discussed on the issue it silently caps the number of background workers and masks the lifecycle instead of modeling it; the select guard extends the function's existing convention. An errgroup restructure would be a rewrite, out of proportion for this bug.

Tests

TestRunLiveNoGoroutineLeak (new base_flow_live_test.go) fakes the Gemini Live wire protocol with an httptest + gorilla/websocket server (the same approach as go-genai's own live_test.go), pointed at via HTTPOptions.BaseURL with a ws:// scheme — no new dependencies, no production test hooks.

case server script asserts
reader resumable error reconnects without leak conn 1: hard close after setup (client sees resumable 1006); conn 2: serves a turn turn received after reconnect, 2 connections, no leak
sender error after connection loss does not leak conn 1: hard close; conn 2: echoes a turn per client message retried Send succeeds on conn 2, no leak
non-resumable error terminates close frame code 1002 error surfaces through the iterator, exactly 1 connection, no leak
parent ctx cancel exits cleanly server stays silent; test cancels ctx RunLive unwinds fully, no leak

The leak assertion is dependency-free (goleak is not in go.mod): it snapshots runtime.Stack(all=true), counts goroutines still inside llminternal.(*Flow).RunLive, and polls until the count returns to the subtest's baseline or a 5s deadline expires, failing with the captured stacks.

Testing Plan

go build -mod=readonly ./...
go test -race -run TestRunLiveNoGoroutineLeak ./internal/llminternal/ -count=20   # 100/100 pass
go test -race -mod=readonly -count=1 -shuffle=on ./...                            # 2408 passed, 151 packages
golangci-lint run internal/llminternal/...                                        # no issues
go mod tidy -diff                                                                 # clean (no new deps)

Behavior-change evidence — the new test on the unfixed code fails deterministically, with the leaked producers parked in chan send at the exact bare-send site:

--- FAIL: TestRunLiveNoGoroutineLeak/reader_resumable_error_reconnects_without_leak
    base_flow_live_test.go:316: leaked 1 RunLive goroutines (baseline 0):
    goroutine 38 [chan send]:
    google.golang.org/adk/v2/internal/llminternal.(*Flow).RunLive.func1.2(...)
        internal/llminternal/base_flow.go:389 +0x6b5
    created by google.golang.org/adk/v2/internal/llminternal.(*Flow).RunLive.func1 in goroutine 22
--- FAIL: TestRunLiveNoGoroutineLeak/sender_error_after_connection_loss_does_not_leak
    base_flow_live_test.go:316: leaked 2 RunLive goroutines (baseline 1): ...
--- FAIL: TestRunLiveNoGoroutineLeak/parent_ctx_cancel_exits_cleanly
    base_flow_live_test.go:316: leaked 1 RunLive goroutines (baseline 3): ...

With the fix applied, all four cases pass 100/100 under -race -count=20.

…exit on reconnect

RunLive's reader and sender goroutines did bare sends on the unbuffered,
per-attempt errChan. Once the consumer loop stops reading that channel --
after a resumable-error reconnect, a parent context cancellation, or any
other teardown -- cleanup() closes the connection, the producer's in-flight
call fails, and its send blocks forever, leaking one goroutine (plus its
captured connection state) per occurrence. Long-lived /run_live servers
accumulate these across sessions and reconnects.

Guard all three sends with the same select-on-connCtx idiom the function
already uses for eventsChan. cleanup() always cancels connCtx before the
next attempt (and before returning), so a guarded send is guaranteed to
unblock. connCtx, not ctx: ctx outlives reconnects and would not unblock
an abandoned producer.

The regression test fakes the Gemini Live wire protocol over a local
websocket server and fails deterministically on the unfixed code with
goroutines parked in 'chan send' at the bare-send sites.

Fixes google#1152
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Flow.RunLive leaks one goroutine per reconnect: unbuffered errChan abandoned on resumable errors

1 participant