fix: resolve busy-loop in heartbeat ServerHandler causing high CPU#134
Open
medcl wants to merge 2 commits into
Open
fix: resolve busy-loop in heartbeat ServerHandler causing high CPU#134medcl wants to merge 2 commits into
medcl wants to merge 2 commits into
Conversation
The ServerHandler registration loop ignored conn.Read errors and return length. When a client connects then immediately disconnects (e.g. port scan, health check probe), Read returns (0, io.EOF) which was ignored, causing an infinite busy-loop of read()+write() syscalls that pins CPU in kernel mode and leaks goroutines permanently. Changes: - Check conn.Read error and byte count; return on error to release goroutine - Use data[2:n] instead of data[2:] for uid to avoid reading garbage - Add 30s read deadline during registration phase as safety net - Add read lock protection for concurrent CMap access in PushGRT and ServerWHandler to prevent data race - Add 100ms backoff in accept loop on error to prevent tight spin Fixes: GATEWAY-CPU-HIGH (goroutine leak via port 61111 heartbeat server)
…odules heartbeat/client.go: - Replace close-then-send pattern with sync.Once based Stop() - Add signalDone() helper to safely signal disconnection without panicking on closed channel - Use buffered select for Rch writes to prevent blocking - Add Dch select case in ClientWHandler/ClientWork to enable clean exit heartbeat/server.go: - Add timeout (2s) on PushGRT channel write to prevent permanent deadlock when consumer goroutine dies floating_ip/floating_ip.go: - Use buffered aliveChan (cap 1) in SwitchToStandbyMode and StateMachine to prevent goroutine leak from double-send to unbuffered channel - Use non-blocking select for all aliveChan sends to handle race between defer, onConnect, and onDisconnect callbacks floating_ip/broadcast.go: - Continue on UDP ReadFromUDP error instead of passing nil/zero data to handler function
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The ServerHandler registration loop ignored conn.Read errors and return length. When a client connects then immediately disconnects (e.g. port scan, health check probe), Read returns (0, io.EOF) which was ignored, causing an infinite busy-loop of read()+write() syscalls that pins CPU in kernel mode and leaks goroutines permanently.
Changes:
Fixes: GATEWAY-CPU-HIGH (goroutine leak via port 61111 heartbeat server)
What does this PR do
Rationale for this change
Standards checklist