Stop poll_top_streams hanging forever on a lost PART confirmation - #16
Merged
Conversation
poll_top_streams runs under APScheduler with max_instances=1, so a single
hung call silently stops every subsequent poll ("maximum number of running
instances reached") until the process restarts. Throughput then decays over
hours rather than failing outright. Two independent causes, both fixed here.
Chat side, the real one. twitchAPI's Chat.leave_room() adds the channel to
_room_leave_locks, sends PART, then waits for the confirmation to clear it --
with no timeout. Chat.__connect() does not clear those locks on reconnect;
only the full _stop() does. So a reconnect during leave_room() throws away
the confirmation and the wait loop never ends. The trigger is our own
no_message_reset_time=0.5, which drops the websocket receive timeout to 30s,
so any 30-second lull reconnects. The failure compounds: less chat means more
reconnects, which freezes the channel set, which means less chat.
We depend on plain PyPI twitchAPI rather than a fork, so patches/ rewrites
the installed package at image build time to add a leave_timeout, mirroring
the join_timeout that join_room already has. The script is idempotent and
exits non-zero if the upstream method body has moved, so a version bump fails
the build instead of silently shipping unpatched. Same fix is open upstream
as Teekeks/pyTwitchAPI#365; drop patches/ once that lands in a release.
API side, defence in depth. get_streams is bounded only by aiohttp's 300s
default, well past the 120s poll interval, so a stall eats several cycles
before raising. Measured median is ~0.1s; 10s leaves ~100x headroom while
still recovering inside one interval. Not lower: a skipped poll opens a 240s
gap against the 180s REDIS_STREAMER_TTL, expiring online keys and churning
lifecycle events.
Deliberately no wait_for around join_room/leave_room. The library already
bounds joins, a legal rate-limited join can take up to 20s, and cancelling
from outside leaks the very lock entry this patch exists to release.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The bug
poll_top_streamsruns under APScheduler withmax_instances=1. One hung call therefore stops every subsequent poll — the log fills withskipped: maximum number of running instances reached— until the process restarts. Throughput decays over hours instead of failing outright, so it does not look like a crash.Hit twice in one day (after 1h15m, then 3h10m).
Cause 1 —
Chat.leave_room()waits with no deadlineleave_room()adds the channel to_room_leave_locks, sends PART, then waits for the confirmation to clear it. That wait has no timeout.Chat.__connect()does not clear those locks on reconnect — only the full_stop()does. So a reconnect mid-leave_room()throws away the confirmation the wait is blocked on, and it never returns.The trigger is our own
no_message_reset_time=0.5, which drops the websocket receive timeout to 30s. Any 30-second lull reconnects. The failure compounds: less chat → more reconnects → poller freezes → channel set freezes → less chat.We depend on plain PyPI
twitchAPI, not a fork, sopatches/twitchapi_leave_room_timeout.pyrewrites the installed package at image build time to add aleave_timeout, mirroring thejoin_timeoutthatjoin_roomalready has.The script is idempotent, and exits non-zero if the upstream method body has moved — a version bump fails the build rather than silently shipping an unpatched image.
Same fix is open upstream as Teekeks/pyTwitchAPI#365. Drop
patches/once it lands in a release.Cause 2 — unbounded
get_streamsBounded only by aiohttp's 300s default, well past the 120s poll interval, so a stall eats several cycles before raising. Now 10s.
Measured median is ~0.1s, so that is ~100x headroom while still recovering inside one interval. Not lower: a skipped poll opens a 240s gap against the 180s
REDIS_STREAMER_TTL, expiring online keys and churning lifecycle events.Deliberately not done
No
wait_foraroundjoin_room/leave_room. The library already bounds joins, a legal rate-limited join can take up to 20s (10.05s bucket wait + 10sjoin_timeout), and cancelling from outside leaks the very_room_join_locks/_room_leave_locksentry this patch exists to release.How to tell it is working
active_stream_countkeeps moving (it is set bypoll_top_streams, so a frozen gauge means a hung poller, not quiet streams)docker logs streamscout-stream-monitoring | grep "skipped: maximum"stays emptyNote
patches/was previously untracked, sodocker compose build stream-monitoringcould not succeed from a fresh clone — the DockerfileCOPYreferenced a path not in git. This commit fixes that too.🤖 Generated with Claude Code