-
Notifications
You must be signed in to change notification settings - Fork 55
docs: add evidence-driven RTDS troubleshooting guide #48
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
osr21
wants to merge
3
commits into
Polymarket:main
Choose a base branch
from
osr21:docs/evidence-driven-troubleshooting
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| # RTDS troubleshooting and actionable bug reports | ||
|
|
||
| The most useful production reports let another builder reproduce the failure and let maintainers identify the responsible layer. Use this sequence: | ||
|
|
||
| > **Symptom → evidence → workaround → requested platform fix** | ||
|
|
||
| Do not include API keys, wallet private keys, authentication headers, session tokens, or other credentials in an issue or log attachment. | ||
|
|
||
| ## Report template | ||
|
|
||
| ### Symptom | ||
|
|
||
| Describe the externally visible failure in one or two sentences. | ||
|
|
||
| Examples: | ||
|
|
||
| - "The socket remains `OPEN` and answers pings, but all subscribed topics stop delivering payloads." | ||
| - "A subscription is accepted, but the requested topic never produces an initial snapshot." | ||
| - "Messages continue to arrive, but their source timestamps are delayed by more than 30 minutes." | ||
|
|
||
| Include whether the failure is intermittent or permanent and whether a new connection recovers it. | ||
|
|
||
| ### Evidence | ||
|
|
||
| Provide enough detail to distinguish transport, subscription, and data-source failures: | ||
|
|
||
| - UTC start time, failure time, and recovery time | ||
| - RTDS host and exact subscription frame (`topic`, `type`, and `filters`) | ||
| - Client package version, Node/browser version, operating system, and WebSocket implementation | ||
| - Last valid payload time per topic—not only the last ping, pong, or metadata frame | ||
| - Socket `readyState`, close code/reason, and error callbacks | ||
| - Sanitized raw frames immediately before and after the failure | ||
| - Message counts and reconnect attempts | ||
| - Whether multiple topics on the same socket stopped together | ||
| - Whether a hard reconnect restored data | ||
|
|
||
| When possible, run a raw-frame capture beside the SDK client. This distinguishes an SDK parsing problem from a server-side publishing problem. | ||
|
|
||
| ### Workaround | ||
|
|
||
| Document the smallest mitigation that restored service, including thresholds and possible false positives. A useful workaround should be safe for other builders to copy while the root cause is investigated. | ||
|
|
||
| ### Requested platform fix | ||
|
|
||
| Name the invariant the platform should provide rather than only asking maintainers to "fix the socket." | ||
|
|
||
| Examples: | ||
|
|
||
| - A live gateway connection must retain valid backend subscription state. | ||
| - If backend subscription state is removed, the gateway should close the socket with an explicit reason. | ||
| - Subscription failures should return stable application-level error codes rather than internal database errors. | ||
|
|
||
| ## Distinguish transport liveness from data liveness | ||
|
|
||
| A WebSocket that is `OPEN` and answering pings is not necessarily delivering subscribed data. Ping/pong may only prove that the gateway and network path are alive. | ||
|
|
||
| For production consumers: | ||
|
|
||
| 1. Track the last **valid data payload** independently from ping/pong traffic. | ||
| 2. For naturally quiet subscriptions, add a normally active topic to the same socket as a data-plane heartbeat. | ||
| 3. If all expected payloads become stale, hard-close the socket and create a new connection. | ||
| 4. Restore subscriptions from `onConnect` after reconnecting. | ||
| 5. Add jitter to reconnect delays to avoid synchronized reconnect storms. | ||
|
|
||
| Do not use repeated duplicate subscriptions or intentionally triggered server/database errors as a production health check. They add load and couple the client to unstable implementation details. | ||
|
|
||
| ## Known silent-stall failure mode | ||
|
|
||
| Issue [#26](https://github.com/Polymarket/real-time-data-client/issues/26) documents a failure where all topics on a connection can stop while the socket remains open and ping/pong continues. In the reported tests, frozen connections did not recover through re-subscription; a new socket restored delivery. | ||
|
|
||
| The current client reconnects after socket close/error events. Applications that require continuous delivery should also implement data-payload staleness detection until the client or server provides an explicit data-plane liveness contract. | ||
|
|
||
| ## Minimal watchdog pattern | ||
|
|
||
| ```ts | ||
| const STALE_AFTER_MS = 15_000; | ||
| let lastPayloadAt = Date.now(); | ||
|
|
||
| const client = new RealTimeDataClient({ | ||
| onConnect: current => { | ||
| current.subscribe({ subscriptions: activeSubscriptions }); | ||
| }, | ||
| onMessage: (_current, message) => { | ||
| if (message?.payload) lastPayloadAt = Date.now(); | ||
| handleMessage(message); | ||
| }, | ||
| }); | ||
|
|
||
| setInterval(() => { | ||
| if (Date.now() - lastPayloadAt <= STALE_AFTER_MS) return; | ||
|
|
||
| // Replace the socket. Re-subscribing on a stale connection may not restore | ||
| // backend subscription state. | ||
| client.disconnect(); | ||
| client.connect(); | ||
| lastPayloadAt = Date.now(); | ||
| }, 1_000); | ||
| ``` | ||
|
|
||
| Choose the threshold based on the expected cadence of the heartbeat topic, not on a naturally quiet market topic. Production implementations should also guard against overlapping reconnects and reset/cancel watchdog timers during intentional shutdown. | ||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.