Add 'failed' terminal state for games that die mid-run#134
Open
byte-the-bot wants to merge 1 commit into
Open
Conversation
Games whose runner died (OOM-killed worker, retries exhausted) sat in 'running' forever with no way out — seven have shown as 'live now' on leaderboard pages since April/May. Adds GameStatus::Failed: - games_in_progress counts only waiting/running (positive list) - game page shows an 'Incomplete' pill and explanatory note - engine API reports failed games as 'complete' (no more frames coming) - a straggling GameRunnerJob retry on a failed game short-circuits instead of resurrecting it No migration needed: status is unconstrained TEXT. Backfilling the seven stuck rows is a manual UPDATE after deploy.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Games whose runner died (OOM-killed worker whose job exhausted its retries) had no way out of
running— seven have shown as "live now" on the leaderboard since the April/May stress tests. This addsGameStatus::Failedas a terminal failure state.Changes
GameStatus::Failed↔"failed"(round-trip tested). No migration:statusis unconstrained TEXT with an index.games_in_progresscounts onlywaiting/running— a positive list, so terminal states can never show as live again."complete": the board's only concern is whether more frames are coming.run_gametreats failed as terminal — a straggling job retry logs and short-circuits instead of resurrecting a game whose live window is long gone.Manual cleanup after deploy
The seven stuck games (all from Apr 11 – May 7):
(Safe to run any time after this deploys; anything genuinely live is minutes old, not a day. Verify with
SELECT game_id, status, created_at FROM games WHERE status IN ('waiting','running') ORDER BY created_at;first if you want to eyeball it.)Follow-up (tracked)
A stuck-game sweeper that fails games automatically once they're stuck well past the turn ceiling, so this never needs manual SQL again.
🤖 Generated with Claude Code