Skip to content

Add 'failed' terminal state for games that die mid-run#134

Open
byte-the-bot wants to merge 1 commit into
mainfrom
game-failed-status
Open

Add 'failed' terminal state for games that die mid-run#134
byte-the-bot wants to merge 1 commit into
mainfrom
game-failed-status

Conversation

@byte-the-bot

Copy link
Copy Markdown
Collaborator

Games whose runner died (OOM-killed worker whose job exhausted its retries) had no way out of running — seven have shown as "live now" on the leaderboard since the April/May stress tests. This adds GameStatus::Failed as a terminal failure state.

Changes

  • GameStatus::Failed"failed" (round-trip tested). No migration: status is unconstrained TEXT with an index.
  • Leaderboard games_in_progress counts only waiting/running — a positive list, so terminal states can never show as live again.
  • Game page: failed games get an "Incomplete" pill and a note ("never finished — its runner died partway through; no results, didn't affect ratings") instead of a perpetual Live badge.
  • Engine API maps failed → "complete": the board's only concern is whether more frames are coming.
  • run_game treats failed as terminal — a straggling job retry logs and short-circuits instead of resurrecting a game whose live window is long gone.

Manual cleanup after deploy

The seven stuck games (all from Apr 11 – May 7):

UPDATE games
SET status = 'failed', updated_at = NOW()
WHERE status IN ('waiting', 'running')
  AND created_at < NOW() - INTERVAL '1 day';

(Safe to run any time after this deploys; anything genuinely live is minutes old, not a day. Verify with SELECT game_id, status, created_at FROM games WHERE status IN ('waiting','running') ORDER BY created_at; first if you want to eyeball it.)

Follow-up (tracked)

A stuck-game sweeper that fails games automatically once they're stuck well past the turn ceiling, so this never needs manual SQL again.

🤖 Generated with Claude Code

Games whose runner died (OOM-killed worker, retries exhausted) sat in
'running' forever with no way out — seven have shown as 'live now' on
leaderboard pages since April/May. Adds GameStatus::Failed:

- games_in_progress counts only waiting/running (positive list)
- game page shows an 'Incomplete' pill and explanatory note
- engine API reports failed games as 'complete' (no more frames coming)
- a straggling GameRunnerJob retry on a failed game short-circuits
  instead of resurrecting it

No migration needed: status is unconstrained TEXT. Backfilling the
seven stuck rows is a manual UPDATE after deploy.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant