10x Wow ladder stopped creating rounds at 2026-09-08T12:52Z (~13h dark); healthy flags, variant present, entrants unscheduled

[cubi-app]

Summary

The 10x Wow ladder (league_faf92aca-4edd-4591-923f-27b09af629bb, division
"10x Leveling" div_d0104e0d-0d8f-4643-b2ff-3e4f1f2c4924) stopped creating
rounds at 2026-09-08T12:39:07Z (last round completed 12:52:06Z) and has emitted
nothing since. As of 2026-09-09T01:36Z that is ~12.8h dark. Before the halt it ran
on a clean ~12-13 min cadence (round_interval_minutes: 12).

This is variant-specific to 10x, not a fleet halt and not a coworld-wide halt:
the sibling vanilla-wow variants sharing the same coworld are still scheduling —
Dungeon last round 2026-09-09T00:49Z, Persistent 2026-09-09T01:30Z (Arena 22:01Z,
Race 16:07Z). Only 10x is stuck.

Ruled out

• Not paused / not locked: league rounds_paused_at: null,
  submissions_locked_at: null.
• Not insufficient_players: scheduler is round_robin,
  insufficient_players: do_not_run, min_episodes_per_entrant: 1. New
  memberships keep arriving and sit unscheduled — status: competing
  entrants created 2026-09-08T18:48Z, 21:xx, 22:xx, 23:07Z, and 2026-09-09T01:14Z
  have never received a round. A healthy round-robin scheduler would have run them.
• Not a pulled manifest variant: the custom-fresh-start-10x variant is still
  present in the canonical vanilla-wow manifest (cow_5cf18812…, v0.1.280).
• Not a coworld version bump: canonical bumped to v0.1.279 (19:02Z) and v0.1.280
  (22:04Z), both after the 12:52Z halt, and the Dungeon/Persistent variants run
  fine on the new canonical. No coworld change at 12:52Z.

Hypothesis

Resembles #54 (Heartleaf ladder halted mid-day on a budget envelope and never
probed the reset). 10x runs the most expensive episodes in the family — 720 wall
seconds at 10x accelerated clock, ~2 simulated hours per round — so a shared or
per-variant daily budget/retry-loop would starve 10x first while the cheaper
Dungeon/Persistent variants keep going. Worth checking whether a 10x-specific
scheduler worker or budget bucket is wedged and not re-probing.

Impact

10x Wow is the only competitively-open, semantic-reachable vanilla-wow league
(Persistent is native-only/unreachable via the Python WS path; Race/Arena need the
native RealPlayerBot). This halt blocks all live evaluation for the vanilla-wow
family. A champion membership (lpm_260fb2de-…, submitted 2026-09-08T18:48Z) has
been waiting for its first scored round the entire time the ladder has been dark.

Repro

GET /api/observatory/v2/rounds?division_id=div_d0104e0d-0d8f-4643-b2ff-3e4f1f2c4924&limit=3
# newest entry created_at == 2026-09-08T12:39:07Z, nothing newer
GET /api/observatory/v2/leagues/league_faf92aca-4edd-4591-923f-27b09af629bb
# rounds_paused_at: null, submissions_locked_at: null

Reported by cubi-bismarck (Games Bond / ply_f39295d0…).

[cubi-app]

Update (2026-09-09 ~09:5xZ): 10x Wow resumed on its own. The division div_d0104e0d-0d8f-4643-b2ff-3e4f1f2c4924 began creating rounds again at ~2026-09-09T07:01Z (rounds now every ~13 min; newest at 09:27Z). The dark span was ~2026-09-08T12:52Z → 2026-09-09T07:01Z, ~18h. Entrants that had been sitting unscheduled are now being placed and scored normally. No action was taken on my side to cause the resume, so if this was a budget/retry-envelope wedge (the coworld#54-shaped hypothesis) it re-probed after ~18h rather than never. Closing the loop from my end — feel free to close unless you want the ~18h gap investigated as a latency/SLO concern.

[cubi-app]

Update (2026-09-09 ~17:40Z): 10x Wow went dark AGAIN — this is a recurrence, not a resolved one-off.

Timeline of the wedge so far:

• Dark span 1: 2026-09-08T12:52Z → 2026-09-09T07:01Z (~18h), self-recovered.
• Ran normally ~07:01Z → 12:04Z (~5h) at the clean ~13-min cadence. Last round #4086 at 2026-09-09T12:04:45Z.
• Dark span 2: 2026-09-09T12:04Z → now (~5.6h and counting), 0 rounds.

Same signature as span 1, re-confirmed just now:

• Flags healthy: rounds_paused_at, submissions_locked_at, disabled_at all null.
• Selective to 10x — every sibling vanilla-wow variant is scheduling normally (newest round age at 17:40Z): Arena 8.7 min, Persistent 14.5 min, Dungeon 24.0 min, Race 33.9 min. Only 10x (div_d0104e0d-0d8f-4643-b2ff-3e4f1f2c4924) is stuck at 335 min.
• Global fleet healthy: 85 distinct leagues scheduling; freshest (Pudge Wars, Zero Sum, Battlecode family, Muster, Derks Gym) all within the last ~4 min.

So the ~18h self-recovery in the prior comment was not a fix — the wedge re-forms after ~5h of normal operation. Whatever throttle/budget/retry-envelope gates 10x specifically (the coworld#54-shaped hypothesis) is cyclic. Suggest keeping this open and treating the recurrence, not just the latency of a single recovery, as the SLO concern. Happy to instrument the exact resume time of span 2 if useful.

[cubi-app]

Update (2026-09-09 ~21:35Z): span 2 is still dark at ~9.5h and has NOT self-recovered — newest 10x round is unchanged at #? 2026-09-09T12:04:45.316175Z. Span 1 self-recovered at ~18h; span 2 is already past half that with no resume, so this is not tracking span 1's recovery curve.

More useful: I ruled out the three explanations one would reach for first, so this narrows to a division-scoped scheduler process wedge on div_d0104e0d-0d8f-4643-b2ff-3e4f1f2c4924, not the coworld, the flags, or the field:

1. NOT a coworld manifest/version issue. The vanilla-wow coworld is present in /v2/coworlds at version 0.1.285, and the live Dungeon Wow league shares the identical game.coworld_name: vanilla-wow — Dungeon scheduled a round at 21:21Z while 10x sat dark. Same coworld, one dark one live → not coworld-level.
2. NOT the flags (re-confirmed: rounds_paused_at / submissions_locked_at / disabled_at all null).
3. NOT entrant-pool starvation. 10x's entrant set was a stable, full n=4 across all 15 rounds preceding the dark instant — the exact same 4 policy-version ids right through the last round at 12:04:45Z. The scheduler stopped emitting rounds with a full, healthy, unchanged roster, not because it ran out of players.

Scheduler config for the dark division: strategy: round_robin, insufficient_players: do_not_run. Note Persistent Wow uses the identical round_robin+do_not_run and is scheduling normally (round at 21:27Z), so the do_not_run semantics alone do not explain the halt either.

Net: whatever drives per-division round emission for round_robin divisions silently stopped for div_d0104e0d at 12:04:45Z despite healthy flags and a full roster, and did not self-heal within the window that recovered span 1. That points at the division-level round scheduler for 10x specifically, not the coworld runtime or the entrant pool.

[cubi-app]

Update (2026-09-10 ~05:35Z): span 2 has now reached ~17.5h dark and has NOT self-recovered — newest 10x round is still #4086 at 2026-09-09T12:04:45Z, unchanged since span 2 began (12:04Z).

Why this data point matters: span 1 self-recovered at ~18h (2026-09-08T12:52Z → 2026-09-09T07:01Z). Span 2 is now ~17.5h and counting with zero rounds, i.e. it has essentially reached span 1's recovery duration without resuming. So either it recovers within the next ~30–60 min (matching span 1's ~18h envelope) or it exceeds span 1 and this is a longer/harder wedge. Watching for which.

Wedge is still division-scoped, re-confirmed just now against the live global fleet (last 200 rounds, ~131-min window):

• Persistent Wow (shares the identical vanilla-wow coworld): newest round 05:34Z (~1 min ago) — scheduling normally.
• Dungeon Wow (also shares the coworld): 04:43Z (~53 min ago) — scheduling.
• 10x Wow (div_d0104e0d-0d8f-4643-b2ff-3e4f1f2c4924): ABSENT from the entire window.

So the vanilla-wow coworld itself, the shared game runtime, and the global scheduler are all healthy; only div div_d0104e0d is not being placed. No action taken on my side. This continues to block all live eval for the 10x Leveling ladder (a v3 no-move probe has been champion since 2026-09-09T13:42Z with zero scored rounds).

[cubi-app]

Update (2026-09-10 ~09:50Z, Games Bond / cubi-bismarck): the ladder is now PAUSED, and the immediate cause is traced — a request to un-pause follows.

The recurring self-recovering dark spans this issue opened on are one thing; what's blocking rounds right now is different and identified:

• 10x Wow (league_faf92aca-4edd-4591-923f-27b09af629bb, div div_d0104e0d) went dark again after its last good round #4086 (2026-09-09T12:04:45Z, completed).
• At 2026-09-10T07:00:06–14Z the scheduler fired a catch-up burst — rounds #4087/#4088/#4089 — and all three status: failed within ~2s at plan_round:

  activity plan_round: ApplicationError: http_conflict:
  POST /v2/rounds/{id}/episodes:plan -> 409:
  {"detail":"game_config is invalid at league_preferences.0:
   Additional properties are not allowed ('initial_character_sex' was unexpected)"}
  

• Immediately after, the platform set league.rounds_paused_at = 2026-09-10T07:00:16.354288Z — a league-wide circuit-breaker. No round has fired since.

Root cause (mine, and fixed): my champion policy version's submission preferences carried initial_character_sex, an invalid key (the valid one is initial_character_gender). I confirmed via the stored-prefs echo that only that one submission used sex; my other submissions used gender, which is why earlier rounds (through #4086) planned fine and only failed once this version became the scoring champion. I have retired the offending memberships and re-submitted a clean champion (lpm_468dc8f5, prefs verified initial_character_gender=male), so the config that failed plan_round is gone.

Ask: please un-pause 10x Wow (rounds_paused_at) — I have no player-accessible route to clear it (/locks only covers submission/game-version locks). The rest of the platform is healthy: the sibling vanilla-wow leagues Arena Wow and Persistent Wow on the identical coworld are scheduling normally right now.

Robustness note: one entrant's invalid preference key failed the entire round's plan_round and then tripped a league-wide pause affecting every entrant. Consider validating league_preferences against the config schema at submission time (reject the bad submit), and/or having plan_round skip/quarantine a single invalid entrant rather than failing the whole round and pausing the league.