Billing and scheduling anomalies on paid Zero Sum Experience Requests (external researcher report, 22 Sep)

[arisklar6]

Reported to me by Manel Enrico on 22 Sep 2026, running metered Experience Requests on hosted Zero Sum (coworld zero-sum, league_6764202e). These are billed runs, so the billing items are the ones to look at first.

1. Pod price is per minute of pod life including queue time. The same queue length produced $0.78 and $0.21 for one game on 22 Sep; normal is $0.04 to $0.08. The per-minute rate also varies about 3x between seeds with identical queue and play time.
2. Episodes that closed with scores but were never billed. Five in one series, two in another. Examples: A/20889290, A/20679832.
3. Requests moving from running back to submitted, and about 26 hours with no scheduling on 22 Sep from 07:10 UTC.
4. One seat played and scored but no player artifact was produced. Request 20784561, agent 10. A retry does not create it.
5. spend_limit is not settable per request (422), and cost_preview only exists at creation.
   Item 3 is the one I would look at first. The Battle Royale ladder (league_b88a269b) also stopped advancing rounds around the same window, and an outside entrant has an entry stuck on "awaiting", so this may be one scheduler problem showing up in two places.

Happy to relay follow-up questions to the reporter, or put you in touch directly.

[softmax-cogent]

Tracked by CoMax Coworld Dev as COM-1941. We'll follow up here when there's a fix or a question.

[softmax-cogent]

Thanks for the detailed report — relaying findings from our platform team (Metta-AI/metta owns hosted Experience Request billing/scheduling; this repo is the CLI/manifests only).

1. ~3x pod price variance for identical queue/play time is by design, not a bug. Each completed episode JobRequest is priced at the node's full hourly list rate × the runner pod's wall clock — node occupancy, not container time — so queue/startup time bills as episode cost, and the same game on a different instance type costs a different amount. Source: docs/ai/onboarding/services/observatory/coworld-cost-baseline.md §1.1 (Metta-AI/metta). We agree the fair part of the complaint — that the rate/node/wall-clock split isn't visible to a requester — is a real disclosure gap, and it's noted for improvement.

2. spend_limit / cost_preview is already shipped on main. episode_player_llm_spend_limit_usd is accepted at experience-request creation (split evenly across player seats, enforced against min(league PPPE, requester PPPE)), and the effective player_pod_llm_spend_limit_usd now rides in cost_preview on the request detail — not only at creation time (metta #23789 + #23821). Could you confirm which client/version you're on and the exact payload that returns 422? Note: changing the limit after creation is unsupported by design, only at creation.

3. Scored-but-unbilled episodes (A/20889290, A/20679832) and the missing player artifact (request 20784561, agent 10) — accepted as candidate defects, filed to the platform team for investigation.

4. Scheduler stall + running→submitted flap (both leagues) — filed to the platform team. One thing that would help the diagnosis: you wrote "~26h with no scheduling on 22 Sep from 07:10 UTC" — that start time and duration don't both fit inside a same-day report, so could you pin down the exact window (start and end timestamps, UTC)? That drives which code path we look at.

None of this is coworld-CLI/manifest work, so there's no fix landing in this repo — the billing/scheduling items are being tracked and worked on the hosted platform side. We'll close the loop here once there's a resolution; happy to take follow-up questions in this thread.

[arisklar6]

Posted by Ari's agent, from Ari's account.

Thank you for the fast triage. One correction to Ari's framing: he wrote "paid" and "billed" runs, and he has since been told Softmax does not invoice for these, so please read those as cost-recorded rather than charged. The two candidate defects and the scheduler stall stand as reported.

Ari has passed your two questions to the reporter (client version plus the exact payload that returned 422, and the exact UTC start and end of the scheduling stall) and will post the answers in this thread when they come back.

[arisklar6]

This comment is from Ari's agent (an AI assistant), not from Ari personally. It is posted from Ari's account with Ari's approval.

While we wait for the external reporter's exact window, here is what the league data shows. All times are UTC, from /v2/rounds and /v2/leagues/health on 2026-09-23.

Zero Sum (league_6764202e): stalls every day on the league credit pool

• Since 09-17, each day's first round is created at 07:00–07:01Z, right after the daily reset. The league then runs until mid-day and idles until the next 07:00Z. Last round completed: 09-17 16:08, 09-18 16:43, 09-19 15:21, 09-20 15:21, 09-21 17:04, 09-22 14:00, 09-23 11:16.
• GET /v2/leagues/health returns refusal_cause: "league_pool" (checked 09-23 19:13Z), and the division rewards show league_pool_credits: 12.92. The coworld envelope is not the limit ($12.01 of $25 spent).
• These refusals never appear in /v2/rounds as budget_refused and emit no competition event, so from the round list a budget stall looks the same as a dead scheduler.
• Likely trigger (see Zero Sum league: team_count 16 seats every entrant alone, so submitted policies are teammates and 12 of 16 seats are filler #44): rounds went from 1 episode to 32 between R5301 (09-16 21:46Z) and R5302 (21:56Z). That is the num_episodes: 32 activation @aaln flagged in Zero Sum league: team_count 16 seats every entrant alone, so submitted policies are teammates and 12 of 16 seats are filler #44 on 09-02, and it coincides with metta#23351.

Battle Royale (league_b88a269b): same daily shape

• Rounds restart around 07:01Z and stop between 10:00 and 23:00Z: 8–31 rounds a day against the 48 the 30-minute interval allows. Overnight gaps include 783 min (09-22 17:57 to 09-23 07:01), 902, 633 and 543 min. R2216 (09-19 20:27Z) was refused: "round needs 4 credits (84 seats) but league Battle Royale's pool holds 4".
• Separately, 5 of 23 rounds failed in the 24h to 18:52Z (R2285, R2290–R2293). Each lost 1 of 6 episodes to node_disruption ("DeletionByPodGC: PodGC: node no longer exists") or one game_error, and no replacement episode-request was created despite retry_times: 2.

Happy to share the raw JSON for any of these.

[arisklar6]

This comment is from Ari's agent (an AI assistant), not from Ari personally. Posted from Ari's account with Ari's approval. Root-cause analysis and draft PRs by Devin; the data is ours.

@aaln (cc @treeform for Battle Royale): here are three settings changes and some PR reviews, following up on the stall data above. Nothing else is requested: no reward-pool, budget, upload or merge changes beyond what's listed.

Why: ladder.fulfillment.allowed_failures = 0.05, compared as failed/planned <= allowed_failures, where every episode without a blamed seat counts as failed (updater.py:280-314 on metta 4baa2948). That tolerates 0 failures in a 6-episode Battle Royale round and 1 in a 32-episode Zero Sum round, so a single node_disruption or game_error voids a whole round, including the episodes that completed (which are still paid for). In our data:

• Zero Sum: 8 of 51 32-episode rounds failed between 09-16 and 09-23. 236 completed episodes were discarded; none of the failures was blamed on a policy.
• Battle Royale: 5 of 23 rounds failed in the 24h to 09-23 18:52Z.
• Replaying Zero Sum's real rounds at 0.10: failed rounds drop from 8 to 1, and lost episodes from 16.1% to 3.4%.
• #24726 (episode redispatch of infra failures, merged 09-23 20:10Z) should cut these further, but an episode that exhausts its redispatches or fails with game_error still fails the round at 0.05. For what it's worth, Battle Royale ran R2299 through R2307 (20:27Z to 00:27Z) clean, 6 of 6 each.

Only staff can write ladder.* (leagues.py:3040, ladder_write_allowed), so:

C1. Zero Sum (league_6764202e-2ac1-4c98-a77e-cabc75582939): ladder.fulfillment.allowed_failures 0.05 → 0.10. Keep num_episodes 32 and k_factor 16.
C2. Battle Royale (league_b88a269b-0de7-4723-b1c7-06dab50fe61d): allowed_failures 0.05 → 0.17 (tolerates exactly 1 of 6; 2 of 6 still fails). Longer term, B1 below removes infra failures from the denominator.

POST replaces the whole settings document. This guarded script changes exactly one key and aborts otherwise: GET snapshot, merge, diff, validate, then POST with If-Match. Re-POSTing with ladder.enabled already true doesn't clear leaderboards (the enabling_ladder guard).

#!/usr/bin/env bash
# usage: set_allowed_failures.sh <league_id> <new_allowed_failures>
set -euo pipefail
API=https://api.observatory.softmax-research.net
LEAGUE=${1:?league id}; NEW=${2:?new allowed_failures}
AUTH="Authorization: Bearer ${OBSERVATORY_TOKEN:?}"

curl -fsS -H "$AUTH" "$API/v2/leagues/$LEAGUE/settings/snapshot" > snapshot.json
REV=$(jq -er '.revision' snapshot.json)
jq -e '.settings' snapshot.json > before.json
for key in .ladder.scheduler .ladder.divisions .ladder.ranking .ladder.fulfillment .counterfactual_eval; do
  jq -e "$key != null" before.json >/dev/null || { echo "missing $key; aborting"; exit 1; }
done
jq --argjson v "$NEW" '.ladder.fulfillment.allowed_failures = $v' before.json > after.json
# Guard: nothing but ladder.fulfillment.allowed_failures may differ.
cmp -s <(jq -S 'del(.ladder.fulfillment.allowed_failures)' before.json) \
       <(jq -S 'del(.ladder.fulfillment.allowed_failures)' after.json) \
  || { echo "unexpected change beyond allowed_failures; aborting"; exit 1; }
diff <(jq -S . before.json) <(jq -S . after.json) || true
curl -fsS -H "$AUTH" -H "Content-Type: application/json" -H "If-Match: $REV" \
  -X POST "$API/v2/leagues/$LEAGUE/settings/validate" --data @after.json >/dev/null
read -r -p "Apply to $LEAGUE? [y/N] " ok; [[ "$ok" == "y" ]] || exit 1
curl -fsS -H "$AUTH" -H "Content-Type: application/json" -H "If-Match: $REV" \
  -X POST "$API/v2/leagues/$LEAGUE/settings" --data @after.json | jq '.settings.ladder.fulfillment'

./set_allowed_failures.sh league_6764202e-2ac1-4c98-a77e-cabc75582939 0.10   # C1 Zero Sum
./set_allowed_failures.sh league_b88a269b-0de7-4723-b1c7-06dab50fe61d 0.17   # C2 Battle Royale

(GET …/settings/snapshot returns revision and settings; I checked it on the Zero Sum league with the owner token.)

C3. Disqualification: keep disqualify_after_consecutive_failures = 3 on both leagues. Please review:

• metta#24778 (B3): the streak resets when a new champion version starts. daf-battleroyale-v1:v1 was DQ'd in its first round with 3 failures inherited from daf-cogame-carrier:v2.
• metta#24780 (B4): innocent co-seated policies no longer get 1.0 plus a win in blamed-failure episodes. That credit inflates win rates league-wide.

Also, if you have a minute:

• Reviews: metta#24772 (B1: infra failures leave the allowed_failures denominator, behind a Temporal patch gate) and metta#24776 (B2: keeps a history of round-budget prechecks in /v2/leagues/health; optional).
• Merge-ready: metta#20412, #22880 and #20419 are approved, and CI is green (#20419 was freshly rebased).

Thanks!

[arisklar6]

This comment is from Ari's agent (an AI assistant), not from Ari personally. Posted from Ari's account with Ari's approval.

@aaln: please hold C1 (Zero Sum allowed_failures 0.05 → 0.10) until we've confirmed it with the researcher who filed this issue. Their work runs on Zero Sum, and we're checking with them before changing anything that touches it. We'll post here once they've replied.

C2 (Battle Royale 0.17) and the PR reviews (B1–B4) are unaffected and fine to proceed.

[arisklar6]

This comment is from Ari's agent (an AI assistant), not from Ari personally. Posted from Ari's account with Ari's approval.

The reporter audited their own logs and answered both questions above, with corrections to the original list.

1. Pod price variance. Understood as node rate × pod wall clock. The $0.78 and $0.21 games were queue time recorded as cost, consistent with your explanation. For the disclosure gap, their specific ask: show queue time and instance type per episode to the requester. That alone would be enough.

2. Scored but never cost-recorded: six episodes, not seven (3.0% of 200 completed, across four series). Rechecked 22 Sep 04:45Z: all still completed, cost_usd null, error null, the oldest for over two days. Three are from an arm that calls the model and three from an arm that calls nothing, so the LLM sidecar is not the cause. From the requester side, "never cost-recorded" and "cost-recorded but not reported" look the same.

series	run	xp request	episode request
P5-6C	F/20679832	xreq_98327df9-b33a-417d-8a64-1c876035d635	ereq_90707b08-132b-45df-bf49-68ee0c91351e
P5-6C	F/22564954	xreq_bedb0ab0-acf2-4499-8594-3364c616177e	ereq_26eb8c79-e2f7-4e1b-a3ca-269d90c514b3
P5-6C	F/23402786	xreq_d91be50f-c24e-4d12-a8c4-b119f510da9e	ereq_e956aa43-2fe5-49c1-8508-c8568ae5fb88
P5-7C	A/20889290	xreq_0a56e647-8799-4225-ba18-94489f7aa134	ereq_f3546500-d73a-45c4-b8f0-093bfbc3babf
P5-8I	A/20679832	xreq_cc3e7315-7916-46b6-ac43-71da5a0451bc	ereq_0334be26-b3b1-4553-aa6b-d1d668a370ff
P5-8M	A/21622393	xreq_104822db-7094-44fb-8c55-f79dc8767e21	ereq_d18467e9-6ea5-4de5-a1c6-9a00be9e6912

3. Scheduler stall: 21 Sep, 6 h 30 min (not ~26 h on 22 Sep):

• last request scheduled: 2026-09-21 06:44:29Z (T/23926431)
• first left unscheduled: 07:10:16Z (F/24031160); requests from then were never scheduled and had to be cancelled
• first to start again: 13:40:43Z (F/24135889), after a 7.1 min queue

Episodes either side of the window were cost-recorded at normal rates. 22 Sep had no stall on their side: all 58 episodes started and finished, and the 06:58Z–12:55Z gap was simply no launches. One oddity: the first request after it (13:04:23Z) queued 8.8 min against a 1.0 min median that day, then ran normally. Nothing new on the running → submitted flap.

4. Missing player artifact: stands. Request 20784561, agent 10: the episode completed and was cost-recorded ($0.097807), the seat has a score, and the server answers Agent 10 has no artifact. Available agent indices: 11. It is the only case in 280 episodes.

Re-runs: no sign of any in these seven anomalies (items 2 and 4): job_index 0 and round_id null in all seven. The three P5-6C episodes were created before the 21 Sep stall, so none is one of the six requests cancelled and relaunched then (those were cost-recorded normally).

5. spend_limit: withdrawn. The 422 came from a malformed first request (wrong roster and target shapes), not from spend_limit, and the reporter has since found episode_player_llm_spend_limit_usd. Client: coworld 0.1.44. The CLI can't print its own version, so that came from pip metadata: filed separately as #69.

Charges: confirmed that nobody charges the reporter money. Their credits are a balance granted by Softmax, and hosted games call the model through the sidecar with platform credentials.

C1: the reporter doesn't use the Zero Sum league standings, so C1 (allowed_failures 0.05 → 0.10) doesn't touch their work. Hold lifted: @aaln, C1 can go ahead if it's still wanted.

Still open on the platform side: the six episodes, the 21 Sep stall, the missing artifact and the disclosure ask.