Skip to content

Fix the two flaky CI tests: stale torus readback and random landscape parameters - #287

Merged
genixpro merged 1 commit into
masterfrom
fix-flaky-custom-game-harness
Sep 14, 2026
Merged

genixpro merged 1 commit into
masterfrom
fix-flaky-custom-game-harness

Conversation

@genixpro

Copy link
Copy Markdown
Contributor

Two flaky tests have been failing CI runs on master and PRs whatever their code: CustomGameSetupHarness (from #240) and TorusRenderIntegrationTest. Both are test bugs, not game bugs. This PR changes only the two test files.

How it was reproduced

In a Docker container with CI's Ubuntu 24.04 packages, software GL and xvfb, running CI's exact commands in a loop on master 4ba685c:

Test Before After this PR
torus-render-test (scale 1, then scale 3 in the same profile) 8 failures in 45 runs 0 failures in 20 runs
CustomGameSetupHarness (visual) 3 failures in 52 runs 0 failures in 60 runs

The full CI custom-game step also passes with the fix.

TorusRenderIntegrationTest: stale readback

The failing check reads the torus atlas back with glGetTexImage right after drawing it. Mesa's llvmpipe renders on worker threads, and the readback sometimes returned the previous frame:

  • The "with worker" capture came back identical to the "without worker" one (503 -> 503; passing runs are 503 -> 639).
  • Instrumenting the draw showed the circle was drawn, with the same GL state as in passing runs and no GL error.
  • A later recapture of the same scene showed the marker.
  • With single-threaded llvmpipe (LP_NUM_THREADS=0), the unchanged test passed 16 of 16.

Fix: glFinish() before the readback.

CustomGameSetupHarness: random parameters leaking into later checks

It failed at three different assertions:

  • screen.validMap (line 1174 on master). After "Randomize parameters", the test pressed Regenerate, which keeps the random draw. It meant Reset, as its own comment says ("back to the landscapes' own parameters"). So the final resize ran on a random rain-shadow ridge layout, which could fail the lobby's validation ("The valleys are too narrow for the homes…"). Instrumented, the tile's parameters were non-default at that point in 6 of 6 runs. Fix: press Reset.
  • Starts differ between picker and lobby (line 1152). The test applied the picked landscape without the picker's request. The lobby's real Use path (CustomGameScreen.cpp:322) passes it, so without it the lobby kept the random draw it was given earlier. Fix: pass picker.chosenRequest(), and assert the options arrived.
  • Corner pixel not black (line 902). The terrain palette has no black, but a colony start near the map corner puts its black number label on that pixel. Fix: check every pixel of the first row and column that no marker covers, so a real letterbox band is still caught.

To rule out a real engine problem behind the start-position mismatch, I also generated 300 requests across every landscape with random parameters three ways: on the main thread, on a new thread, and again after other maps. All matched, so map generation is deterministic.

Limits

  • Reproduced on arm64 Docker, not on the x86-64 GitHub runners. CI on this PR is the check there.
  • The corner-pixel cause is inferred from the render code (black labels are the only black drawn in the preview). I didn't capture a failing image.
  • Players aren't affected by either bug. The game's own screen capture uses glReadPixels on the framebuffer being drawn, not a texture readback, but I didn't test whether it has a similar timing issue under llvmpipe.

🤖 Generated with Claude Code

https://claude.ai/code/session_01MN62g3sFzcFWiMzums3qge

… parameters

TorusRenderIntegrationTest read the atlas back with glGetTexImage straight
after drawing it. Under Mesa llvmpipe, which renders on worker threads, the
readback sometimes returned the previous frame, so the capture with a worker
assigned matched the capture without one (503 -> 503) and the marker check
failed about one run in five at interface scale 3. glFinish() before the
readback makes it wait for the frame.

CustomGameSetupHarness failed about one run in twenty at three points:
- After "Randomize parameters" it pressed Regenerate, which keeps the random
  draw, not Reset, which restores each landscape's own parameters, so the
  final resize checks ran on a random ridge layout that could fail
  validation (screen.validMap).
- It applied the picked landscape without the picker's request, unlike the
  lobby's Use, so the lobby kept an earlier random draw and could roll a
  different map from the same seed (start positions differ).
- It asserted one corner pixel was not black, but a colony start near the
  map corner puts its black number label on that pixel. It now checks the
  whole first row and column, skipping pixels under markers.

Map generation itself is deterministic: the same request generated on the
main thread, on a new thread and again after other maps matched for 300
requests across every landscape with random parameters.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MN62g3sFzcFWiMzums3qge
@genixpro
genixpro merged commit 1c2526e into master Sep 14, 2026
7 checks passed
@genixpro
genixpro deleted the fix-flaky-custom-game-harness branch September 14, 2026 23:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant