Stop autosave from stalling the game at high speed

[genixpro]

At 8x, 13x and 40x the game runs smoothly for a second or two, then freezes for a fraction of a second, over and over. The freeze is the autosave. Every 256 ticks GameGUI::syncStep serialized the whole game, including every cached pathfinding gradient, and wrote it to Auto_save.game on the game thread. Late-game saves reach 65–160 MB, and at 8x 256 ticks is only about 1.3 seconds.

Changes

1. Bulk gradient writes. Gradient fields go out as one run of bytes through a new OutputStream::writeUint16Sections. Each value used to cost three virtual calls, a name string the binary stream ignores, and its own 2-byte SHA1 update. The output bytes and SHA1 are identical; TextOutputStream keeps the per-value loop.
2. No whole-file SHA1 for autosaves. The header keeps zeros instead. The only reader of that hash is Engine::haveMap, when a client joins a YOG game. It now treats a file without a hash as unverifiable and downloads the host's copy, instead of trusting a local file with the same name. Manual saves, maps and the replay header still hash.
3. Autosave spacing follows the speed preset. Saves stay about 10 seconds of real time apart: every 256 ticks at 1x, 2,048 at 8x, and 10,240 at 40x or maximum.
   • Normal speed saves on the same ticks as before (79, 335, …).
   • Counting from the last save also stops a soft-paused game, stopped on a save tick, from autosaving every frame.
4. The disk write moves off the game thread.
   • Where the work happens: the game is still serialized between ticks, into memory. A new GAGCore::BackgroundFileWriter then replaces the file through FileManager::writeAtomically on a worker thread.
   • Overlapping saves: a snapshot that hasn't started writing is replaced by a newer one.
   • Thread lifetime: the worker exists only while there's something to write, so waiting leaves no thread behind and a forked process never inherits one.
   • Waits: the engine waits for a pending write before a session ends, before an in-game save, and when GameGUI is destroyed.
   • Faster memory stream: MemoryStreamBackend appends past its end instead of resizing, which zero-filled every byte before writing it. It also gains reserve() and takeContents(), so the snapshot moves to the writer without a copy.
5. Setting to turn autosave off: Settings → Gameplay → Autosave (autosaveGames, on by default), translated for every language.

For review: behavior changes

• Autosave cadence. At faster presets autosaves are fewer in ticks but the same in real time. A crash at 8x can lose up to ~2,048 ticks, about the same ~10 seconds of play as at 1x.
• The new setting is a player-facing feature.
• Autosave hash. Autosave files carry a zero SHA1.
• Joining from an autosave. A client joining a YOG game hosted from an unhashed save always downloads the file.
• Timing of the file update. Autosave now reaches the disk shortly after its tick instead of within it. The late-game writes took 30–70 ms median, up to 187 ms.

Compatibility

• Simulation: unchanged. gd-bigarena-long for 20,000 ticks ends on master's checksum, 5abcb33f. A 3,000-tick run ends on 2a2689ed with autosave on and off.
• Save format and fields: unchanged. The final background-written Auto_save.game is byte-identical to the one this PR's previous commit wrote synchronously.
• Replays: the 20,000-tick replay is byte-identical to master's. The autosave-on and autosave-off replays are byte-identical to each other.
• Network protocol: no message changes. An old client joining a game hosted from a new unhashed autosave still uses the old header comparison; see the known issue below.

Known issue, not fixed here

MapHeader::operator== has compared the SHA1 inverted since 8cf81c9 (2008): std::equal(...)==0 is true only when the hashes differ. So haveMap re-downloads identical files and trusts a different file with the same name, team count and map offset. I'll fix that in a separate PR with a two-client join test.

Verification

All results are from this branch's final tree.

• Harnesses:
  • BufferedFileStreamHarness passes. It now also checks that MemoryStreamBackend overwrites, appends, zero-fills seek gaps and hands over its contents.
  • SavegameSafetyHarness passes, with new checks:
    • background writes keep the newest snapshot, finish on destruction and continue after a failed write
    • the background-written autosave matches serializing straight to a file
    • with autosave turned off, the previous save stays untouched
  • Linux CI runs both harnesses; Windows CI runs the savegame harness.
• Settings: test/run-settings-tests.py and test/run-game-speed-tests.py --settings-only pass. SettingsScreenTest checks that the Autosave toggle persists off and back on.
• Translations: data/check_translations.py --strict gives the same report as before this change, and test/test_translations.py passes.
• Test suite: scons -C test builds, and TestsRunner (187 tests), WinningConditionsHarness and ReplayStepCounterTest pass.
• Server build: succeeds.

Test games

• games/gd-bigarena-long.game: Oazis, 256×256, 11 Castor/Warrush teams.
• A no-Warrush game on the same map:
  GLOB2_TEST_SEED=7 glob2 -test-games-nox 1 --map Oazis --matchup nicowar,econo,nicowar,econo,nicowar,econo,nicowar,econo,nicowar,econo,castor --save-game-as <file>

How the runs were done: release build, glob2 --nox <game> <ticks> 1, with a disposable HOME whose preferences.txt sets gameSpeed and, for the on/off runs, autosaveGames.

Game-thread stall per autosave

Method:

• Inputs: late-game autosaves of the two test games, each loaded and run for 35 seconds at 1x.
• Instrumentation: local tick profiling that isn't part of this PR.
• Excluded: the first autosave after loading, which has no size hint yet.

loaded save	written on the game thread (previous commit)	written in the background (this commit)
gd-bigarena-long, 84 MB	52 ms median, 102 ms worst (31 saves)	27.6 ms median, 39.6 ms worst (34 saves)
no-Warrush game, 128 MB	90 ms median, 138 ms worst (23 saves)	36.4 ms median, 47.2 ms worst (27 saves)

• Previous-commit column: measured in an experiment build that ran both save paths on the same loaded saves.
• What the game thread still does: serialize the game into memory.
• Master, for scale: the stall averaged 260 ms (worst 435 ms) and 378 ms (worst 773 ms) over full 20,000-tick runs of the same two games. That is a different measure, so it isn't in the table.

Limits

• Remaining stall: each late-game autosave still holds the game thread for about 30–50 ms while it serializes into memory, roughly every 10 seconds of real time.
• Memory: a late-game snapshot, up to about 160 MB, stays in memory until it's written.
• At 40x and maximum: the 10,240-tick interval assumes the machine keeps up. A game that can't run that fast saves less often in real time.
• Translations: the two new setting strings are machine-drafted for all 33 languages. The Arabic help text was converted to presentation forms by script, to match that catalog. They need review by native speakers.
• Timing: measured on one macOS machine only.
• Interrupted writes: if the process dies while an autosave is still being written, that snapshot is lost. The previous Auto_save.game survives, because the file is only replaced by rename.
• Settings test in CI: CI doesn't run test/run-settings-tests.py, which covers the new toggle. It passed locally.
• Not tested: a manual two-client YOG join from an unhashed autosave.

🤖 Generated with Claude Code

https://claude.ai/code/session_01LPPkz7Vd7QLMsbQfqy3kHU

[genixpro]

Merged current master (95e24a76) via 3d9909649 to clear the conflict.

Two files conflicted, both append/append:

• src/EngineTiming.h — this branch's autosave cadence constants vs master's GRADIENT_DIRTY_REBUILD_TICKS (from Let a cached route field notice the ground moved, wherever it moved #232). Independent blocks; kept both.
• test/SavegameSafetyHarness.cpp — master added an #include "Version.h" that the file already had a few lines above. Kept the single existing include plus this branch's BackgroundFileWriter.h.

No production logic was touched by the resolution.

Verified locally on the merged head (macOS / Apple M3) before pushing:

• scons -j8 release=1 server=0 savegame-safety-test builds clean.
• python3 test/run-savegame-safety-tests.py build/src/SavegameSafetyHarness — 15 PASS, exit 0.

CI is running on 3d9909649. Still needs a review; requesting a look from @Giszmo.

[Giszmo]

If savegames are up to 160MB, maybe we should save them differently? In a perfect world we could store replays with a timestamp but of course, every logic change breaks such savegames. We could also decide that gradients don't get saved and get computed first thing loading a savegame. That's probably besides the point of this PR but it would be the simpler change.

2. No whole-file SHA1 for autosaves. The header keeps zeros instead. The only reader of that hash is Engine::haveMap, when a client joins a YOG game. It now treats a file without a hash as unverifiable and downloads the host's copy, instead of trusting a local file with the same name. Manual saves, maps and the replay header still hash.

Why not compute the hash instead of downloading the map? It's 200ms to compute a 160MB hash but uploads from the host to 3 players ... takes much longer.

• Overlapping saves: a snapshot that hasn't started writing is replaced by a newer one.

If that happens, something stops writing the savegame within 10s, so it might happen continuously. What happens if the prior has started writing but not ended? Will the next one skip and we drop to every 20s? I guess that would be a fair logic and probably never trigger.

• Autosave hash. Autosave files carry a zero SHA1.

Can't the thread saving the game store that hash? It's literally dirt cheap. If you threw away data needed to hash, it's ok to store a different hash as obviously the data is not needed to play the game. Just to verify it a map was distributed already, so the recipient should be able to hash and verify anyway ...

• Joining from an autosave. A client joining a YOG game hosted from an unhashed save always downloads the file.

NACK