Concurrent `mops build` invocations race on the moc toolchain download and extraction

[marc0olo]

Problem

Two mops build invocations for different canisters in the same project, started concurrently on a cold toolchain cache, intermittently fail with no diagnostics:

> build canister publisher
> Build failed for canister publisher (exit code: undefined)

When the timing lands differently the archive handling fails instead:

Gt [ZlibError]: zlib: unexpected end of file
  code: 'Z_BUF_ERROR',
  errno: -5,
  recoverable: false,
  file: '<project>/.mops/_tmp/motoko-Linux-x86_64-1.14.1.tar.gz',
  cwd: '/github/home/.cache/mops/moc/1.14.1',
  tarCode: 'TAR_ABORT',
  [cause]: Error: unexpected end of file

This is the same class of bug as #472 (per-canister build locks), #532 (shared scratch dirs) and #543 (install cache), but on the toolchain download/extract path, which none of those covered. Still present on 3.2.0.

Root cause

downloadAndExtract in commands/toolchain/toolchain-utils.ts has no inter-process coordination on either of the two paths it touches:

let tmpDir = path.join(getRootDir(), ".mops", "_tmp");
let archive = path.join(tmpDir, path.basename(url));   // (1) same path in every process
fs.mkdirSync(tmpDir, {recursive: true});
fs.writeFileSync(archive, buffer);
fs.mkdirSync(destDir, {recursive: true});
try {
    // ...
    else if (archive.endsWith("tar.gz")) {
        await extractTar({file: archive, cwd: destDir});   // (3) overwrites in place
    }
}
finally {
    deleteSync([tmpDir], {force: true});                   // (2) removes the shared dir
}

1. archive is derived from path.basename(url), so every concurrent process writes the same file. One writeFileSync clobbers an archive a peer is mid-way through reading.
2. The finally block deletes the whole shared tmpDir, not just this process's archive. Whichever process finishes first pulls .mops/_tmp out from under a peer that is still streaming archive through extractTar({file: archive}) — that is exactly the Z_BUF_ERROR / TAR_ABORT above.
3. extractTar writes into a shared destDir in place, so moc and mo-doc are overwritten while another process is exec'ing them.

Evidence for (3): after a failure the cache directory holds moc and mo-doc with the first extraction's mtimes and moc.js with the second's — caught mid-overwrite.

Why the error message is empty

commands/build.ts runs moc through execa(mocPath, args, {reject: false}) and reports result.exitCode, but on this failure execa never gets an exit code, so the message degrades to exit code: undefined. The actual cause is in result.signal and result.shortMessage, and build.ts prints neither — it only prints result.stderr and result.stdout, which are both empty here.

Patching build.ts to dump the whole result shows two variants across runs, both consequences of (3):

{"signal":"SIGKILL","shortMessage":"Command was killed with SIGKILL (Forced termination): .../moc/1.15.0/moc -c --idl ..."}
{"code":"Unknown system error -88","shortMessage":"Command failed with Unknown system error -88: .../moc/1.15.0/moc -c --idl ...\nspawn Unknown system error -88"}

i.e. either the running moc is killed because its executable image was replaced underneath it, or the exec fails outright against a partially written binary. Surfacing signal / shortMessage when exitCode is undefined would have made this diagnosable immediately, independently of the fix below.

Reproduction

A project with two canisters, pinned to a moc version that is not in the cache:

[toolchain]
moc = "1.15.0"

[canisters.a]
main = "a/main.mo"

[canisters.b]
main = "b/main.mo"

rm -rf ~/.cache/mops/moc/1.15.0 .mops/_tmp     # macOS: ~/Library/Caches/mops/moc
mops build a --output .mops/.build &
mops build b --output .mops/.build &
wait

3/6 iterations fail on 3.2.0, 3/5 on 3.1.0. macOS arm64, Node 24.

End-to-end via icp-cli, which is how we hit it — it runs one mops build <canister> per canister concurrently, so a project with two Motoko canisters fails on the first build after checkout. On dfinity/examples' motoko/pub-sub, icp build on a cold cache is 3/3 failures.

Impact

Not only CI: a user cloning a two-canister Motoko example and running icp deploy gets a build failure whose only message is exit code: undefined, with nothing pointing at the toolchain cache.

Our weekly example runs have hit it nine times since late July, always in one of our two two-canister Motoko projects, and the canister that fails is random across runs:

Date	Run	Canister
2026-09-14	34816046576	publisher
2026-09-14	34815779499	caller
2026-08-31	33366235565	callee
2026-08-27	33084691018	callee
2026-08-25	32841108284	callee (ZlibError)
2026-08-24	32750148987	callee (ZlibError)
2026-08-17	32003242298	subscriber
2026-08-03	30791308980	subscriber
2026-07-27	30283146796	subscriber

Suggested fix

Same shape as #543: download to a process-unique temp name, extract into a temp directory and rename it into place, and take a proper-lockfile lock on the version directory so a peer waits rather than re-extracting. The finally cleanup should remove only this process's own archive, never the shared tmpDir.

Separately, and worth doing regardless: report signal / shortMessage from build.ts when exitCode is undefined.

Workaround

mops install once before any concurrent build, which makes the toolchain download happen exactly once. That took the two affected examples from 3/3 failures to 4/4 clean on a cold cache, and is what we have landed in dfinity/examples#1482 until this is fixed upstream.