Failing tests on Silicon Mac

[kamidev]

Test environment

➜  zig git:(master) sw_vers; clang --version; zig version
ProductName:        macOS
ProductVersion:        13.2
BuildVersion:        22D49
Homebrew clang version 15.0.7
Target: arm64-apple-darwin22.3.0
Thread model: posix
InstalledDir: /opt/homebrew/opt/llvm/bin
0.11.0-dev.6316+a5b34a61a
➜  zig-gamedev git:(main) ✗ git log -1 | tee
commit 0348e79b3927f6b4e3e635dbe0582cfa48efef36
Author: Michal Ziulek <michalziulek@gmail.com>
Date:   Mon Feb 6 20:26:24 2023 +0100

Updated build scripts to the latest Zig build API changes (part 3, all done except docs).

Steps to reproduce

➜  zig-gamedev git:(main) cd libs/zmath
➜  zmath git:(main) zig build test
Test [7/79] test.zmath.isNearEqual... SKIP
Test [17/79] test.zmath.min... FAIL (TestUnexpectedResult)
/Users/jonas/src/zig/zig/build/stage3/lib/zig/std/testing.zig:509:14: 0x104b9aaf3 in expect (test)
    if (!ok) return error.TestUnexpectedResult;
             ^
/Users/jonas/src/zig/zig-gamedev/libs/zmath/src/zmath.zig:806:9: 0x104ba0403 in test.zmath.min (test)
        try expect(v[1] == 1.0);
        ^
Test [18/79] test.zmath.max... FAIL (TestUnexpectedResult)
/Users/jonas/src/zig/zig/build/stage3/lib/zig/std/testing.zig:509:14: 0x104b9aaf3 in expect (test)
    if (!ok) return error.TestUnexpectedResult;
             ^
/Users/jonas/src/zig/zig-gamedev/libs/zmath/src/zmath.zig:847:9: 0x104ba0acf in test.zmath.max (test)
        try expect(v[1] == 1.0);
        ^
Test [23/79] test.zmath.clamp... FAIL (TestUnexpectedResult)
/Users/jonas/src/zig/zig/build/stage3/lib/zig/std/testing.zig:509:14: 0x104b9aaf3 in expect (test)
    if (!ok) return error.TestUnexpectedResult;
             ^
/Users/jonas/src/zig/zig-gamedev/libs/zmath/src/zmath.zig:1255:9: 0x104ba671f in test.zmath.clamp (test)
        try expect(approxEqAbs(v, f32x4(-100.0, 100.0, -100.0, 0.0), 0.0001));
        ^
Test [25/79] test.zmath.saturate... FAIL (TestUnexpectedResult)
/Users/jonas/src/zig/zig/build/stage3/lib/zig/std/testing.zig:509:14: 0x104b9aaf3 in expect (test)
    if (!ok) return error.TestUnexpectedResult;
             ^
/Users/jonas/src/zig/zig-gamedev/libs/zmath/src/zmath.zig:1297:9: 0x104ba6d3f in test.zmath.saturate (test)
        try expect(approxEqAbs(v, f32x4(0.0, 1.0, 0.0, 0.0), 0.0001));
        ^
Test [27/79] test.zmath.mod... SKIP
Test [38/79] test.zmath.length3... SKIP
72 passed; 3 skipped; 4 failed.

[kamidev]

After discussions on Discord, the most likely cause seem to be Zig miscompilations. No zig-gamedev sample apps seem to be affected, so our short-term fix is to skip these tests on macOS for now.

[Pyrolistical]

Are those zig miscompilations issues tracked in the ziglang repo?

[kamidev]

Are those zig miscompilations issues tracked in the ziglang repo?

No, we were waiting for the next zig release at the time. I'll check if they are still failing once zig-gamedev starts using zig 0.12.

[kamidev]

I tried the four zmath tests we skip on Apple Silicon again.
Using zig ig-gamedev unstable branch and zig 0.13.0-dev.47+c231d9496, all four tests still fail.

[thedaneeffect]

This appears to be resolved as of Zig 0.15.2 on Apple Silicon. All four tests (min, max, clamp, saturate) pass with the aarch64 skip guards removed.

The root cause was a difference in how x86 SSE and ARM NEON handle NaN in min/max operations. x86 MINPS/MAXPS mechanically return the second operand when either input is NaN (Intel SDM), while ARM has two instruction variants: FMIN/FMAX (NaN-propagating) and FMINNM/FMAXNM (returns the number over quiet NaN, per IEEE 754-2008 minNum). The fix likely came through an LLVM upgrade improving how @min/@max lower to ARM instructions.

The skip guards and associated comments can be removed. There are still minor behavioral differences with signaling NaN (snan), but these don't affect the test assertions and snan behavior is implementation-defined across architectures (C WG14 N1011, LLVM #27737).

Tested on: macOS 15.6.1 / Apple M1 / Zig 0.15.2