C++ library broken on big-endian

[Matias314]

Hello, DAVE appears to be non functional on BE systems.

No audio is being sent and when receiving audio I get a loads of these errors:

[2026-05-31 20:15:14.893] [voice] [error] [DAVE] /usr/src/debug/abaddon/abaddon/subprojects/libdave/cpp/src/openssl_cryptor.cpp:193 Failed to finalize decryption
[2026-05-31 20:15:14.894] [voice] [warning] [DAVE] /usr/src/debug/abaddon/abaddon/subprojects/libdave/cpp/src/decryptor.cpp:127 Decrypt failed, no valid cryptor found, type: audio, encrypted frame size: 16, plaintext frame size: 16, number of cryptor managers: 1, pass through enabled: no
[2026-05-31 20:15:14.915] [voice] [error] [DAVE] /usr/src/debug/abaddon/abaddon/subprojects/libdave/cpp/src/openssl_cryptor.cpp:193 Failed to finalize decryption
[2026-05-31 20:15:14.915] [voice] [warning] [DAVE] /usr/src/debug/abaddon/abaddon/subprojects/libdave/cpp/src/decryptor.cpp:127 Decrypt failed, no valid cryptor found, type: audio, encrypted frame size: 16, plaintext frame size: 16, number of cryptor managers: 1, pass through enabled: no
[2026-05-31 20:15:14.937] [voice] [error] [DAVE] /usr/src/debug/abaddon/abaddon/subprojects/libdave/cpp/src/openssl_cryptor.cpp:193 Failed to finalize decryption
[2026-05-31 20:15:14.937] [voice] [warning] [DAVE] /usr/src/debug/abaddon/abaddon/subprojects/libdave/cpp/src/decryptor.cpp:127 Decrypt failed, no valid cryptor found, type: audio, encrypted frame size: 16, plaintext frame size: 16, number of cryptor managers: 1, pass through enabled: no
[2026-05-31 20:15:14.958] [voice] [error] [DAVE] /usr/src/debug/abaddon/abaddon/subprojects/libdave/cpp/src/openssl_cryptor.cpp:193 Failed to finalize decryption
[2026-05-31 20:15:14.958] [voice] [warning] [DAVE] /usr/src/debug/abaddon/abaddon/subprojects/libdave/cpp/src/decryptor.cpp:127 Decrypt failed, no valid cryptor found, type: audio, encrypted frame size: 16, plaintext frame size: 16, number of cryptor managers: 1, pass through enabled: no

[barracuda156]

@catap Kirill, I see you have updated abaddon in OpenBSD, do you have an idea what fails for us here? Apparently neither macOS nor Linux work.

[catap]

@barracuda156 I had commited @Izder456 changes to her port. Better to ask her.

[Matias314]

After some further investigation, it appears all tests pass:

[matias@ArchPOWER-G5 test]$ uname -a
Linux ArchPOWER-G5 6.19.11-1-ppc64 #1 SMP PREEMPT_DYNAMIC Tue, 07 Apr 2026 20:49:56 +0000 ppc64 GNU/Linux
[matias@ArchPOWER-G5 test]$ ./libdave_test 
Running main() from /usr/src/debug/gtest/googletest-1.17.0/googletest/src/gtest_main.cc
[==========] Running 29 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 29 tests from DaveTests
[ RUN      ] DaveTests.RandomOpusFrame
[       OK ] DaveTests.RandomOpusFrame (0 ms)
[ RUN      ] DaveTests.SplitReconstruct
[       OK ] DaveTests.SplitReconstruct (0 ms)
[ RUN      ] DaveTests.H264SliceOneByteExpGolomb
[       OK ] DaveTests.H264SliceOneByteExpGolomb (0 ms)
[ RUN      ] DaveTests.H264ShortIDROneByteExpGolomb
[       OK ] DaveTests.H264ShortIDROneByteExpGolomb (0 ms)
[ RUN      ] DaveTests.H264ShortIDRTwoByteExpGolomb
[       OK ] DaveTests.H264ShortIDRTwoByteExpGolomb (0 ms)
[ RUN      ] DaveTests.H264LongIDROneByteExpGolomb
[       OK ] DaveTests.H264LongIDROneByteExpGolomb (0 ms)
[ RUN      ] DaveTests.H264LongIDRTwoByteExpGolomb
[       OK ] DaveTests.H264LongIDRTwoByteExpGolomb (0 ms)
[ RUN      ] DaveTests.H264EmulationPreventionInEarlyExpGolomb
[       OK ] DaveTests.H264EmulationPreventionInEarlyExpGolomb (0 ms)
[ RUN      ] DaveTests.H264ThreeByteShortCodeExtension
[       OK ] DaveTests.H264ThreeByteShortCodeExtension (0 ms)
[ RUN      ] DaveTests.H264TwoSliceTest
[       OK ] DaveTests.H264TwoSliceTest (0 ms)
[ RUN      ] DaveTests.H265IdrSlice
[       OK ] DaveTests.H265IdrSlice (0 ms)
[ RUN      ] DaveTests.H265TsaSlice
[       OK ] DaveTests.H265TsaSlice (0 ms)
[ RUN      ] DaveTests.H265SimpleThreeByteCodeExtension
[       OK ] DaveTests.H265SimpleThreeByteCodeExtension (0 ms)
[ RUN      ] DaveTests.H265MultipleThreeByteCodeExtensions
[       OK ] DaveTests.H265MultipleThreeByteCodeExtensions (0 ms)
[ RUN      ] DaveTests.H265TwoIdrSlice
[       OK ] DaveTests.H265TwoIdrSlice (0 ms)
[ RUN      ] DaveTests.CryptorManagerCheckMaxGap
(cryptor_manager.cpp:155) Creating cryptor for new generation: 0
(cryptor_manager.cpp:155) Creating cryptor for new generation: 250
(cryptor_manager.cpp:61) Received frame with future generation: 251, newest generation: 0
(cryptor_manager.cpp:123) Reporting cryptor success, generation: 250
(cryptor_manager.cpp:130) Updating expiry for cryptor, generation: 0
(cryptor_manager.cpp:155) Creating cryptor for new generation: 251
[       OK ] DaveTests.CryptorManagerCheckMaxGap (7 ms)
[ RUN      ] DaveTests.CryptorManagerCheckExpiry
(cryptor_manager.cpp:155) Creating cryptor for new generation: 0
(cryptor_manager.cpp:155) Creating cryptor for new generation: 1
(cryptor_manager.cpp:123) Reporting cryptor success, generation: 1
(cryptor_manager.cpp:130) Updating expiry for cryptor, generation: 0
(cryptor_manager.cpp:168) Removing expired cryptor, generation: 0
(cryptor_manager.cpp:176) Deleting key for old generation: 0
(cryptor_manager.cpp:55) Received frame with old generation: 0, oldest generation: 1
[       OK ] DaveTests.CryptorManagerCheckExpiry (0 ms)
[ RUN      ] DaveTests.CryptorManagerDeleteOldKeys
(cryptor_manager.cpp:155) Creating cryptor for new generation: 0
(cryptor_manager.cpp:155) Creating cryptor for new generation: 5
(cryptor_manager.cpp:123) Reporting cryptor success, generation: 5
(cryptor_manager.cpp:130) Updating expiry for cryptor, generation: 0
(cryptor_manager.cpp:168) Removing expired cryptor, generation: 0
(cryptor_manager.cpp:176) Deleting key for old generation: 0
(cryptor_manager.cpp:176) Deleting key for old generation: 1
(cryptor_manager.cpp:176) Deleting key for old generation: 2
(cryptor_manager.cpp:176) Deleting key for old generation: 3
(cryptor_manager.cpp:176) Deleting key for old generation: 4
[       OK ] DaveTests.CryptorManagerDeleteOldKeys (1 ms)
[ RUN      ] DaveTests.CryptorManagerGenerationWrap
[       OK ] DaveTests.CryptorManagerGenerationWrap (0 ms)
[ RUN      ] DaveTests.CryptorManagerBigNonce
[       OK ] DaveTests.CryptorManagerBigNonce (0 ms)
[ RUN      ] DaveTests.CryptorManagerNoReprocess
(cryptor_manager.cpp:155) Creating cryptor for new generation: 0
[       OK ] DaveTests.CryptorManagerNoReprocess (2 ms)
[ RUN      ] DaveTests.PassthroughInOutBuffer
[       OK ] DaveTests.PassthroughInOutBuffer (0 ms)
[ RUN      ] DaveTests.PassthroughTwoBuffers
[       OK ] DaveTests.PassthroughTwoBuffers (0 ms)
[ RUN      ] DaveTests.SilencePacketPassthrough
(decryptor.cpp:27) Transitioning to new key ratchet: 0x11b6c30, expiry: 0
(decryptor.cpp:71) Decrypt skipping silence of size: 3
[       OK ] DaveTests.SilencePacketPassthrough (0 ms)
[ RUN      ] DaveTests.RandomOpusFrameEncryptDecrypt
(decryptor.cpp:27) Transitioning to new key ratchet: 0x11b6430, expiry: 0
(static_key_ratchet.cpp:37) Retrieving static key for generation 0 for user 123456789876543210
(encryptor.cpp:214) Encrypted audio: 12737953859754787279, video: 18417586403877465664. Failed audio: 18417586129140738452, video: 17831250521181126836
(encryptor.cpp:218) Last encrypted frame, type: audio, ssrc: 0, size: 76
(static_key_ratchet.cpp:37) Retrieving static key for generation 0 for user 123456789876543210
(cryptor_manager.cpp:155) Creating cryptor for new generation: 0
(decryptor.cpp:145) Decrypted audio: 8583145553, video: 18417586266437987276. Failed audio: 18417588225084778576, video: 18510192733
[       OK ] DaveTests.RandomOpusFrameEncryptDecrypt (0 ms)
[ RUN      ] DaveTests.XSSLEncryptDecrypt
[       OK ] DaveTests.XSSLEncryptDecrypt (0 ms)
[ RUN      ] DaveTests.XSSLAdditionalDataAuth
(openssl_cryptor.cpp:193) Failed to finalize decryption
[       OK ] DaveTests.XSSLAdditionalDataAuth (0 ms)
[ RUN      ] DaveTests.XSSLKeyDiff
[       OK ] DaveTests.XSSLKeyDiff (0 ms)
[ RUN      ] DaveTests.XSSLNonceDiff
[       OK ] DaveTests.XSSLNonceDiff (0 ms)
[----------] 29 tests from DaveTests (19 ms total)

[----------] Global test environment tear-down
[==========] 29 tests from 1 test suite ran. (19 ms total)
[  PASSED  ] 29 tests.

So it might be an issue with the client I'm using instead of libdave. Unsure at this point.

[barracuda156]

@Matias314 If you can test (not test suite but actual functionality) on the same platform but BE vs LE, that will likely show if endianness is to blame.

[ouwou]

my suspicion:

libdave/cpp/src/encryptor.cpp

Lines 123 to 127 in 52cd56d

	// write the truncated nonce to our temporary full nonce array
	// (since the encryption call expects a full size nonce)
	memcpy(nonceBuffer.data() + kAesGcm128TruncatedSyncNonceOffset,
	&truncatedNonce,
	kAesGcm128TruncatedSyncNonceBytes);

libdave/cpp/src/decryptor.cpp

Lines 168 to 172 in 52cd56d

	// expand the truncated nonce to the full sized one needed for decryption
	auto nonceBuffer = std::array<uint8_t, kAesGcm128NonceBytes>();
	memcpy(nonceBuffer.data() + kAesGcm128TruncatedSyncNonceOffset,
	&truncatedNonce,
	kAesGcm128TruncatedSyncNonceBytes);

tests in xssl_cryptor_tests.cpp would pass since 1. encryptor and decryptor are both running on the same machine 2. nonce is all zeros anyway

libdave/cpp/test/xssl_cryptor_tests.cpp

Line 67 in 52cd56d

auto nonceBuffer = std::vector<uint8_t>(kAesGcm128NonceBytes, 0);

[Izder456]

@catap Kirill, I see you have updated abaddon in OpenBSD, do you have an idea what fails for us here? Apparently neither macOS nor Linux work.

The issue is abaddon's use of mlspp which libdave depends on. It doesn't compile with LibreSSL. The endian-ness isn't related to why i disabled libdave/voice in that port. Nor did i test on BE systems. The version available in -stable OpenBSD doesn't have libdave support which would explain the broken behavior on -stable. See this mailing list message i sent about the port:

https://marc.info/?l=openbsd-ports&m=177561076001057&w=2

[barracuda156]

Firstly, this release includes libdave, in which depends on mlspp. both are needed for voice support, and need to be in the build tree otherwise it tries to fetch it from your installed system

@Izder456 This is true by default, but easy to fix. I build abaddon against external mlspp port (though I still use in-tree libdave at the moment): https://github.com/macos-powerpc/powerpc-ports/blob/649dc8b9760b201fc3caf0e724c68d9a04986f0b/net/abaddon/Portfile

[Izder456]

Firstly, this release includes libdave, in which depends on mlspp. both are needed for voice support, and need to be in the build tree otherwise it tries to fetch it from your installed system

@Izder456 This is true by default, but easy to fix. I build abaddon against external mlspp port (though I still use in-tree libdave at the moment): https://github.com/macos-powerpc/powerpc-ports/blob/649dc8b9760b201fc3caf0e724c68d9a04986f0b/net/abaddon/Portfile

That wouldn't matter anyways, it fails to compile on our LibreSSL in base.

[catap]

On Tue, 02 Jun 2026 21:32:24 +0200, Izzy Meyer ***@***.***> wrote: That wouldn't matter anyways, it fails to compile on our LibreSSL in base.

I wonder why? Do you recall?

…

-- wbr, Kirill

[Izder456]

On Tue, 02 Jun 2026 21:32:24 +0200,
Izzy Meyer @.***> wrote:

That wouldn't matter anyways, it fails to compile on our LibreSSL in base.

I wonder why? Do you recall?

--
wbr, Kirill

It needs some experimental crypto standard that LibreSSL doesn't support (yet), but OpenSSL does. I'll need to check when I'm home from $dayjob