Add BenchmarkDotNet benchmark project - #12720
Closed
niksedk wants to merge 2 commits into
Closed
Conversation
Introduces tests/benchmarks with a first suite comparing the hand-rolled byte-shifting helpers this branch removed against the BinaryPrimitives calls that replaced them. The legacy implementations are kept as verbatim private copies in the benchmark, since the originals are gone from libse. GlobalSetup asserts both implementations agree on every offset of the test buffer before any timing runs, so the benchmark also guards the byte-exactness claim. Each benchmark sweeps a 64 KB buffer and uses OperationsPerInvoke to normalize to ns/read, keeping the measurement above BenchmarkDotNet's single-call resolution floor. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Runs the benchmark suite on real x64 hardware (linux-x64 and windows-x64) plus macos-arm64 for comparison, since endianness intrinsics differ sharply between arm64 (rev) and x64 (movbe/bswap) and local arm64 numbers do not transfer. Triggers on workflow_dispatch and on PRs touching tests/benchmarks, not on every push - a full run costs a few minutes per runner and the numbers are only useful when someone is looking at them. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds
tests/benchmarks— a BenchmarkDotNet project for libse hot paths — plus aRun benchmarksworkflow, with a first suite motivated by #12716 (replacing hand-rolled byte-shifting helpers withBinaryPrimitives).BinaryPrimitivesBenchmarkscompares the removed helpers against theBinaryPrimitivescalls that replace them. The legacy implementations are kept as verbatim private copies inside the benchmark, since #12716 deletes the originals from libse — so this project builds and runs againstmainas well, independent of whether #12716 merges.GlobalSetupasserts that both implementations agree on every offset of the test buffer before any timing runs, so the benchmark also guards #12716's byte-exactness claim rather than just timing it.Workflow
.github/workflows/benchmarks.ymlruns the suite onlinux-x64,windows-x64andmacos-arm64. It triggers onworkflow_dispatch(with a filter input) and on PRs touchingtests/benchmarks/**— deliberately not on every push, since a full run costs a few minutes per runner and the numbers only matter when someone is reading them. Results go to the job summary and are uploaded as artifacts.Results
Ratio vs the legacy shift-chain, lower is better, 1.00 = legacy baseline. Each benchmark sweeps a 64 KB buffer with
OperationsPerInvokenormalizing back to ns/read.Xeon 6973P
EPYC 7763
M1 (virtual)
BinaryPrimitiveswins in nearly every category, and the win is larger on x64 than arm64 — up to 3x on the 64-bit read. That fitsmovbe/bswapcollapsing a chain of 8 byte loads, shifts and ors into roughly one instruction, so the widest read gains the most.The 16-bit case is a wash everywhere (0.89–1.07); a 2-byte swap was never going to be a bottleneck. The 1.07 on the macos-arm64 runner is noise at sub-nanosecond scale on a shared virtual runner, not a real regression.
These are sub-nanosecond, machine-specific measurements — the ratios are the signal, not the absolute values. Both implementations allocate nothing.
Test plan
dotnet build tests/benchmarks/LibSEBenchmarks.csproj -c Release— clean, 0 warningsmainand on the Replace manual byte-shifting helpers with BinaryPrimitives #12716 branch — identical resultsGlobalSetupequality assertion passes on x64 and arm64BenchmarkDotNet.Artifacts/was already in.gitignoretestsfolder🤖 Generated with Claude Code