ci(bench): WASM + native perf-regression tracking with sticky PR comments

[WiktorStarczewski]

"The perf looks really nice, and we should definitely start tracking WASM performance in the E2E benchmarking infrastructure to make sure it stays that way!"

— @adr1anh on Slack, reviewing #998

This PR is a direct response to that. Adrian's right — measuring a one-time win is less valuable than catching the regression that erodes it back. So this PR sets up automated perf-regression tracking for 0xMiden/crypto, both in WASM (the runtime that ships to wallet/dApp users via V8) and natively (Linux x86_64 GHA runners), with sticky PR comments showing the diff vs the latest next baseline on every single PR going forward.

---

What this PR adds

A new CI workflow (.github/workflows/bench.yml) with two jobs that run on every PR + every push to next, both feeding benchmark-action/github-action-benchmark:

1. bench-wasm — headless-Chromium harness in miden-bench-wasm/. Builds the bench WASM via wasm-pack, drives Chromium via Playwright, runs the benches, posts the medians.
2. bench-native — cargo bench --bench {hash,word,transpose}. Same primitives at native speed for comparison + native-side regression detection.

The crypto repo's first piece of automated perf tracking. No bench had previously been wired into CI — the existing benches/README.md is hand-curated and doesn't catch regressions between runs.

---

Why a sticky PR comment, specifically

When a PR opens, every reviewer asks the same question: "is this PR going to make things slower?" Today that question gets answered the way it always has: someone runs benches by hand, eyeballs the numbers, hopes nothing slipped. That doesn't scale and it's not reliable.

A sticky perf comment puts the answer to that question right in the PR review thread, updated automatically on every push:

| Bench                           | Before  | After   | Δ      |
| ------------------------------- | ------- | ------- | ------ |
| rpo256_merge                    | 4.83 µs | 4.91 µs | +1.7%  |
| rpo256_packed_permute           | 4.85 µs | 2.42 µs | -50.0% ⚡
| lifted_stark_prove_blake3       | 4.63 s  | 3.21 s  | -30.7% ⚡

Six properties make this a load-bearing pattern, not just a nice-to-have:

1. Single source of truth per PR. The comment updates in place on every push, so the review thread doesn't fill with stale snapshots.
2. Diff vs baseline at a glance. Reviewers don't need to know what numbers used to look like — the action computes the delta against the latest next median.
3. Native to the GitHub review flow. No external dashboard to remember, no auth, no "go check Grafana" tax. The signal is where the conversation already is.
4. Symmetric on wins and losses. Same comment that flags a regression also celebrates the optimisation — it's how PR feat(field,crypto,stark): wasm32+simd128 PackedFelt + trait-generic packed Permutation + DEEP/FRI hot-path fix #998 will show its win once both PRs land.
5. Zero maintainer overhead per PR. No manual perf review step. The bot does it.
6. Cheap to ignore false positives. GHA Linux runners share CPUs — single-digit-percent variance is real. We threshold alerts at 10% on both wasm and native, don't fail the workflow, and let humans decide whether the diff is real or noise. False-positive comments are cheap; false-negative regressions are expensive. (See "On the alert threshold itself" below for why 10% and not 15%-20%.)

---

Industry pattern

This pattern (run benches in CI → store medians on a side branch → post a sticky comment with diff vs baseline) is the standard perf-regression-detection setup in 2026 for projects that take perf seriously but don't want a full perf-engineering team. A non-exhaustive list of users of this exact action or its close peers:

• Yarn — uses github-action-benchmark for cross-version perf tracking on package-install workloads.
• realm/SwiftLint — sticky comments on Swift codebase analysis perf.
• hadolint/hadolint — Dockerfile linter, regression alerts on parser benchmarks.
• GoogleChrome/web-vitals — same pattern, sticky comment with median + IQR per metric.
• Pydantic — uses CodSpeed (a SaaS variant of the same pattern) for similar coverage.
• Turborepo — CodSpeed-based, sticky comments on every PR.
• DOMPurify — uses Bencher.dev, another variant of the pattern.

The common insight across all of these: catching a regression at the PR that introduces it is dramatically cheaper than catching it three months later in a release-readiness audit. Every tool that implements this pattern has the same UX shape: bench output → side-branch storage → sticky PR comment → optional alert threshold. We're picking the most-used Rust-friendly variant of it.

---

Three layers of tracking

The bench set was chosen carefully — different layers catch different classes of regression. None of them on its own is sufficient; together they cover the relevant failure modes.

Layer 1 — Public hash-primitive throughput

bench_rpo256_merge / bench_rpo256_sequential_felt_100
bench_rpx256_merge / bench_rpx256_sequential_felt_100
bench_poseidon2_merge / bench_poseidon2_sequential_felt_100
bench_blake3_256_merge / bench_blake3_256_sequential_felt_100
bench_keccak256_merge / bench_keccak256_sequential_felt_100

Tests the public Hash API (merge, hash_elements) end-to-end. Catches general regressions in the hash primitives — anyone who slows down the Rescue round function, the Poseidon2 MDS layer, or the Blake3 chunk processor will show up here.

These benches go through the scalar (WIDTH=1) fast path of the trait-generic Permutation impl. By construction this is byte-identical between next and PR #998 — the const-folded WIDTH=1 path in #998 is explicitly preserved as a zero-cost no-op. So these metrics will NOT show the simd128 win. They're orthogonal to it. They catch a different class of regression.

Layer 2 — Packed-permutation throughput

bench_rpo256_packed_permute
bench_rpx256_packed_permute
bench_poseidon2_packed_permute

These call permute_mut(&mut [<Felt as Field>::Packing; 12]). <Felt as Field>::Packing resolves to:

• Felt on next (no simd128 PR) → WIDTH=1 → scalar permutation. Numerically identical to Layer 1's *_merge benches; useful as a sanity check that the const-folded fast path is intact.
• PackedFelt on PR feat(field,crypto,stark): wasm32+simd128 PackedFelt + trait-generic packed Permutation + DEEP/FRI hot-path fix #998 → WIDTH=2 → packed permutation processing 2 candidates in parallel. ns/iter halves.

The dashboard step-down on these three metrics, the moment #998 lands, is the simd128 win. Automatic. Unambiguous. No bench-source changes required at the PR-merge boundary.

Layer 3 — End-to-end synthetic prove

bench_lifted_stark_prove_blake3

Drives miden-lifted-stark::prove_multi through the complete pipeline: LDE → constraint folding → DEEP composition → FRI → LMCS Merkle commits. Same arithmetic shape any production prove hits, on a synthetic Blake3 AIR with a 4096-row trace.

This is the headline regression-tracking metric. Every PR going forward gets a one-number answer to "did this change slow down the actual prove stack?" — without anyone having to run the wallet's full proving-bench harness manually.

PCS params are calibrated for fast bench runs, NOT production security:

• log_blowup = 1 (vs miden-vm's 3) — smaller LDE, faster bench
• folding_pow_bits = 0 — no PoW, no anti-grinding cost
• num_queries = 27

This is a perf-tracking bench, not a security parameter; only the relative number matters. The arithmetic shape (NTT, ext-field constraint folding, DEEP, FRI, Merkle) is the same as a production prove — that's what makes the metric representative.

The simd128 PR's gain on packed ext-field math should drop this number ~30-50%, since AIR evaluation + NTT (the chunks simd128 actually accelerates) dominate prove time. That gives a third indicator of #998's impact, complementing Layer 2's direct measurement.

Why all three?

Failure mode	Layer 1 catches	Layer 2 catches	Layer 3 catches
Someone slows down RPO round constants	✅	✅	✅
Someone breaks the WIDTH=1 const-fold (regresses scalar to packed-shape codegen)	✅	❌ (also slows down)	✅
Someone breaks packed permutation (e.g. introduces bounds checks)	❌	✅	✅
Someone slows down LDE / FRI / DEEP / Merkle	❌	❌	✅
Someone speeds up the packed permutation lane (e.g. PR #998)	❌	✅ (50%)	✅ (30-50% on prove)

The matrix shows why each layer is load-bearing. Layer 1 alone misses simd128. Layer 2 alone misses non-permutation regressions. Layer 3 alone is too coarse for diagnostics. Together they triangulate.

---

Architecture

miden-bench-wasm/ — new workspace member

miden-bench-wasm/
├── Cargo.toml          # cdylib + rlib, miden-crypto + miden-lifted-stark deps
├── src/lib.rs          # 14 #[wasm_bindgen] bench functions
├── static/bench.html   # ESM loader page (loads pkg/ from wasm-pack)
├── driver.mjs          # Playwright headless-Chromium driver
├── package.json        # Playwright + serve dev deps
├── package-lock.json   # pinned for CI determinism
└── README.md           # local-run instructions, design rationale, tuning

Each bench function takes either (num_batches, batch_size, warmup) (microbenches) or (num_runs, log_n) (synthetic prove). The driver picks per-bench tuning from BENCH_CONFIG so we can re-balance without rebuilding the WASM.

Why headless Chromium and not wasmtime: V8 (what the wallet ships) and Cranelift (wasmtime's codegen) produce meaningfully different machine code from the same wasm32+simd128 input. SIMD codegen quality can differ 1.5–2×. A wasmtime number could miss a real regression that only shows up in V8. For perf signals we'd actually act on, we need to be measuring what users see.

Why no COOP/COEP: every bench in scope is single-threaded pure compute. No SharedArrayBuffer, no Workers, no rayon, no wasm-bindgen-rayon. Skipping COOP/COEP cuts ~3 s of CI time and removes a known browser SW-reload flake.

.github/workflows/bench.yml — new workflow

Two jobs, separately gated, separately commented:

jobs:
  bench-wasm:    # Headless Chromium / V8
  bench-native:  # cargo bench on Linux x86_64

Both run on pull_request + push to next + workflow_dispatch. Each:

1. Runs the relevant bench
2. Uploads raw results as an artifact (full per-batch sample distribution, ~100 KB) — kept 30 days for post-hoc analysis when an alert fires
3. Calls benchmark-action/github-action-benchmark to:
   • Push the median onto the gh-pages/bench/{wasm,native}/ subdirectory (separate from the existing gh-pages/docs/ deploy used by docs.yml — no collision)
   • Post a sticky PR comment with diff vs the latest next baseline
   • Alert (via comment header + cc @maintainer) on regression > 10% (both wasm and native — see below for the philosophy)

fail-on-alert: false deliberately — a regression alert is a signal for human review, not an automatic block. False-positive comments are cheap to dismiss; false-positive-blocked PRs are expensive.

On the alert threshold itself: 10% for both wasm and native. The instinct on shared GHA Linux runners is to widen the threshold to absorb CPU-contention noise — we explicitly don't do that here. A 15-20% regression IS a regression worth investigating; setting the alarm there normalises noise as acceptable and defeats the point. The right move is to drive variance below the threshold by tuning the bench (longer batches, more samples), not to widen the threshold to fit measured variance.

What's already in place to keep variance under 10%:

• 20+ ms per batch (from batch_size × per-op cost) so performance.now()'s ~100 µs precision contributes < 1 % timer noise per batch.
• 50-sample median instead of mean — robust to single-batch outliers from CPU contention.
• Per-bench tuning so each metric's own profile dictates iteration count.

Tightening further is a roadmap (see miden-bench-wasm/README.md "Noise reduction"):

1. iai-callgrind for native — instruction-count benchmarking, fully deterministic, drops native variance to ~0%.
2. Larger / dedicated GHA runners — ubuntu-latest-4-cores or buildjet. Cuts shared-neighbour interference, paid by the minute.
3. CodSpeed for unified WASM + native — SaaS, free for OSS, valgrind-based instruction counting on both sides.
4. Best-of-N per PR — run each bench 3 times, pick the best median. Doubles CI time but drops variance ~30-50 % on the noisiest benches.

auto-push: ${{ github.event_name == 'push' }} — only push events to next write to gh-pages; PR runs read-only compare against the existing baseline.

.gitignore — wasm-pack output, node_modules, results

miden-bench-wasm/static/pkg/, node_modules/, and results.json are all build artifacts. Repo holds the source.

Cargo.toml (workspace) — miden-bench-wasm added to members

One-line addition. The bench crate uses miden-crypto + miden-lifted-stark workspace deps so its versions stay in lockstep with everything else.

---

How to read the comment

Every PR will get two sticky comments (one per job):

• WASM perf (Chromium / V8) — 14 metrics: 10 hash primitive (Layer 1), 3 packed perm (Layer 2), 1 synthetic prove (Layer 3).
• Native perf (Linux x86_64) — hash + word + transpose benchmarks via criterion's --output-format bencher.

For each metric, the comment shows:

• Current run's median
• Baseline median (latest next)
• Percentage delta
• ⚠️ flag if delta exceeds the regression threshold

To investigate a flagged regression:

1. Click the workflow artifact link in the comment to download the raw JSON.
2. The raw JSON has the full per-batch sample distribution — look at IQR / variance to distinguish "real regression" from "noisy CI run".
3. If real, look at the PR's diff for the obvious culprit. Common ones: cache-line breaks, accidental allocation in a hot loop, wrong feature flag.

To tune a bench's iteration count when variance is too high:

1. Edit BENCH_CONFIG in miden-bench-wasm/driver.mjs — bump batch_size (more amortisation) or num_batches (more samples for the median).
2. The 10% threshold is the floor — any need to widen it indicates the bench is too coarse, not that the regression is acceptable.

---

Local validation

Pre-PR sanity check on Apple M5 (results from node driver.mjs):

Bench                                Median (ns/iter)
─────────────────────────────────────────────────────
blake3_256_merge                                57
blake3_256_sequential_felt_100                 660
keccak256_merge                                160
keccak256_sequential_felt_100                  900
poseidon2_merge                              1 700
poseidon2_sequential_felt_100               21 500
rpo256_merge                                10 900
rpo256_sequential_felt_100                 134 250
rpx256_merge                                 5 750
rpx256_sequential_felt_100                  77 250
rpo256_packed_permute                       11 200   ← matches rpo256_merge ✓ (WIDTH=1 on next)
rpx256_packed_permute                        5 800   ← matches rpx256_merge ✓
poseidon2_packed_permute                     1 675   ← matches poseidon2_merge ✓
lifted_stark_prove_blake3              4 631 200 000   (4.63 s — full prove, log_n=12 trace)

Sanity checks pass:

• Layer 1 numbers match the README's CPU comparison table (closest column: Apple M4 Max).
• Layer 2 packed-perm = Layer 1 merge (within IQR) — confirms WIDTH=1 const-fold is intact on next.
• Layer 3 prove fits inside CI budget (~5 s × 5 runs = 25 s for the bench, ~5 min total job runtime).

---

Caveats / known limits

1. First run on next will have an empty baseline. Until the first push to next after merge, PR comparisons will just show "no baseline" for each metric. Not a blocker — fills in automatically on the first next push.
2. GHA Linux runner variance is real. Single-digit-percent swings between runs on identical code are normal. The 10% threshold is the floor — see the dedicated section above for why we drive variance below the threshold (longer batches, 50-sample medians, per-bench tuning) rather than widening it. If 10% turns out to false-positive in practice, the fix is one of the four roadmap items in miden-bench-wasm/README.md (iai-callgrind, larger runners, CodSpeed, best-of-N), not relaxing the threshold.
3. No browser matrix yet. Chromium-only on Linux. Firefox / Safari / WebKit produce different SIMD codegen and would catch engine-specific regressions, but the wallet's user base is dominated by Chromium-based browsers, so this is the right initial scope.
4. Per-phase prove timing not yet exposed. The synthetic prove tracks total wall-clock. Splitting that into LDE / constraint-eval / DEEP / FRI / Merkle phases requires a custom tracing subscriber to capture span timings — ~100 LoC, deliberately deferred. Useful follow-up if a regression alert fires and we want to know which phase moved.

---

Out-of-scope (explicit)

• PackedFelt mul/add micro-benches — would directly measure the simd128 lane. Adding them now would build on PR feat(field,crypto,stark): wasm32+simd128 PackedFelt + trait-generic packed Permutation + DEEP/FRI hot-path fix #998's source, coupling this PR to that one. The Layer 2 packed-perm benches already capture the same signal at one level higher.
• SMT / Merkle Tree / large_smt benches in CI — multi-minute runtimes don't fit per-PR CI. Ideal for a separate nightly bench job (follow-up).
• Browser engine matrix (Firefox, Safari) — useful but expands CI cost; deferred.

---

Files changed

File	Change	Purpose
Cargo.toml	+1 line	Add miden-bench-wasm to workspace members
Cargo.lock	auto	Lockfile updates for new deps
.gitignore	+5 lines	wasm-pack output, node_modules, results.json
.github/workflows/bench.yml	+195 lines (new)	Two-job CI workflow
miden-bench-wasm/Cargo.toml	+66 lines (new)	Crate manifest
miden-bench-wasm/src/lib.rs	+320 lines (new)	14 bench functions
miden-bench-wasm/static/bench.html	+36 lines (new)	ESM loader page
miden-bench-wasm/driver.mjs	+152 lines (new)	Playwright driver
miden-bench-wasm/package.json	+14 lines (new)	npm dev deps
miden-bench-wasm/package-lock.json	+1105 lines (new)	npm lockfile
miden-bench-wasm/README.md	+160 lines (new)	Design + local-run docs

Net: 2 commits, ~2k lines added (most of which is the npm lockfile + the bench harness — actual logic + config is ~600 lines).

---

cc @adr1anh — this is the response to your Slack comment. Want your eyes on it before merge.

[huitseeker]

Keep workflow permissions minimal, pin actions, and avoid mixing untrusted code execution with privileged automation. The neighboring miden-vm non-regression workflows (1 2) are a good local pattern: the benchmark job is read-only, checkout uses persist-credentials: false, and comment/write permissions live in a separate push-only follow-up job.

[huitseeker]

Thanks for the changes. You might want to update the description/title.

[huitseeker]

Could this be gated to same-repo PRs? Fork PRs get a read-only GITHUB_TOKEN, so comment-always: true can make the publish step fail with Resource not accessible by integration when the action tries to write the sticky comment.

[huitseeker]

Could we add set -o pipefail here before the pipeline? Right now a failed cargo bench compile or run can be hidden because tee exits successfully, and then the job can upload/publish a bad bench-native.txt.

[huitseeker]

Could these two publish jobs be serialized before they write gh-pages? On a push run both publishers can fetch the same branch, make separate commits, and then race on push. Making publish-native depend on publish-wasm, or merging the publishers, would avoid non-fast-forward failures and lost benchmark updates.

[huitseeker]

The native publisher handles the first run by checking whether DATA_PATH exists before reading it, but the WASM publisher reads it unconditionally here. On a fresh setup with no gh-pages benchmark history yet, this throws before the intended "skip comment" path; mirroring the native guard keeps the first PR/push from failing.

[huitseeker]

This decision is coming from metadata.json, which was produced in the untrusted benchmark workflow. A fork PR can change the workflow or runner output to make that artifact say event: "push", and then this trusted workflow would publish PR-controlled benchmark data with the base repo token. It would be safer to derive push/comment eligibility from github.event.workflow_run and GitHub API data, then only use artifact metadata after checking it matches the triggering run and PR.