Repository navigation
ds4 PR #964: +30% GLM-5.3-Flash decode on Metal (M3 Ultra), bit-exact -- the first outside claim to clear our resolution bar #118
Description
Sweep 2026-09-03T20:47Z. Related to #40 (GLM-5.3 quant ladder on ds4).
The claim
ds4 PR #964 by @trueimage: "Over 30% faster GLM 5.3 Flash decode on Metal (M3 Ultra), bit-exact". antirez/ds4#964
Direct from the PR body, on an Apple M3 Ultra, 80 GPU cores, 512 GB, macOS 26.5.2, GLM-5.3-Flash-Q4_K fully resident, measured with ds4-bench on promessi_sposi.txt, 128 greedy tokens per frontier:
| frontier | main decode | branch decode | delta |
|---|---|---|---|
| 2048 | 20.97 / 21.06 | 27.76 / 27.76 | +32.10% |
| 4096 | 20.67 / 20.73 | 27.06 / 27.05 | +30.70% |
| 8192 | 20.63 / 20.69 | 26.94 / 26.96 | +30.45% |
| 16384 | 20.55 / 20.59 | 26.93 / 26.81 | +30.63% |
Bit-exact. Greedy generation from the tip is byte-identical to main over 128 tokens on 1471, 3841, and 10352-token prompts.
Two non-exact optimisations tried along the way (widened BF16 loads, grouped/split DSA kernel) are not on the default path. Other models (DeepSeek V4 Flash MXFP4, GLM 5.3 IQ2_XXS) untouched — byte-identical to main within 0.5%.
Why this matters for this machine
30% is above #23's ~26% resolution bar with room — this is one of the very few outside claims that would be measurable here at three trials.
- Metal, our platform. Measured on M3 Ultra rather than M5 Max, but per
AGENTS.mdthe mechanism transfers even when the absolute number does not. - Bit-exact — no quality trade to argue about, unlike PR Preflight misses renamed engine workers and GPU-only memory; vllm_spec reads a mid-trial restart as no traffic (follow-up to #951) #953's precision-adjacent claim.
- GLM-5.3-Flash on ds4 measured 22/24 at a 368s median on
glm53ds4— the largest median in our list. If decode picks up 30%, that median moves. - Prefill flat, so this does not confound with the #87 prefill-decay curve.
Real caveats
- Q4_K in the PR, not our on-disk Q2 (
GLM-5.3-Flash-Q2.gguf). The mechanism is DSA attention arithmetic (details in the PR body), which should apply to any GLM-5.3-Flash quant that uses the same attention path — but that is an assumption. Worth confirming in the PR discussion. - Not merged. 41 commits on top of
b0a147a, "open" status. It may land or it may not. - Measured on M3 Ultra 512 GB. Our 128 GB with a 90 GB Q2 model has less headroom and a different memory-bandwidth ratio; the +30% may not transfer 1:1.
- Decode rate, not agent wall time. Repeat of the standing rule — this is a reason to test, not a result to quote.
Suggested order
- Track the PR to merge (
upstream_sweep.pyalready covers ds4 PRs). - When merged, rebuild
~/git/ds4and re-measureglm53ds4under restart-between-trials (#112). Three trials against the current 368s median. No backend x client combination clears 90% reliability with confidence #23 says the gap needs to be ~26% before it is real, and a claimed 30% barely clears it. - If the gain holds on Q2 too, revisit #40's q2 vs q4 comparison — the ranking may move.
Not queued yet. Below #112, #115, #116. And it depends on the PR landing.
Activity
sweep 2026-09-03T20:47Z: record state, and the four issues it filed/u…
NEXT: re-rank queue after #112 server-state confirmation; add #116/#117/
NEXT: pick the top 10 -- diagnose #112 mechanism, then use the clean …
NEXT: close #77, re-scope #112, split successors into #119 and #120
evandhoffman commentedon Sep 4, 2026
The Q2 half is now measured upstream, and @evandhoffman was tagged on it
Two comments landed on antirez/ds4#964 on
2026-09-03 that answer the conditional this issue was filed with — "if it holds on Q2 too,
revisit #40". It holds.
Q2, and the whole gain is decode
@trueimage, 2026-09-03T21:51:54Z, on
GLM-5.3-Flash-Q2.gguf:Prefill does not move at all — four context points, every one within 0.1%. That is a
clean signature: the change is in decode arithmetic and nothing else, which is consistent
with the bit-exact claim and makes it easy to falsify here.
Their method is worth noting because it is better than ours: ABBA order
(
main → branch → branch → main), one process at a time, each figure the mean of twointerleaved runs, and
mainbuilt in a separate worktree so Metal shaders load from thematching tree. That last detail is a trap we have not written down — a shared build tree
would have shaders from one side loaded against the other. See #130.
The comment ends by tagging @evandhoffman, so this is an open invitation to reproduce on
M5 Max, not just news to read.
The follow-up is more interesting than the Q2 result
@trueimage, 2026-09-03T22:19:29Z. They requantized KDA and head tensors from BF16 to Q8
inside the Q4_K file:
Their summary: "Q2 speed with a 'Q4' file. Have not tested quality degradation."
That is #40's question with a concrete recipe attached, and it may dissolve the q2-vs-q4
tradeoff rather than measure it. Commented there.
Scope, before anyone gets excited
transfer to M5 Max; that is an expectation, not a measurement.
KDA/head requantization, which is a different file.
not predict it. A +35% decode figure is a reason to run the suite, not a result.
Unchanged
Still waiting on merge; track via
upstream_sweep.py. When it lands: rebuild~/git/ds4and re-measure
glm53ds4under restart-between-trials against the current 368 s median —now with alternating arm order per #130, since a 35% claim deserves a protocol that
cannot be accused of position bias.
sweep 2026-09-04T04Z: the inbox command hid every read mention
NEXT: re-rank the queue against the goal, not against engine speed
evandhoffman commentedon Sep 4, 2026
Starting the M5 Max reproduction. Method, fixed before any numbers land:
Arms. Two worktrees off the same clone (
~/git/ds4), each built in place withmake -j8 ds4-bench, so each arm reads its ownmetal/*.metalat runtime — twobuilds in one tree would silently share shaders, which is the trap the PR's own
methodology calls out.
main:upstream/main(antirez) — now supportsglm5-next, the architectureour copy of
GLM-5.3-Flash-Q2.ggufdeclarespr964: PR #912: arm A (TR3 control, MiaAI recipe v1.8) rows, 41/42 #964 head8969dbb(glm53flash-metal-exact, 42 commits)Weights. The 90 GB
~/git/ds4/gguf/GLM-5.3-Flash-Q2.gguf— the exact filethe
glm53-q2download target serves, not a re-quant.Decode A/B.
scripts/decode_ab_engine.sh main <tree-main> pr964 <tree-pr964> <gguf>:frontiers 2048→16384 step 2048 (8 points), 128 greedy tokens per frontier
(
ds4_session_argmax_excluding(eos)— greedy by construction), 3 repetitions,interleaved with arm order alternating per repetition (odd A-B, even B-A, per
#130's rule). One corpus (
speed-bench/promessi_sposi.txt, taken from themaintree so it is byte-identical across arms). Paired per-frontier medians via
scripts/decode_ab_report.py, which printspr964/mainas the direction.Bit-exactness. One extra
ds4-bench --show-outputinvocation per arm on thesame corpus and frontiers; it prints each frontier's 128 decoded tokens to stderr
(
gen[ctx=N] decoded text: "…"). The two streams must be byte-identical — thePR reports bit-exact output and that claim is as load-bearing as the speed one.
Machine state. The leftover #77 servers (
ds4-serveron :8000 with MTP on,holding ~74 GiB, plus the tool shim on :8101) are stopped first; preflight before
and after. Metal ceiling 112 GiB (raised, persisted).
The claim under test: +34.8% to +37.3% decode at Q2, prefill flat within
0.1%, measured on M3 Ultra. Every other Apple number on the PR is M3 Ultra;
this is the first M5 Max data point.
Results land here. Nothing gets posted to antirez/ds4 — anything worth telling
@trueimage will be drafted as a comment for Evan to post.
evandhoffman commentedon Sep 4, 2026
M5 Max reproduction results
The claim reproduces at about half strength on this hardware.
scripts/decode_ab_report.pyon the paired sweep (3 reps, 8 frontiers, arm orderalternating per rep; per-frontier paired medians of
gen_steady_tps):Tokens/s are the 128-greedy-token steady rate. The gain is flat across
context — no decay with depth, the same shape as the PR's own curve — and it
holds at every frontier, 8 of 8. A pooled statistic over all 24 paired points
gives the same story from a different angle: mean +17.6%, median +16.9%.
Against the PR's Q2 numbers (+34.8% to +37.3% on M3 Ultra), that is the
claim reproducing direction, shape, and bit-exactness, at roughly 55-60% of
the magnitude.
Prefill is flat, as claimed. Per-frontier paired medians of the prefill
interval rate: 0.990 to 1.013 across the eight frontiers, paired median
-0.3%. The PR's "prefill flat within 0.1%" holds here within 1.3%.
Bit-exactness holds. One extra
ds4-bench --show-outputinvocation perarm at two frontiers — 2048 (full attention) and 16384 (compact DSA) — and the
decoded streams are byte-identical across arms at both;
diffis clean. Onecaveat stated plainly: on this corpus the greedy tail beyond the first few
tokens decodes to empty-text tokens, so the printed stream is short and the
check proves the printed bytes identical while token-ID identity is strongly
suggested, not strictly proven (two distinct empty-text tokens would print
identically). The PR's own exactness evidence covers rich-text prompts at
1,471/3,841/10,352 tokens; this corroborates it on this hardware and across
both attention regimes.
Why about half the magnitude — one verified fact and no guess
This machine reports Apple M5 Max, 40 GPU cores. Every number on the PR is
from an M3 Ultra, 80 GPU cores — twice the cores. An optimization that wins
by improving occupancy or parallelism should pay less on half the cores, which
fits what we measured; one that wins by cutting memory traffic should pay more
on the narrower part, which does not fit. That does not settle the mechanism,
but it is the cleanest available discriminator between the two.
A second candidate — a stale baseline absorbing part of the PR — is ruled out:
git merge-base pr-964 upstream/mainis exactlyb0a147a= currentupstream/main, 42 commits ahead and zero behind. We measured the PR againstits own base.
What the machine did, and two corrections to the method comment above
glm53flash-metal-exact), confirmed unchangedat run time;
mainis b0a147a. Two worktrees, each built in place, soeach arm reads its own
metal/*.metal— verified differing shader filesbetween the trees.
Makefile has no header-dependency tracking (
grep -MMDfinds nothing), so anincremental build after the
ab06d19 -> b0a147acheckout left 3 objects(
ds4_gpu_args.o,ds4_ssd.o,ds4_layer_pack.o) compiled against the oldds4.h. Nothing had been measured yet; both arms are full clean builds ofthe same vintage now. Worth knowing for anyone reproducing:
make cleanbefore the A/B, or the baseline links mixed-vintage objects silently.
second and pages fault in during inference. There is no "load 2 is faster"
signal to read; don't look for one.
faulting in): baseline at 2048 went 30.22 -> 27.42 -> 27.50 t/s, pr964
36.40 -> 33.71 -> 32.17. Both arms drifted together, which is why the paired
design holds — the per-frontier medians absorb it, and the ratio moves about
2 pp. Reps 2-3 alone give +18.1% / +17.3%, consistent with the +20.0%
three-rep figure.
Data:
benchmarks/ds4/decode-ab-964/(12 CSVs, committed). Machine stateduring the run: 128 GiB M5 Max, Metal ceiling 112 GiB (raised, persisted),
swap flat at 2.1 GiB, disk idle during decode (
iostat0.07-0.91 MB/ssamples).
Nothing here gets posted to antirez/ds4 by me. A draft comment for the PR
thread, for Evan to post or not, follows in the next comment.
evandhoffman commentedon Sep 4, 2026
Draft for @evandhoffman to post on antirez/ds4#964 — written here, not posted there. Numbers from this issue's run. (Corrected 2026-09-04: the first version of this draft said +20.0%, a ratio-of-medians artifact of our report script; the paired figure is +16.5%. See the correction comment above.)
Reproduced on M5 Max (128 GB, 40 GPU cores) with
GLM-5.3-Flash-Q2.gguf,PR head
8969dbbagainstmainatb0a147a— two clean worktree builds soeach arm reads its own
metal/*.metal, 128 greedy tokens per frontier,frontiers 2048–16384, 3 repetitions with alternating arm order, ratios paired
within each repetition:
+15.4% to +20.5%), faster at 8 of 8 frontiers, flat across context.
Pooled over all 24 (frontier, repetition) pairs: median +16.9%, mean +17.6%.
approximately flat; a small consistent prefill cost on this machine.
full-attention regime (ctx 2048) and the compact DSA regime (ctx 16384).
Direction, shape (flat across context) and exactness reproduce. Prefill is
within ~2% here rather than the claimed 0.1%. The decode magnitude is about
half the M3 Ultra numbers — consistent with an occupancy/parallelism win
paying less on half the GPU cores (40 vs 80), though we have not isolated
the mechanism.
Method and data: evanwtf/local-llm#118.
One reproduction note worth having in the thread: the ds4 Makefile has no
header-dependency tracking, so anyone building this PR incrementally over an
older checkout can link mixed-vintage objects silently —
make cleanbeforethe A/B.
evandhoffman commentedon Sep 4, 2026
Closing as complete. One correction to the results comment above: the data directory holds 6 A/B CSVs and 4 exactness captures, not 12 CSVs. The durable record is the 2026-09-04 changelog entry (docs/changelog.md), the make-clean trap in docs/m5max-runbook.md, the harness scripts/decode_ab_engine.sh at c1c72b0, and the data in benchmarks/ds4/decode-ab-964/. The draft comment for the upstream PR thread stays in this issue for Evan to post or not.
benchmarks: decode A/B statistic now pairs within reps; #118 correcte…
evandhoffman commentedon Sep 4, 2026
CORRECTION: the +20.0% headline was a statistic defect. The paired figure is +16.5%.
scripts/decode_ab_report.pytook each arm's median independently and divided one by the other. Its own docstring called the result "the paired median ratio"; a paired statistic it was not — the pairing is exactly what two separate medians throw away. With ~9% rep-to-rep drift, the two medians can come from different repetitions, and the drift re-enters the headline as noise. At ctx 2048 the reported 1.226 divided rep 3's baseline by rep 2's branch.The data is unchanged. Only the statistic was wrong. Fixed script (pairing the ratio within each repetition, then taking medians), run on the same 6 committed CSVs — decode:
Prefill corrects too — and it is the bigger change:
The earlier "prefill −0.3%, the claim holds within 1.3%" was the same defect. Paired, prefill is a consistent ~1.5% cost on this machine — small, in the same direction at 8 of 8 frontiers, with the pooled mean pulled to −4.0% by rep 1's cold pages. "Prefill flat within 0.1%" reproduces approximately here, not exactly.
What survives unchanged: 8 of 8 frontiers, flat across context, byte-identical output, the merge-base ruling, the make-clean trap, the method. The magnitude comparison becomes: +16.5% against the PR's M3 Ultra +34.8–37.3% — ~45–50%, still consistent with an occupancy win paying less on 40 cores than on 80. The pooled +16.9%/+17.6% figures were already in the results comment above as a cross-check, which is how this was caught: the headline and the cross-check disagreed, and the cross-check was right.
One more reproduction note: the exactness captures carry the full
ds4-benchstderr, including nondeterministic Metal timing banners (Metal model views created in 2.768 ms...). A whole-filediffof the committed pairs reports a difference on those lines even though everydecoded text:line matches exactly. Compare thedecoded text:lines, not the whole files — the statement "byte-identical" refers to those lines.Fix committed in 98bc79b with tests (tests/test_decode_ab_report.py) pinning the pairing. The prior A/Bs summarized by the same script were recomputed where their CSVs are committed — #52's two August passes (2bf44c1, 95880b4), which read 1.146/1.155 unpaired vs 1.157/1.141 paired (pass-2 prefill 0.998 → 0.979). #91's September runs kept their raw CSVs in /tmp, now gone, so those figures stay unrecomputable — corrections are posted on both issues, and the first versions of those corrections misattributed #52's directories to #91; the commit dates settle it.
The upstream draft below is a file reference; the draft text lives at that path and carries the corrected figures. Nothing has been posted outside this repo.
12 remaining items
decode_ab_report: per-rep ratio and per-arm drift, so they stop being…
scripts: a durable repeat-runner for decode A/Bs (#136)
benchmarks: four q4/q8 runs at 6a20b13 -- 1.155 median, and my audit …
NEXT: issue number, title, and a summary under thirty words
decode_ab_report: --legacy, to answer whether the old statistic hid s…
thermals: record fan RPM next to die temperature (#116, #120)
decode_ab_report: what k runs would have bought (#136 item 2)
evandhoffman commentedon Sep 5, 2026
Re-run requested on a new head — tracked in #143
@trueimage asked on antirez/ds4#964 at 2026-09-05T01:07:47Z:
The PR title now claims 16.6–26.0% faster prefill alongside the decode gain, on head
4b00b59a.This issue is the control. What we measured here, four runs, same harness, same prompt, same machine:
Prefill was 2.0% slower on the branch at the head this issue tested. A ~19–28 pp move on a figure with a known 2.9 pp four-run spread is about as cleanly posed as an outside claim gets, and it is why the re-run is worth the machine time rather than a polite acknowledgement.
Staying closed — the measurement it records stands and is not superseded. #143 carries the new head.
queue: #143 to the top -- @trueimage asked for a re-test on the new d…
decode_ab_engine: refuse a missing build early, and stamp both arms' …
#162: make the draft's raw-data links clickable and name its two stat…
bitexact_ab: run each arm from its own tree, or the Metal arm never s…
decode_ab_engine in Python, and two ordering defects in decode_ab (#235…