[EngineeringWithJay]

What this adds

25 throughput rows (sorted by TG128 into the existing tables) + 10 per-device reports across all three families:

Family	New rows	Hardware
Bonsai 2	8	TITAN V, Tesla P40, RTX A4000 (PTQ1_0+PQ2_0 each), base Apple-M5 Air 16 GB (both bands)
Ternary-Bonsai (gen-1)	4	TITAN V 576.6/49.20, RTX 2060 415.8/25.71 (Q2_g64, fork b10777), M5 Air 80.2/6.68 (upstream b11630) + 34.1/6.06 (fork b9596, legacy Q2_0)
Bonsai 1-bit	3	TITAN V 552.4/56.47, RTX 2060 431.2/29.96, M5 Air 84.2/11.54 (all upstream-compatible)

Cloud measurements: vast.ai pods (destroyed after capture), fork prism-b10777-e3de715 linux-cuda-12.4, llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 3. Model files pinned by sha256 (PTQ1_0 53107f53…, PQ2_0 3907dc16…, gen-1 Q2_g64 59a45d1e…, legacy Q2_0 868c1171…, Q1_0 17ef842e…) — identical digests across every machine.

Findings worth a doc mention

1. Format preference is per-card, not per-family: on our P40 (Pascal) PQ2_0 nearly doubles PTQ1_0 prefill (254.4 vs 139.9) — echoing the table's Blackwell/Ada PQ2_0 rows — while on desktop Volta (TITAN V) and Ampere (A4000) PTQ1_0 takes decode. Notably our TITAN V result disagrees with the table's V100-SXM2 row on decode, so Volta is not internally consistent across tiers; per-card guidance beats family rules.
2. Two version floors, reproduced: upstream ggml 0.15.3 refuses gen-1 ternary files (error: failed to load model); upstream b11630 loads group-64 but cleanly refuses legacy group-128; and official b11630 ubuntu-cuda binaries fail to start in the common vastai/pytorch container (GLIBC_2.38 not found). Users conclude "the model is broken" at the OS level — a Forward-Deployment-grade trap.
3. Type-id collision visible in raw output: llama-bench prints Q2_0 for both the 7.05 GiB group-64 file and the 6.66 GiB legacy group-128 file (sizes distinguish them). All refusals observed were clean errors — no silent misload on our hosts.

Honesty notes

• The M5 Air g64-vs-legacy rows differ in both build and file (confounded) — presented as two independent rows, not a format comparison.
• Passively cooled Air = short-run llama-bench samples (3 reps, mean ± std. dev.); single pod per cloud card; no accuracy/quality measurements at all (nothing vendor-reported is restated as ours).
• gen-1 PQ2_0 (current packing) not yet run on CUDA boxes — follow-up planned; and decode rows for a first batch of cards (Titan Xp, P100, QRTX 5000, A4000 gen-1 prefill) are omitted entirely rather than shown half-complete.

Paper trail (full ± tables, commands, hashes): each report file.

[khosravipasha]

🤖 Reviewing on behalf of the maintainers

Thanks for contributing these benchmarks across so many GPUs and the M5 Air! This is useful coverage and looks good overall, mostly ready to merge.

Just two small things to double-check:

• The two 1-bit CUDA reports list a ternary model filename/hash in their configuration. Could you confirm the runs used the intended 1-bit model and correct those details if they were copied over?
• Please check the comparison wording: the existing 3090 Ti results favor PQ2_0 for decode, and the Titan V 1-bit report uses 51.25 tok/s as the older ternary baseline, but that is the Bonsai 2 result; the older ternary result is 49.20 tok/s. Also, if the Air numbers come directly from llama-bench, please label them mean ± standard deviation rather than medians.

Thanks again for the contribution!

[EngineeringWithJay]

Thanks for the review — all three points fixed in the follow-up commit:

1. The two 1-bit CUDA reports (bonsai/cuda-titanv-12gb-linux.md, bonsai/cuda-rtx2060-12gb-linux.md) now pin the file that was actually run: prism-ml/Bonsai-27B-gguf — Bonsai-27B-Q1_0.gguf, sha256 17ef842e…9aa0. The ternary Q2_g64 pin and the group-64 note were copied from the sibling ternary reports and are removed.
2. Titan V 1-bit report: older-ternary baseline corrected to 49.20 t/s (Q2_g64 on that board); 51.25 was the Bonsai 2 PTQ1_0 result. The claim remains card-scoped — Q1_0 decode 56.47 > 49.20 on the Titan V — and I've kept per-card selection language elsewhere consistent with the 3090 Ti rows favoring PQ2_0.
3. M5 Air rows are llama-bench avg_ts ± stddev_ts over 3 reps: the two Air reports and the PR description now say mean ± std. dev. (no medians), and the cross-referenced Q1_0 row carries its dispersion.

[khosravipasha]

🤖 Reviewing on behalf of the maintainers

Thanks for the quick corrections and for contributing benchmarks across so many devices and model families! The updated reports look good. We resolved the table conflict with current main, preserving your entries and the recently merged B580 results. Approved from our side; a maintainer will take a final pass before merging.

[EngineeringWithJay]

Thanks for the review approval and for resolving the table conflict while preserving both my submitted entries and the B580 results. Glad the corrections addressed the review points. I'll leave the PR for the maintainer's final pass.