From 7623c5224486006dc557f95c5716bcb74850aeab Mon Sep 17 00:00:00 2001 From: Jay Glidden Date: Wed, 14 Oct 2026 19:54:54 -0700 Subject: [PATCH 1/3] community-benchmarks: TITAN V, RTX 2060, Tesla P40, RTX A4000, base Apple-M5 Air rows Adds 25 throughput rows across all three families (bonsai2: 8, ternary: 4, bonsai 1-bit: 3) + per-device reports (10 new files). Cloud pods (vast.ai) and a base-M5 MacBook Air (16 GB) measured with llama-bench -p 512 -n 128 -r 3 on the prism-b10777-e3de715 fork and upstream b11630 where applicable. Also documents two compatibility floors found during the sweep: - upstream ggml 0.15.3 refuses gen-1 ternary files; b11630 loads group-64, refuses legacy group-128 (clean errors, no silent misload) - official b11630 ubuntu-cuda build requires GLIBC 2.38 (fails to start in the common pytorch container image) Model files pinned by sha256. Full notes per report. --- community-benchmarks/README.md | 17 ++++++++++- community-benchmarks/bonsai/README.md | 3 ++ .../bonsai/cuda-rtx2060-12gb-linux.md | 27 +++++++++++++++++ .../bonsai/cuda-titanv-12gb-linux.md | 27 +++++++++++++++++ .../bonsai/metal-m5-air-16gb-macos.md | 28 ++++++++++++++++++ community-benchmarks/bonsai2/README.md | 12 ++++++-- .../bonsai2/cuda-rtxa4000-16gb-linux.md | 25 ++++++++++++++++ .../bonsai2/cuda-teslap40-24gb-linux.md | 25 ++++++++++++++++ .../bonsai2/cuda-titanv-12gb-linux.md | 25 ++++++++++++++++ .../bonsai2/metal-m5-air-16gb-macos.md | 27 +++++++++++++++++ community-benchmarks/ternary-bonsai/README.md | 6 +++- .../ternary-bonsai/cuda-rtx2060-12gb-linux.md | 27 +++++++++++++++++ .../ternary-bonsai/cuda-titanv-12gb-linux.md | 27 +++++++++++++++++ .../ternary-bonsai/metal-m5-air-16gb-macos.md | 29 +++++++++++++++++++ 14 files changed, 301 insertions(+), 4 deletions(-) create mode 100644 community-benchmarks/bonsai/cuda-rtx2060-12gb-linux.md create mode 100644 community-benchmarks/bonsai/cuda-titanv-12gb-linux.md create mode 100644 community-benchmarks/bonsai/metal-m5-air-16gb-macos.md create mode 100644 community-benchmarks/bonsai2/cuda-rtxa4000-16gb-linux.md create mode 100644 community-benchmarks/bonsai2/cuda-teslap40-24gb-linux.md create mode 100644 community-benchmarks/bonsai2/cuda-titanv-12gb-linux.md create mode 100644 community-benchmarks/bonsai2/metal-m5-air-16gb-macos.md create mode 100644 community-benchmarks/ternary-bonsai/cuda-rtx2060-12gb-linux.md create mode 100644 community-benchmarks/ternary-bonsai/cuda-titanv-12gb-linux.md create mode 100644 community-benchmarks/ternary-bonsai/metal-m5-air-16gb-macos.md diff --git a/community-benchmarks/README.md b/community-benchmarks/README.md index aebcbebf..ee24bcf0 100644 --- a/community-benchmarks/README.md +++ b/community-benchmarks/README.md @@ -19,18 +19,26 @@ for commands, raw output, memory use, and configuration. | `PQ2_0` | NVIDIA RTX 4090 24 GB | llama.cpp CUDA (Windows) | 3,285 | 84.9 | [link](bonsai2/cuda-rtx4090-windows.md) | | `PQ2_0` | NVIDIA RTX 3090 Ti 24 GB | llama.cpp CUDA (Linux) | 1,552 | 81.6 | [link](bonsai2/cuda-rtx3090ti-linux.md) | | `PTQ1_0` | NVIDIA RTX 3090 Ti 24 GB | llama.cpp CUDA (Linux) | 805 | 67.7 | [link](bonsai2/cuda-rtx3090ti-linux.md) | +| `PTQ1_0` | NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 562.2 | 51.25 | [link](bonsai2/cuda-titanv-12gb-linux.md) | +| `PTQ1_0` | NVIDIA RTX A4000 16 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 769.9 | 49.49 | [link](bonsai2/cuda-rtxa4000-16gb-linux.md) | | `PQ2_0` (MTP bundle, plain inference) | AMD Radeon RX 9070 XT 16 GB | llama.cpp ROCm/HIP (Proxmox) | 1,258 | 48.64 | [link](bonsai2/rocm-hip-rx9070xt-proxmox.md) | | `PQ2_0` | AMD Radeon RX 9070 XT 16 GB | llama.cpp ROCm/HIP (Proxmox) | 1,259 | 48.53 | [link](bonsai2/rocm-hip-rx9070xt-proxmox.md) | | `PQ2_0` | AMD Radeon RX 7800 XT 16 GB | llama.cpp ROCm/HIP | 316.3 | 46.8 | [link](bonsai2/rocm-hip-rx7800xt-fedora44.md) | +| `PQ2_0` | NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 528.2 | 46.66 | [link](bonsai2/cuda-titanv-12gb-linux.md) | | `PQ2_0` | NVIDIA Tesla V100-SXM2 16 GB | llama.cpp CUDA (Windows) | 798 | 46.0 | [link](bonsai2/cuda-tesla-v100-windows.md) | | `PQ2_0` | AMD Radeon RX 6800 XT 16 GB | llama.cpp ROCm/HIP (Windows) | 266.4 | 44.9 | [link](bonsai2/rocm-hip-rx6800xt-windows.md) | | `PQ2_0` | NVIDIA RTX 3070 8 GB | llama.cpp CUDA (Windows) | 838.9 | 43.7 | [link](bonsai2/cuda-rtx3070-8gb-windows.md) | +| `PQ2_0` | NVIDIA RTX A4000 16 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 758.2 | 42.42 | [link](bonsai2/cuda-rtxa4000-16gb-linux.md) | | `PTQ1_0` | NVIDIA Tesla V100-SXM2 16 GB | llama.cpp CUDA (Windows) | 852 | 34.3 | [link](bonsai2/cuda-tesla-v100-windows.md) | | `PTQ1_0` | NVIDIA RTX 4070 Laptop 8 GB | llama.cpp CUDA (Windows) | 435.60 | 32.44 | [link](bonsai2/cuda-rtx4070-laptop-windows.md) | | `2-bit` | Apple M3 Max 48 GB | MLX (mlx 0.32.2, mlx-vlm 0.6.17) | 165.9 | 27.1 | [link](bonsai2/mlx-m3-max-48gb-macos.md) | | `PQ2_0` | Apple M3 Max 36 GB | llama.cpp Metal | 162.2 | 24.3 | [link](bonsai2/metal-m3-max-36gb-macos.md) | | `PQ2_0` | AMD BC-250 16 GB UMA | llama.cpp Vulkan (Linux, local patches) | 142.58 | 24.11 | [link](bonsai2/vulkan-bc250-linux.md) | +| `PQ2_0` | Tesla P40 24 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 254.4 | 23.16 | [link](bonsai2/cuda-teslap40-24gb-linux.md) | | `PTQ1_0` | Apple M3 Max 36 GB | llama.cpp Metal | 136.8 | 21.9 | [link](bonsai2/metal-m3-max-36gb-macos.md) | +| `PTQ1_0` | Tesla P40 24 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 139.9 | 19.86 | [link](bonsai2/cuda-teslap40-24gb-linux.md) | +| `PQ2_0` | Apple M5 (Air) 16 GB | llama.cpp Metal (fork b10777) | 88.8 | 7.82 | [link](bonsai2/metal-m5-air-16gb-macos.md) | +| `PTQ1_0` | Apple M5 (Air) 16 GB | llama.cpp Metal (fork b10777) | 79.7 | 7.03 | [link](bonsai2/metal-m5-air-16gb-macos.md) | ## Earlier 27B families @@ -52,19 +60,26 @@ not directly comparable; each report should identify its harness and workload. | Ternary | NVIDIA L40S 48 GB | llama.cpp CUDA | 3,036 | 74.3 | ~150 (2.06x, 2.4x math) | [link](ternary-bonsai/cuda-l40s-linux.md) | | Ternary | NVIDIA RTX 4070 Ti SUPER 16 GB | llama.cpp CUDA (Windows) | 1,717 | 69.6 | | [link](ternary-bonsai/cuda-rtx4070tisuper-windows.md) | | Bonsai (1-bit) | Apple M5 Max 48 GB | llama.cpp Metal | 796 | 63.9 | slower on this HW | [link](bonsai/metal-m5-max-48gb-macos.md) | +| Bonsai (1-bit) | NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 552.4 | 56.47 | | [link](bonsai/cuda-titanv-12gb-linux.md) | +| Ternary | NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 576.6 | 49.20 | | [link](ternary-bonsai/cuda-titanv-12gb-linux.md) | | Ternary | NVIDIA RTX A5000 24 GB | llama.cpp CUDA | 1,036 | 48.2 | | [link](ternary-bonsai/cuda-rtxa5000-ubuntu.md) | | Ternary | Apple M5 Max 48 GB | llama.cpp Metal | 816 | 45.8 | ~1.2x code/math only | [link](ternary-bonsai/metal-m5-max-48gb-macos.md) | | Bonsai (1-bit) | NVIDIA DGX Spark (GB10) | llama.cpp CUDA | 1,024 | 45.4 | ~96.1 (2.21x, code) | [link](bonsai/cuda-gb10-27b-linux.md) | | Ternary | NVIDIA RTX 5060 Ti 16 GB | llama.cpp CUDA | 1,029 | 44.4 | ~79 (1.78x) | [link](ternary-bonsai/cuda-rtx5060ti-linux.md) | +| Bonsai (1-bit) | NVIDIA GeForce RTX 2060 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 431.2 | 29.96 | | [link](bonsai/cuda-rtx2060-12gb-linux.md) | | Ternary | Apple M5 Pro 64 GB | MLX 2-bit | 466 | 29.5 | 34-49 (community dspark-mlx) | [link](ternary-bonsai/mlx-m5-pro-macos.md) | | Ternary | NVIDIA DGX Spark (GB10) | llama.cpp CUDA | 1,005 | 29.2 | ~70.0 (2.45x, code) | [link](ternary-bonsai/cuda-gb10-27b-linux.md) | | Bonsai (1-bit) | NVIDIA GeForce GTX 1080 Ti 11 GB | llama.cpp CUDA | 285 | 28.3 | | [link](bonsai/cuda-gtx1080ti-linux.md) | | Ternary | Apple M5 Pro 64 GB | llama.cpp Metal | 130 | 26.5 | | [link](ternary-bonsai/mlx-m5-pro-macos.md) | +| Ternary | NVIDIA GeForce RTX 2060 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 415.8 | 25.71 | | [link](ternary-bonsai/cuda-rtx2060-12gb-linux.md) | | Ternary | Apple M4 Pro 64 GB | MLX 2-bit | 120 | 24.8 | | [link](ternary-bonsai/mlx-m4-pro-64gb-macos.md) | -| Ternary | Apple M4 Pro 64 GB | llama.cpp Metal | 116 | 19.0 | slower on this HW | [link](ternary-bonsai/metal-m4-pro-64gb-macos.md) | | Ternary | NVIDIA GeForce GTX 1080 Ti 11 GB | llama.cpp CUDA | 278 | 20.5 | | [link](ternary-bonsai/cuda-gtx1080ti-linux.md) | +| Ternary | Apple M4 Pro 64 GB | llama.cpp Metal | 116 | 19.0 | slower on this HW | [link](ternary-bonsai/metal-m4-pro-64gb-macos.md) | | Ternary | Apple M4 24 GB | MLX 2-bit | 65.2 | 12.7 | | [link](ternary-bonsai/mlx-m4-24gb-macos.md) | | Ternary | Apple M3 Pro 18 GB | llama.cpp Metal | 78.6 | 12.6 | | [link](ternary-bonsai/metal-m3-pro-macos.md) | +| Bonsai (1-bit) | Apple M5 (Air) 16 GB | llama.cpp Metal (upstream b11630) | 84.2 | 11.54 | | [link](bonsai/metal-m5-air-16gb-macos.md) | +| Ternary | Apple M5 (Air) 16 GB | llama.cpp Metal (upstream b11630, `Q2_g64`) | 80.2 | 6.68 | | [link](ternary-bonsai/metal-m5-air-16gb-macos.md) | +| Ternary | Apple M5 (Air) 16 GB | llama.cpp Metal (fork b9596, legacy `Q2_0`) | 34.1 | 6.06 | | [link](ternary-bonsai/metal-m5-air-16gb-macos.md) | ## 8B and smaller diff --git a/community-benchmarks/bonsai/README.md b/community-benchmarks/bonsai/README.md index 5b74e0db..12be5c09 100644 --- a/community-benchmarks/bonsai/README.md +++ b/community-benchmarks/bonsai/README.md @@ -11,10 +11,13 @@ Benchmark results submitted by the community running [Bonsai](https://huggingfac | NVIDIA L40S 48 GB | llama.cpp CUDA | 2,937 | 107.5 | [link](cuda-l40s-27b-linux.md) | | NVIDIA RTX 3090 24 GB | llama.cpp CUDA | 1,480 | 89.2 | [link](cuda-rtx3090-linux.md) | | Apple M5 Max 48 GB | llama.cpp Metal | 796 | 63.9 | [link](metal-m5-max-48gb-macos.md) | +| NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 552.4 | 56.47 | [link](cuda-titanv-12gb-linux.md) | | NVIDIA DGX Spark (GB10) | llama.cpp CUDA | 1,024 | 45.4 | [link](cuda-gb10-27b-linux.md) | | NVIDIA RTX 2060 SUPER 8 GB | llama.cpp CUDA (WSL) | 494 | 32.7 | [link](cuda-rtx2060-linux.md) | +| NVIDIA GeForce RTX 2060 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 431.2 | 29.96 | [link](cuda-rtx2060-12gb-linux.md) | | NVIDIA GeForce GTX 1080 Ti 11 GB | llama.cpp CUDA | 285 | 28.3 | [link](cuda-gtx1080ti-linux.md) | | Apple M3 MacBook Air 16 GB | MLX 1-bit | 51.0 | 18.3 | [link](mlx-m3-air-macos.md) | +| Apple M5 (Air) 16 GB | llama.cpp Metal (upstream b11630) | 84.2 | 11.54 | [link](metal-m5-air-16gb-macos.md) | ### 8B and smaller diff --git a/community-benchmarks/bonsai/cuda-rtx2060-12gb-linux.md b/community-benchmarks/bonsai/cuda-rtx2060-12gb-linux.md new file mode 100644 index 00000000..b89f761a --- /dev/null +++ b/community-benchmarks/bonsai/cuda-rtx2060-12gb-linux.md @@ -0,0 +1,27 @@ +# NVIDIA GeForce RTX 2060 12 GB — CUDA — Bonsai-27B (1-bit) + +## Summary + +RTX 2060 12 GB (TU106, Turing sm_75; driver 580.178.04; ~$0.100/hr) measured with the PrismML llama.cpp fork `prism-b10777-e3de715` +(linux-cuda-12.4 assets) on a vast.ai pod, `llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 3`. + +| file | pp512 (t/s) | tg128 (t/s) | +|------|------------:|------------:| +| `Q1_0` (3.53 GiB) | 431.23 ± 4.28 | 29.96 ± 0.05 | + +## Configuration + +- fork `prism-b10777-e3de715` CUDA-12.4 +- Model: `prism-ml/Ternary-Bonsai-27B-gguf` — `Ternary-Bonsai-27B-Q2_g64.gguf` sha256 `59a45d1ecef702b14531b06d22949f33b25c1897da31a8c0b298e01e4d9138eb` (digest identical across the whole fleet). +- Per the formats guide, current builds expect `PQ2_0` (group-128) **and** `Q2_0` group-64; this entry measures the group-64 file on a current fork build. `PQ2_0` (this gen) not yet run — follow-up planned. +- Rented pod destroyed after capture; ± = llama-bench dispersion over 3 reps. + +## Notes + +Same card as the ternary entry: 1-bit gives +3.7% prefill, +17% decode, half the bytes. + +## Hardware + +``` +NVIDIA GeForce RTX 2060, 12288 MiB; driver 580.178.04 +``` diff --git a/community-benchmarks/bonsai/cuda-titanv-12gb-linux.md b/community-benchmarks/bonsai/cuda-titanv-12gb-linux.md new file mode 100644 index 00000000..830060d4 --- /dev/null +++ b/community-benchmarks/bonsai/cuda-titanv-12gb-linux.md @@ -0,0 +1,27 @@ +# NVIDIA TITAN V 12 GB — CUDA — Bonsai-27B (1-bit) + +## Summary + +TITAN V 12 GB (GV100, Volta sm_70, HBM2; driver 580.173.02; ~$0.114/hr) measured with the PrismML llama.cpp fork `prism-b10777-e3de715` +(linux-cuda-12.4 assets) on a vast.ai pod, `llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 3`. + +| file | pp512 (t/s) | tg128 (t/s) | +|------|------------:|------------:| +| `Q1_0` (3.53 GiB) | 552.38 ± 0.51 | 56.47 ± 0.10 | + +## Configuration + +- fork `prism-b10777-e3de715` CUDA-12.4; upstream b11630 runs `Q1_0` too (fully merged) +- Model: `prism-ml/Ternary-Bonsai-27B-gguf` — `Ternary-Bonsai-27B-Q2_g64.gguf` sha256 `59a45d1ecef702b14531b06d22949f33b25c1897da31a8c0b298e01e4d9138eb` (digest identical across the whole fleet). +- Per the formats guide, current builds expect `PQ2_0` (group-128) **and** `Q2_0` group-64; this entry measures the group-64 file on a current fork build. `PQ2_0` (this gen) not yet run — follow-up planned. +- Rented pod destroyed after capture; ± = llama-bench dispersion over 3 reps. + +## Notes + +3.53 GiB leaves ~8 GB for context on a 12 GB card; decode beats the same machine's ternary row (51.25 t/s, [../ternary-bonsai/cuda-titanv-12gb-linux.md]). + +## Hardware + +``` +NVIDIA TITAN V, 12288 MiB; driver 580.173.02 +``` diff --git a/community-benchmarks/bonsai/metal-m5-air-16gb-macos.md b/community-benchmarks/bonsai/metal-m5-air-16gb-macos.md new file mode 100644 index 00000000..0396fecf --- /dev/null +++ b/community-benchmarks/bonsai/metal-m5-air-16gb-macos.md @@ -0,0 +1,28 @@ +# Apple M5 (MacBook Air, 16 GB) — Metal — Bonsai-27B (1-bit) + +## Summary + +Base M5 Air, 16 GB unified. Upstream llama.cpp **b11630** (no fork needed — `Q1_0` is fully merged +upstream), `-ngl 99 -fa 1 -p 512 -n 128 -r 3`: + +| file | pp512 (t/s) | tg128 (t/s) | +|------|------------:|------------:| +| `Q1_0` (3.53 GiB) | 84.18 ± 0.36 | 11.54 ± 0.08 | + +Decode 1.7x the same laptop's ternary rows; the 27B sits in 3.53 GiB — twice over, with KV headroom. + +## Configuration + +- Command: `LD_LIBRARY_PATH=./bin/b11630 ./bin/b11630/llama-bench -m Bonsai-27B-Q1_0.gguf -ngl 99 -fa 1 -p 512 -n 128 -r 3` (build 6868a52cb). +- File: `prism-ml/Bonsai-27B-gguf` — `Bonsai-27B-Q1_0.gguf` sha256 `17ef842e47450caeb8eaa3ebfbbab5d2f2278b62b79be107985fb69a2f819aa0`. +- 27B only on this box (8B/4B/1.7B skipped: passive cooling makes long sweeps thermally noisy). + +## Notes + +- The whole point of 1-bit: a 27B-class model that fits *twice* in 16 GB with room for context, on upstream llama.cpp, in a $999 laptop. + +## Hardware + +``` +Apple M5 (base), 16 GB unified, Metal4/tensor-API +``` diff --git a/community-benchmarks/bonsai2/README.md b/community-benchmarks/bonsai2/README.md index b2cc92ad..d76c210a 100644 --- a/community-benchmarks/bonsai2/README.md +++ b/community-benchmarks/bonsai2/README.md @@ -17,23 +17,31 @@ Benchmark results submitted by the community running | `PQ2_0` | NVIDIA RTX 4090 24 GB | llama.cpp CUDA (Windows) | 3,285 | 84.9 | [link](cuda-rtx4090-windows.md) | | `PQ2_0` | NVIDIA RTX 3090 Ti 24 GB | llama.cpp CUDA (Linux) | 1,552 | 81.6 | [link](cuda-rtx3090ti-linux.md) | | `PTQ1_0` | NVIDIA RTX 3090 Ti 24 GB | llama.cpp CUDA (Linux) | 805 | 67.7 | [link](cuda-rtx3090ti-linux.md) | +| `PTQ1_0` | NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 562.2 | 51.25 | [link](cuda-titanv-12gb-linux.md) | +| `PTQ1_0` | NVIDIA RTX A4000 16 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 769.9 | 49.49 | [link](cuda-rtxa4000-16gb-linux.md) | | `PQ2_0` (community MTP file, plain inference) | NVIDIA RTX 5070 Ti Laptop 12 GB | llama.cpp CUDA (Windows) | 1,135 | 49.0 | [link](cuda-rtx5070ti-laptop-windows.md) | +| `PTQ1_0` | NVIDIA RTX 5070 Ti Laptop 12 GB | llama.cpp CUDA (Windows) | 527 | 49.0 | [link](cuda-rtx5070ti-laptop-windows.md) | | `PQ2_0` (MTP bundle, plain inference) | AMD Radeon RX 9070 XT 16 GB | llama.cpp ROCm/HIP (Proxmox) | 1,258 | 48.64 | [link](rocm-hip-rx9070xt-proxmox.md) | | `PQ2_0` | AMD Radeon RX 9070 XT 16 GB | llama.cpp ROCm/HIP (Proxmox) | 1,259 | 48.53 | [link](rocm-hip-rx9070xt-proxmox.md) | -| `PTQ1_0` | NVIDIA RTX 5070 Ti Laptop 12 GB | llama.cpp CUDA (Windows) | 527 | 49.0 | [link](cuda-rtx5070ti-laptop-windows.md) | | `PQ2_0` | AMD Radeon RX 7800 XT 16 GB | llama.cpp ROCm/HIP | 316.3 | 46.8 | [link](rocm-hip-rx7800xt-fedora44.md) | +| `PQ2_0` | NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 528.2 | 46.66 | [link](cuda-titanv-12gb-linux.md) | | `PQ2_0` | NVIDIA Tesla V100-SXM2 16 GB | llama.cpp CUDA (Windows) | 798 | 46.0 | [link](cuda-tesla-v100-windows.md) | | `PQ2_0` | AMD Radeon RX 6800 XT 16 GB | llama.cpp ROCm/HIP (Windows) | 266.4 | 44.9 | [link](rocm-hip-rx6800xt-windows.md) | | `PQ2_0` | NVIDIA RTX 3070 8 GB | llama.cpp CUDA (Windows) | 838.9 | 43.7 | [link](cuda-rtx3070-8gb-windows.md) | +| `PQ2_0` | NVIDIA RTX A4000 16 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 758.2 | 42.42 | [link](cuda-rtxa4000-16gb-linux.md) | | `PTQ1_0` | NVIDIA Tesla V100-SXM2 16 GB | llama.cpp CUDA (Windows) | 852 | 34.3 | [link](cuda-tesla-v100-windows.md) | | `PTQ1_0` | NVIDIA RTX 4070 Laptop 8 GB | llama.cpp CUDA (Windows) | 435.60 | 32.44 | [link](cuda-rtx4070-laptop-windows.md) | | `2-bit` | Apple M3 Max 48 GB | MLX (mlx 0.32.2, mlx-vlm 0.6.17) | 165.9 | 27.1 | [link](mlx-m3-max-48gb-macos.md) | | `PQ2_0` | Apple M3 Max 36 GB | llama.cpp Metal | 162.2 | 24.3 | [link](metal-m3-max-36gb-macos.md) | | `PQ2_0` | AMD BC-250 16 GB UMA | llama.cpp Vulkan (Linux, local patches) | 142.58 | 24.11 | [link](vulkan-bc250-linux.md) | +| `PQ2_0` | Tesla P40 24 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 254.4 | 23.16 | [link](cuda-teslap40-24gb-linux.md) | | `PTQ1_0` | Apple M3 Max 36 GB | llama.cpp Metal | 136.8 | 21.9 | [link](metal-m3-max-36gb-macos.md) | | `PQ2_0` | Apple M4 Pro 64 GB | llama.cpp Metal | 126.9 | 20.5 | [link](metal-m4-pro-64gb-macos.md) | -| `PTQ1_0` | Apple M4 Pro 64 GB | llama.cpp Metal | 98.6 | 17.3 | [link](metal-m4-pro-64gb-macos.md) | +| `PTQ1_0` | Tesla P40 24 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 139.9 | 19.86 | [link](cuda-teslap40-24gb-linux.md) | | `2-bit` | Apple M4 Pro 64 GB | MLX (mlx_lm 0.31.3) | 97.9 | 18.9 | [link](metal-m4-pro-64gb-macos.md) | +| `PTQ1_0` | Apple M4 Pro 64 GB | llama.cpp Metal | 98.6 | 17.3 | [link](metal-m4-pro-64gb-macos.md) | +| `PQ2_0` | Apple M5 (Air) 16 GB | llama.cpp Metal (fork b10777) | 88.8 | 7.82 | [link](metal-m5-air-16gb-macos.md) | +| `PTQ1_0` | Apple M5 (Air) 16 GB | llama.cpp Metal (fork b10777) | 79.7 | 7.03 | [link](metal-m5-air-16gb-macos.md) | ## How to Submit diff --git a/community-benchmarks/bonsai2/cuda-rtxa4000-16gb-linux.md b/community-benchmarks/bonsai2/cuda-rtxa4000-16gb-linux.md new file mode 100644 index 00000000..a1a9a1b1 --- /dev/null +++ b/community-benchmarks/bonsai2/cuda-rtxa4000-16gb-linux.md @@ -0,0 +1,25 @@ +# NVIDIA RTX A4000 16 GB — CUDA — Bonsai 2 27B (Ternary-Bonsai-2) + +## Summary + +RTX A4000 16 GB (GA104, Ampere pro), driver 580.173.02, ~$0.116/hr on vast.ai. llama.cpp **PrismML fork `prism-b10777-e3de715`** (linux-cuda-12.4) on vast.ai, +`llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 3`. Both current bands measured on the same machine: + +| Band | pp512 (t/s) | tg128 (t/s) | +|------|------------:|------------:| +| `PTQ1_0` (5.53 GiB) | 769.94 ± 22.99 | 49.49 ± 0.11 | +| `PQ2_0` (6.70 GiB) | 758.24 ± 19.84 | 42.42 ± 0.10 | + +`PTQ1_0` edges prefill and takes decode — consistent with the consumer-Ampere (3090 Ti) and desktop-Volta pattern above. 16 GB leaves room for longer context with the 5.53 GiB file. + +## Configuration + +- Files: `prism-ml/Ternary-Bonsai-2-27B-gguf` — `PTQ1_0` sha256 `53107f530aa52eb00912263ab1ee29bd199261c87cd7b4ad4ca1318c1fe33ee3`; `PQ2_0` sha256 `3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1` (identical digests across all machines in this sweep). +- Upstream b11630 cannot load either band (fork-only) and its ubuntu-cuda build additionally cannot start in the `vastai/pytorch` container (`GLIBC_2.38 not found`). +- Rented cloud pod, destroyed after capture. ± = llama-bench dispersion over 3 reps. + +## Hardware + +``` +NVIDIA RTX A4000, 16376 MiB; driver 580.173.02; container vastai/pytorch:latest +``` diff --git a/community-benchmarks/bonsai2/cuda-teslap40-24gb-linux.md b/community-benchmarks/bonsai2/cuda-teslap40-24gb-linux.md new file mode 100644 index 00000000..a7d36375 --- /dev/null +++ b/community-benchmarks/bonsai2/cuda-teslap40-24gb-linux.md @@ -0,0 +1,25 @@ +# Tesla P40 24 GB — CUDA — Bonsai 2 27B (Ternary-Bonsai-2) + +## Summary + +Tesla P40 24 GB (GP102, Pascal sm_61), driver 580.126.09, ~$0.107/hr on vast.ai. llama.cpp **PrismML fork `prism-b10777-e3de715`** (linux-cuda-12.4) on vast.ai, +`llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 3`. Both current bands measured on the same machine: + +| Band | pp512 (t/s) | tg128 (t/s) | +|------|------------:|------------:| +| `PTQ1_0` (5.53 GiB) | 139.85 ± 0.08 | 19.86 ± 0.09 | +| `PQ2_0` (6.70 GiB) | 254.38 ± 0.62 | 23.16 ± 0.00 | + +**Pascal strongly prefers `PQ2_0`** — 1.8x the prefill of `PTQ1_0` on this card, matching the `PQ2_0` prefill dominance on the newer Blackwell/Ada entries already in this table. + +## Configuration + +- Files: `prism-ml/Ternary-Bonsai-2-27B-gguf` — `PTQ1_0` sha256 `53107f530aa52eb00912263ab1ee29bd199261c87cd7b4ad4ca1318c1fe33ee3`; `PQ2_0` sha256 `3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1` (identical digests across all machines in this sweep). +- Upstream b11630 cannot load either band (fork-only) and its ubuntu-cuda build additionally cannot start in the `vastai/pytorch` container (`GLIBC_2.38 not found`). +- Rented cloud pod, destroyed after capture. ± = llama-bench dispersion over 3 reps. + +## Hardware + +``` +Tesla P40, 24576 MiB; driver 580.126.09; container vastai/pytorch:latest +``` diff --git a/community-benchmarks/bonsai2/cuda-titanv-12gb-linux.md b/community-benchmarks/bonsai2/cuda-titanv-12gb-linux.md new file mode 100644 index 00000000..08c06718 --- /dev/null +++ b/community-benchmarks/bonsai2/cuda-titanv-12gb-linux.md @@ -0,0 +1,25 @@ +# NVIDIA TITAN V 12 GB — CUDA — Bonsai 2 27B (Ternary-Bonsai-2) + +## Summary + +TITAN V 12 GB (GV100, Volta sm_70, HBM2), driver 580.173.02, ~$0.114/hr on vast.ai. llama.cpp **PrismML fork `prism-b10777-e3de715`** (linux-cuda-12.4) on vast.ai, +`llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 3`. Both current bands measured on the same machine: + +| Band | pp512 (t/s) | tg128 (t/s) | +|------|------------:|------------:| +| `PTQ1_0` (5.53 GiB) | 562.22 ± 7.73 | 51.25 ± 0.03 | +| `PQ2_0` (6.70 GiB) | 528.20 ± 0.72 | 46.66 ± 0.05 | + +On this desktop-Volta board `PTQ1_0` takes both columns. The table's V100-SXM2 rows disagree on decode (`PQ2_0` 46.0 > `PTQ1_0` 34.3) — Volta is not internally consistent across product tiers, so selection should be per measured card. + +## Configuration + +- Files: `prism-ml/Ternary-Bonsai-2-27B-gguf` — `PTQ1_0` sha256 `53107f530aa52eb00912263ab1ee29bd199261c87cd7b4ad4ca1318c1fe33ee3`; `PQ2_0` sha256 `3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1` (identical digests across all machines in this sweep). +- Upstream b11630 cannot load either band (fork-only) and its ubuntu-cuda build additionally cannot start in the `vastai/pytorch` container (`GLIBC_2.38 not found`). +- Rented cloud pod, destroyed after capture. ± = llama-bench dispersion over 3 reps. + +## Hardware + +``` +NVIDIA TITAN V, 12288 MiB; driver 580.173.02; container vastai/pytorch:latest +``` diff --git a/community-benchmarks/bonsai2/metal-m5-air-16gb-macos.md b/community-benchmarks/bonsai2/metal-m5-air-16gb-macos.md new file mode 100644 index 00000000..d8464cf1 --- /dev/null +++ b/community-benchmarks/bonsai2/metal-m5-air-16gb-macos.md @@ -0,0 +1,27 @@ +# Apple M5 (MacBook Air, 16 GB) — Metal — Bonsai 2 27B + +## Summary + +Base-model M5 Air, 16 GB unified memory. PrismML fork `prism-b10777-e3de715` (macos-arm64), +`llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 3`: + +| Band | pp512 (t/s) | tg128 (t/s) | +|------|------------:|------------:| +| `PTQ1_0` (5.53 GiB) | 79.71 ± 0.12 | 7.03 ± 0.05 | +| `PQ2_0` (6.70 GiB) | 88.75 ± 0.51 | 7.82 ± 0.06 | + +On this silicon `PQ2_0` takes both columns, mirroring the M4 Pro entries. Cross-generation context on +the same laptop: gen-1 upstream `Q2_g64` = 80.2/6.7 and 1-bit `Q1_0` = 84.2/11.5, so gen-2 decode lands +between the gen-1 paths while keeping the smallest 27B ternary footprint at 5.53 GiB. + +## Configuration + +- `prism-ml/Ternary-Bonsai-2-27B-gguf`; sha256 `53107f53…` (PTQ1_0), `3907dc16…` (PQ2_0) — same digests as the CUDA machines. +- Upstream b11630 refuses both files (`llama_model_load: error loading model`) — fork-only bands, clean refusal. +- Passively cooled Air; 3-rep medians, not sustained thermals. + +## Hardware + +``` +Apple M5 (base), 16 GB unified, Metal4/tensor-API, recommendedMaxWorkingSetSize ~12.7 GB +``` diff --git a/community-benchmarks/ternary-bonsai/README.md b/community-benchmarks/ternary-bonsai/README.md index f7228acb..db91d20b 100644 --- a/community-benchmarks/ternary-bonsai/README.md +++ b/community-benchmarks/ternary-bonsai/README.md @@ -12,19 +12,23 @@ Benchmark results submitted by the community running [Ternary-Bonsai](https://hu | NVIDIA RTX 3090 24 GB | llama.cpp CUDA | 1,478 | 80.5 | | [link](cuda-rtx3090-linux.md) | | NVIDIA L40S 48 GB | llama.cpp CUDA | 3,036 | 74.3 | ~133-175 (1.8-2.4x) | [link](cuda-l40s-linux.md) | | NVIDIA RTX 4070 Ti SUPER 16 GB | llama.cpp CUDA (Windows) | 1,717 | 69.6 | | [link](cuda-rtx4070tisuper-windows.md) | +| NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 576.6 | 49.20 | | [link](cuda-titanv-12gb-linux.md) | | NVIDIA RTX A5000 24 GB | llama.cpp CUDA | 1,036 | 48.2 | | [link](cuda-rtxa5000-ubuntu.md) | | Apple M5 Max 48 GB | llama.cpp Metal | 816 | 45.8 | ~1.2x code/math only | [link](metal-m5-max-48gb-macos.md) | | NVIDIA RTX 5060 Ti 16 GB | llama.cpp CUDA | 1,029 | 44.4 | ~79 (1.78x) | [link](cuda-rtx5060ti-linux.md) | | Apple M5 Pro 64 GB | MLX 2-bit | 466 | 29.5 | 34-49 (community dspark-mlx) | [link](mlx-m5-pro-macos.md) | | NVIDIA DGX Spark (GB10) | llama.cpp CUDA | 1,005 | 29.2 | ~70.0 (2.45x) | [link](cuda-gb10-27b-linux.md) | | Apple M5 Pro 64 GB | llama.cpp Metal | 130 | 26.5 | | [link](mlx-m5-pro-macos.md) | +| NVIDIA GeForce RTX 2060 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 415.8 | 25.71 | | [link](cuda-rtx2060-12gb-linux.md) | | Apple M4 Pro 64 GB | MLX 2-bit | 120 | 24.8 | | [link](mlx-m4-pro-64gb-macos.md) | -| Apple M4 Pro 64 GB | llama.cpp Metal | 116 | 19.0 | slower on this HW | [link](metal-m4-pro-64gb-macos.md) | | NVIDIA GeForce GTX 1080 Ti 11 GB | llama.cpp CUDA | 278 | 20.5 | | [link](cuda-gtx1080ti-linux.md) | +| Apple M4 Pro 64 GB | llama.cpp Metal | 116 | 19.0 | slower on this HW | [link](metal-m4-pro-64gb-macos.md) | | Apple M1 Pro 32 GB | MLX 2-bit | 65.1 | 15.0 | ~10 MLX / ~11 code llama.cpp (net slowdown on MLX) | [link](mlx-m1-pro-32gb-macos.md) | | Apple M4 24 GB | MLX 2-bit | 65.2 | 12.7 | | [link](mlx-m4-24gb-macos.md) | | Apple M3 Pro 18 GB | llama.cpp Metal | 78.6 | 12.6 | | [link](metal-m3-pro-macos.md) | | Apple M3 MacBook Air 16 GB | MLX 2-bit | 48.4 | 9.1 | | [link](mlx-m3-air-macos.md) | +| Apple M5 (Air) 16 GB | llama.cpp Metal (upstream b11630, `Q2_g64`) | 80.2 | 6.68 | | [link](metal-m5-air-16gb-macos.md) | +| Apple M5 (Air) 16 GB | llama.cpp Metal (fork b9596, legacy `Q2_0`) | 34.1 | 6.06 | | [link](metal-m5-air-16gb-macos.md) | ### 8B and smaller diff --git a/community-benchmarks/ternary-bonsai/cuda-rtx2060-12gb-linux.md b/community-benchmarks/ternary-bonsai/cuda-rtx2060-12gb-linux.md new file mode 100644 index 00000000..72776d4f --- /dev/null +++ b/community-benchmarks/ternary-bonsai/cuda-rtx2060-12gb-linux.md @@ -0,0 +1,27 @@ +# NVIDIA GeForce RTX 2060 12 GB — CUDA — Ternary-Bonsai-27B + +## Summary + +RTX 2060 12 GB (TU106, Turing sm_75; driver 580.178.04; ~$0.100/hr) measured with the PrismML llama.cpp fork `prism-b10777-e3de715` +(linux-cuda-12.4 assets) on a vast.ai pod, `llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 3`. + +| file | pp512 (t/s) | tg128 (t/s) | +|------|------------:|------------:| +| `Q2_g64` (7.05 GiB) | 415.79 ± 5.51 | 25.71 ± 0.04 | + +## Configuration + +- fork `prism-b10777-e3de715` CUDA-12.4 prebuilt +- Model: `prism-ml/Ternary-Bonsai-27B-gguf` — `Ternary-Bonsai-27B-Q2_g64.gguf` sha256 `59a45d1ecef702b14531b06d22949f33b25c1897da31a8c0b298e01e4d9138eb` (digest identical across the whole fleet). +- Per the formats guide, current builds expect `PQ2_0` (group-128) **and** `Q2_0` group-64; this entry measures the group-64 file on a current fork build. `PQ2_0` (this gen) not yet run — follow-up planned. +- Rented pod destroyed after capture; ± = llama-bench dispersion over 3 reps. + +## Notes + +The cheapest mainstream 12 GB card of its era runs the 27B ternary model with ~4 GB of KV headroom. + +## Hardware + +``` +NVIDIA GeForce RTX 2060, 12288 MiB; driver 580.178.04 +``` diff --git a/community-benchmarks/ternary-bonsai/cuda-titanv-12gb-linux.md b/community-benchmarks/ternary-bonsai/cuda-titanv-12gb-linux.md new file mode 100644 index 00000000..2916485a --- /dev/null +++ b/community-benchmarks/ternary-bonsai/cuda-titanv-12gb-linux.md @@ -0,0 +1,27 @@ +# NVIDIA TITAN V 12 GB — CUDA — Ternary-Bonsai-27B + +## Summary + +TITAN V 12 GB (GV100, Volta sm_70, HBM2; driver 580.173.02; ~$0.114/hr) measured with the PrismML llama.cpp fork `prism-b10777-e3de715` +(linux-cuda-12.4 assets) on a vast.ai pod, `llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 3`. + +| file | pp512 (t/s) | tg128 (t/s) | +|------|------------:|------------:| +| `Q2_g64` (7.05 GiB) | 576.58 ± 11.77 | 49.20 ± 0.09 | + +## Configuration + +- fork `prism-b10777-e3de715` CUDA-12.4 prebuilt +- Model: `prism-ml/Ternary-Bonsai-27B-gguf` — `Ternary-Bonsai-27B-Q2_g64.gguf` sha256 `59a45d1ecef702b14531b06d22949f33b25c1897da31a8c0b298e01e4d9138eb` (digest identical across the whole fleet). +- Per the formats guide, current builds expect `PQ2_0` (group-128) **and** `Q2_0` group-64; this entry measures the group-64 file on a current fork build. `PQ2_0` (this gen) not yet run — follow-up planned. +- Rented pod destroyed after capture; ± = llama-bench dispersion over 3 reps. + +## Notes + +A 2017 Volta flagship on a $0.11/hr pod decodes a 27B ternary model at ~49 t/s — the density story lives on legacy silicon, not only on Blackwell. + +## Hardware + +``` +NVIDIA TITAN V, 12288 MiB; driver 580.173.02 +``` diff --git a/community-benchmarks/ternary-bonsai/metal-m5-air-16gb-macos.md b/community-benchmarks/ternary-bonsai/metal-m5-air-16gb-macos.md new file mode 100644 index 00000000..c3e5d70a --- /dev/null +++ b/community-benchmarks/ternary-bonsai/metal-m5-air-16gb-macos.md @@ -0,0 +1,29 @@ +# Apple M5 (MacBook Air, 16 GB) — Metal — Ternary-Bonsai-27B + +## Summary + +Base-model M5 Air (16 GB unified, recommendedMaxWorkingSetSize ~12.7 GB). Three runtimes on the same laptop: + +| file | build | pp512 (t/s) | tg128 (t/s) | +|------|-------|------------:|------------:| +| `Q2_g64` (7.05 GiB) | upstream llama.cpp **b11630** | 80.22 ± 0.38 | 6.68 ± 0.18 | +| legacy `Q2_0` group-128 (6.66 GiB) | fork **prism-b9596** | 34.05 ± 1.70 | 6.06 ± 0.52 | +| 1-bit `Q1_0` (3.53 GiB) | upstream b11630 → see [../bonsai/metal-m5-air-16gb-macos.md](../bonsai/metal-m5-air-16gb-macos.md) | 84.18 | 11.54 | + +## Configuration + +- Upstream command: `LD_LIBRARY_PATH=./bin/b11630 ./bin/b11630/llama-bench -m Ternary-Bonsai-27B-Q2_g64.gguf -ngl 99 -fa 1 -p 512 -n 128 -r 3` (build 6868a52cb). +- Fork command: `DYLD_LIBRARY_PATH=~/Bonsai-demo/bin/mac ~/Bonsai-demo/bin/mac/llama-bench -m Ternary-Bonsai-27B-Q2_0.gguf …` (release `prism-b9596`, build 9fcaed763). +- **Legacy-format evidence (matches the formats guide):** llama-bench prints the same type string `Q2_0` for the 7.05 GiB group-64 file and the 6.66 GiB group-128 file — same type id, different block size. Upstream ggml **0.15.3 refuses both** (`llama_bench: error: failed to load model`); upstream **b11630 loads the group-64 file and refuses the legacy group-128** file; the old fork b9596 loads the legacy file fine. +- Files sha256: `Q2_g64 59a45d1e…9138eb`; legacy `Q2_0 868c11714cf8fe47f5ec9eeb2be0ab1a337112886f92ee0ede6b855c4fa31757`. + +## Notes + +- Passively cooled Air; 3-rep medians. The cross-build pp512 gap (34.05 b9596 vs 80.22 b11630) is NOT a controlled format comparison — build and file differ together; treat as two independent rows. +- 27B ternary is usable-but-warm on a $999 laptop; the same box's Bonsai-2 and 1-bit entries show better decode. + +## Hardware + +``` +Apple M5 (base), 16 GB unified, Metal4 (MTLGPUFamilyApple10), bfloat + tensor API +``` From 8fae5f0c37418a9b1ee753b13bf83997394586d2 Mon Sep 17 00:00:00 2001 From: Jay Glidden Date: Thu, 15 Oct 2026 05:07:48 -0700 Subject: [PATCH 2/3] community-benchmarks: correct 1-bit model pins, Titan V ternary baseline, Air dispersion labels - 1-bit CUDA reports (bonsai/cuda-titanv, bonsai/cuda-rtx2060): pin the model actually run - prism-ml/Bonsai-27B-gguf Bonsai-27B-Q1_0.gguf sha256 17ef842e... replacing the copied ternary Q2_g64 line and misplaced group-64 note - bonsai/cuda-titanv: older-ternary decode baseline corrected to 49.20 t/s (51.25 was the Bonsai 2 PTQ1_0 result) - M5 Air reports: label llama-bench numbers as 3-rep mean +/- std. dev. --- community-benchmarks/bonsai/cuda-rtx2060-12gb-linux.md | 3 +-- community-benchmarks/bonsai/cuda-titanv-12gb-linux.md | 5 ++--- community-benchmarks/bonsai2/metal-m5-air-16gb-macos.md | 2 +- .../ternary-bonsai/metal-m5-air-16gb-macos.md | 4 ++-- 4 files changed, 6 insertions(+), 8 deletions(-) diff --git a/community-benchmarks/bonsai/cuda-rtx2060-12gb-linux.md b/community-benchmarks/bonsai/cuda-rtx2060-12gb-linux.md index b89f761a..c234e678 100644 --- a/community-benchmarks/bonsai/cuda-rtx2060-12gb-linux.md +++ b/community-benchmarks/bonsai/cuda-rtx2060-12gb-linux.md @@ -12,8 +12,7 @@ RTX 2060 12 GB (TU106, Turing sm_75; driver 580.178.04; ~$0.100/hr) measured wit ## Configuration - fork `prism-b10777-e3de715` CUDA-12.4 -- Model: `prism-ml/Ternary-Bonsai-27B-gguf` — `Ternary-Bonsai-27B-Q2_g64.gguf` sha256 `59a45d1ecef702b14531b06d22949f33b25c1897da31a8c0b298e01e4d9138eb` (digest identical across the whole fleet). -- Per the formats guide, current builds expect `PQ2_0` (group-128) **and** `Q2_0` group-64; this entry measures the group-64 file on a current fork build. `PQ2_0` (this gen) not yet run — follow-up planned. +- Model: `prism-ml/Bonsai-27B-gguf` — `Bonsai-27B-Q1_0.gguf` sha256 `17ef842e47450caeb8eaa3ebfbbab5d2f2278b62b79be107985fb69a2f819aa0` (digest identical across the whole fleet). - Rented pod destroyed after capture; ± = llama-bench dispersion over 3 reps. ## Notes diff --git a/community-benchmarks/bonsai/cuda-titanv-12gb-linux.md b/community-benchmarks/bonsai/cuda-titanv-12gb-linux.md index 830060d4..9d4beb5e 100644 --- a/community-benchmarks/bonsai/cuda-titanv-12gb-linux.md +++ b/community-benchmarks/bonsai/cuda-titanv-12gb-linux.md @@ -12,13 +12,12 @@ TITAN V 12 GB (GV100, Volta sm_70, HBM2; driver 580.173.02; ~$0.114/hr) measured ## Configuration - fork `prism-b10777-e3de715` CUDA-12.4; upstream b11630 runs `Q1_0` too (fully merged) -- Model: `prism-ml/Ternary-Bonsai-27B-gguf` — `Ternary-Bonsai-27B-Q2_g64.gguf` sha256 `59a45d1ecef702b14531b06d22949f33b25c1897da31a8c0b298e01e4d9138eb` (digest identical across the whole fleet). -- Per the formats guide, current builds expect `PQ2_0` (group-128) **and** `Q2_0` group-64; this entry measures the group-64 file on a current fork build. `PQ2_0` (this gen) not yet run — follow-up planned. +- Model: `prism-ml/Bonsai-27B-gguf` — `Bonsai-27B-Q1_0.gguf` sha256 `17ef842e47450caeb8eaa3ebfbbab5d2f2278b62b79be107985fb69a2f819aa0` (digest identical across the whole fleet). - Rented pod destroyed after capture; ± = llama-bench dispersion over 3 reps. ## Notes -3.53 GiB leaves ~8 GB for context on a 12 GB card; decode beats the same machine's ternary row (51.25 t/s, [../ternary-bonsai/cuda-titanv-12gb-linux.md]). +3.53 GiB leaves ~8 GB for context on a 12 GB card; decode beats the same machine's ternary row (49.20 t/s, [../ternary-bonsai/cuda-titanv-12gb-linux.md]). ## Hardware diff --git a/community-benchmarks/bonsai2/metal-m5-air-16gb-macos.md b/community-benchmarks/bonsai2/metal-m5-air-16gb-macos.md index d8464cf1..cd99e95a 100644 --- a/community-benchmarks/bonsai2/metal-m5-air-16gb-macos.md +++ b/community-benchmarks/bonsai2/metal-m5-air-16gb-macos.md @@ -18,7 +18,7 @@ between the gen-1 paths while keeping the smallest 27B ternary footprint at 5.53 - `prism-ml/Ternary-Bonsai-2-27B-gguf`; sha256 `53107f53…` (PTQ1_0), `3907dc16…` (PQ2_0) — same digests as the CUDA machines. - Upstream b11630 refuses both files (`llama_model_load: error loading model`) — fork-only bands, clean refusal. -- Passively cooled Air; 3-rep medians, not sustained thermals. +- Passively cooled Air; reported as llama-bench 3-rep mean ± std. dev., not sustained thermals. ## Hardware diff --git a/community-benchmarks/ternary-bonsai/metal-m5-air-16gb-macos.md b/community-benchmarks/ternary-bonsai/metal-m5-air-16gb-macos.md index c3e5d70a..4dcef4fb 100644 --- a/community-benchmarks/ternary-bonsai/metal-m5-air-16gb-macos.md +++ b/community-benchmarks/ternary-bonsai/metal-m5-air-16gb-macos.md @@ -8,7 +8,7 @@ Base-model M5 Air (16 GB unified, recommendedMaxWorkingSetSize ~12.7 GB). Three |------|-------|------------:|------------:| | `Q2_g64` (7.05 GiB) | upstream llama.cpp **b11630** | 80.22 ± 0.38 | 6.68 ± 0.18 | | legacy `Q2_0` group-128 (6.66 GiB) | fork **prism-b9596** | 34.05 ± 1.70 | 6.06 ± 0.52 | -| 1-bit `Q1_0` (3.53 GiB) | upstream b11630 → see [../bonsai/metal-m5-air-16gb-macos.md](../bonsai/metal-m5-air-16gb-macos.md) | 84.18 | 11.54 | +| 1-bit `Q1_0` (3.53 GiB) | upstream b11630 → see [../bonsai/metal-m5-air-16gb-macos.md](../bonsai/metal-m5-air-16gb-macos.md) | 84.18 ± 0.36 | 11.54 ± 0.08 | ## Configuration @@ -19,7 +19,7 @@ Base-model M5 Air (16 GB unified, recommendedMaxWorkingSetSize ~12.7 GB). Three ## Notes -- Passively cooled Air; 3-rep medians. The cross-build pp512 gap (34.05 b9596 vs 80.22 b11630) is NOT a controlled format comparison — build and file differ together; treat as two independent rows. +- Passively cooled Air; reported as llama-bench 3-rep mean ± std. dev. The cross-build pp512 gap (34.05 b9596 vs 80.22 b11630) is NOT a controlled format comparison — build and file differ together; treat as two independent rows. - 27B ternary is usable-but-warm on a $999 laptop; the same box's Bonsai-2 and 1-bit entries show better decode. ## Hardware From a92f9882016fe342f9c400231b3a256ead19a48c Mon Sep 17 00:00:00 2001 From: Jay Glidden Date: Thu, 15 Oct 2026 07:45:07 -0700 Subject: [PATCH 3/3] community-benchmarks: correct A4000 comparison with RTX 3090 Ti --- community-benchmarks/bonsai2/cuda-rtxa4000-16gb-linux.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/community-benchmarks/bonsai2/cuda-rtxa4000-16gb-linux.md b/community-benchmarks/bonsai2/cuda-rtxa4000-16gb-linux.md index a1a9a1b1..62608049 100644 --- a/community-benchmarks/bonsai2/cuda-rtxa4000-16gb-linux.md +++ b/community-benchmarks/bonsai2/cuda-rtxa4000-16gb-linux.md @@ -10,7 +10,7 @@ RTX A4000 16 GB (GA104, Ampere pro), driver 580.173.02, ~$0.116/hr on vast.ai. l | `PTQ1_0` (5.53 GiB) | 769.94 ± 22.99 | 49.49 ± 0.11 | | `PQ2_0` (6.70 GiB) | 758.24 ± 19.84 | 42.42 ± 0.10 | -`PTQ1_0` edges prefill and takes decode — consistent with the consumer-Ampere (3090 Ti) and desktop-Volta pattern above. 16 GB leaves room for longer context with the 5.53 GiB file. +On this RTX A4000, `PTQ1_0` has higher measured means for both tests. This matches the TITAN V report, but contrasts with the existing RTX 3090 Ti report, where `PQ2_0` leads both columns. Hardware, binary builds, and runtime settings differ between submissions, so these comparisons do not isolate an architecture-level cause; select the band from measurements on the intended setup. ## Configuration