Skip to content

community-benchmarks: TITAN V / RTX 2060 / Tesla P40 / RTX A4000 / base M5 Air rows (all three families) + two version-floor findings #263

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
khosravipasha merged 5 commits into from
Oct 25, 2026
Merged

community-benchmarks: TITAN V / RTX 2060 / Tesla P40 / RTX A4000 / base M5 Air rows (all three families) + two version-floor findings #263

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
7623c52
community-benchmarks: TITAN V, RTX 2060, Tesla P40, RTX A4000, base A…
Oct 15, 2026
8fae5f0
community-benchmarks: correct 1-bit model pins, Titan V ternary basel…
EngineeringWithJay Oct 15, 2026
a92f988
community-benchmarks: correct A4000 comparison with RTX 3090 Ti
EngineeringWithJay Oct 15, 2026
717f669
Merge main and preserve all community benchmark rows
khosravipasha Oct 20, 2026
6c851f3
Merge main and retain all benchmark index entries
khosravipasha Oct 25, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 16 additions & 1 deletion community-benchmarks/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,15 +21,19 @@ for commands, raw output, memory use, and configuration.
| `PQ2_0` | NVIDIA RTX 4090 24 GB | llama.cpp CUDA (Windows) | 3,285 | 84.9 | [link](bonsai2/cuda-rtx4090-windows.md) |
| `PQ2_0` | NVIDIA RTX 3090 Ti 24 GB | llama.cpp CUDA (Linux) | 1,552 | 81.6 | [link](bonsai2/cuda-rtx3090ti-linux.md) |
| `PTQ1_0` | NVIDIA RTX 3090 Ti 24 GB | llama.cpp CUDA (Linux) | 805 | 67.7 | [link](bonsai2/cuda-rtx3090ti-linux.md) |
| `PTQ1_0` | NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 562.2 | 51.25 | [link](bonsai2/cuda-titanv-12gb-linux.md) |
| `PTQ1_0` | NVIDIA RTX A4000 16 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 769.9 | 49.49 | [link](bonsai2/cuda-rtxa4000-16gb-linux.md) |
| `PQ2_0` (MTP bundle, plain inference) | AMD Radeon RX 9070 XT 16 GB | llama.cpp ROCm/HIP (Proxmox) | 1,258 | 48.64 | [link](bonsai2/rocm-hip-rx9070xt-proxmox.md) |
| `PQ2_0` | AMD Radeon RX 9070 XT 16 GB | llama.cpp ROCm/HIP (Proxmox) | 1,259 | 48.53 | [link](bonsai2/rocm-hip-rx9070xt-proxmox.md) |
| `PTQ1_0` | Intel Arc B580 12 GB | llama.cpp SYCL, oneAPI 2026.1 (Linux) | 1,593 | 47.9 | [link](bonsai2/sycl-arc-b580-linux.md) |
| `PQ2_0` | AMD Radeon RX 7800 XT 16 GB | llama.cpp ROCm/HIP | 316.3 | 46.8 | [link](bonsai2/rocm-hip-rx7800xt-fedora44.md) |
| `PQ2_0` | NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 528.2 | 46.66 | [link](bonsai2/cuda-titanv-12gb-linux.md) |
| `PQ2_0` | NVIDIA Tesla V100-SXM2 16 GB | llama.cpp CUDA (Windows) | 798 | 46.0 | [link](bonsai2/cuda-tesla-v100-windows.md) |
| `PTQ1_0` | Intel Arc B580 12 GB | llama.cpp SYCL (Windows, community fork) | 1,560 | 45.89 | [link](bonsai2/sycl-arc-b580-12gb-windows.md) |
| `PQ2_0` | Intel Arc B580 12 GB | llama.cpp SYCL (Windows, community fork) | 1,533 | 45.87 | [link](bonsai2/sycl-arc-b580-12gb-windows.md) |
| `PQ2_0` | AMD Radeon RX 6800 XT 16 GB | llama.cpp ROCm/HIP (Windows) | 266.4 | 44.9 | [link](bonsai2/rocm-hip-rx6800xt-windows.md) |
| `PQ2_0` | NVIDIA RTX 3070 8 GB | llama.cpp CUDA (Windows) | 838.9 | 43.7 | [link](bonsai2/cuda-rtx3070-8gb-windows.md) |
| `PQ2_0` | NVIDIA RTX A4000 16 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 758.2 | 42.42 | [link](bonsai2/cuda-rtxa4000-16gb-linux.md) |
| `PTQ1_0` | Intel Arc B580 12 GB | llama.cpp Vulkan (Windows, PrismML fork) | 380 | 34.84 | [link](bonsai2/vulkan-arc-b580-12gb-windows.md) |
| `PTQ1_0` | NVIDIA Tesla V100-SXM2 16 GB | llama.cpp CUDA (Windows) | 852 | 34.3 | [link](bonsai2/cuda-tesla-v100-windows.md) |
| `PTQ1_0` | NVIDIA RTX 4070 Laptop 8 GB | llama.cpp CUDA (Windows) | 435.60 | 32.44 | [link](bonsai2/cuda-rtx4070-laptop-windows.md) |
Expand All @@ -38,8 +42,12 @@ for commands, raw output, memory use, and configuration.
| `2-bit` | Apple M3 Max 48 GB | MLX (mlx 0.32.2, mlx-vlm 0.6.17) | 165.9 | 27.1 | [link](bonsai2/mlx-m3-max-48gb-macos.md) |
| `PQ2_0` | Apple M3 Max 36 GB | llama.cpp Metal | 162.2 | 24.3 | [link](bonsai2/metal-m3-max-36gb-macos.md) |
| `PQ2_0` | AMD BC-250 16 GB UMA | llama.cpp Vulkan (Linux, local patches) | 142.58 | 24.11 | [link](bonsai2/vulkan-bc250-linux.md) |
| `PQ2_0` | Tesla P40 24 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 254.4 | 23.16 | [link](bonsai2/cuda-teslap40-24gb-linux.md) |
| `PTQ1_0` | Apple M3 Max 36 GB | llama.cpp Metal | 136.8 | 21.9 | [link](bonsai2/metal-m3-max-36gb-macos.md) |
| `PTQ1_0` | Tesla P40 24 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 139.9 | 19.86 | [link](bonsai2/cuda-teslap40-24gb-linux.md) |
| `Q2_0` (development) | Intel Arc B580 12 GB | llama.cpp SYCL (Windows, community fork) | 770 | 10.26 | [link](bonsai2/sycl-arc-b580-12gb-windows.md) |
| `PQ2_0` | Apple M5 (Air) 16 GB | llama.cpp Metal (fork b10777) | 88.8 | 7.82 | [link](bonsai2/metal-m5-air-16gb-macos.md) |
| `PTQ1_0` | Apple M5 (Air) 16 GB | llama.cpp Metal (fork b10777) | 79.7 | 7.03 | [link](bonsai2/metal-m5-air-16gb-macos.md) |

## Earlier 27B families

Expand All @@ -61,19 +69,26 @@ not directly comparable; each report should identify its harness and workload.
| Ternary | NVIDIA L40S 48 GB | llama.cpp CUDA | 3,036 | 74.3 | ~150 (2.06x, 2.4x math) | [link](ternary-bonsai/cuda-l40s-linux.md) |
| Ternary | NVIDIA RTX 4070 Ti SUPER 16 GB | llama.cpp CUDA (Windows) | 1,717 | 69.6 | | [link](ternary-bonsai/cuda-rtx4070tisuper-windows.md) |
| Bonsai (1-bit) | Apple M5 Max 48 GB | llama.cpp Metal | 796 | 63.9 | slower on this HW | [link](bonsai/metal-m5-max-48gb-macos.md) |
| Bonsai (1-bit) | NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 552.4 | 56.47 | | [link](bonsai/cuda-titanv-12gb-linux.md) |
| Ternary | NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 576.6 | 49.20 | | [link](ternary-bonsai/cuda-titanv-12gb-linux.md) |
| Ternary | NVIDIA RTX A5000 24 GB | llama.cpp CUDA | 1,036 | 48.2 | | [link](ternary-bonsai/cuda-rtxa5000-ubuntu.md) |
| Ternary | Apple M5 Max 48 GB | llama.cpp Metal | 816 | 45.8 | ~1.2x code/math only | [link](ternary-bonsai/metal-m5-max-48gb-macos.md) |
| Bonsai (1-bit) | NVIDIA DGX Spark (GB10) | llama.cpp CUDA | 1,024 | 45.4 | ~96.1 (2.21x, code) | [link](bonsai/cuda-gb10-27b-linux.md) |
| Ternary | NVIDIA RTX 5060 Ti 16 GB | llama.cpp CUDA | 1,029 | 44.4 | ~79 (1.78x) | [link](ternary-bonsai/cuda-rtx5060ti-linux.md) |
| Bonsai (1-bit) | NVIDIA GeForce RTX 2060 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 431.2 | 29.96 | | [link](bonsai/cuda-rtx2060-12gb-linux.md) |
| Ternary | Apple M5 Pro 64 GB | MLX 2-bit | 466 | 29.5 | 34-49 (community dspark-mlx) | [link](ternary-bonsai/mlx-m5-pro-macos.md) |
| Ternary | NVIDIA DGX Spark (GB10) | llama.cpp CUDA | 1,005 | 29.2 | ~70.0 (2.45x, code) | [link](ternary-bonsai/cuda-gb10-27b-linux.md) |
| Bonsai (1-bit) | NVIDIA GeForce GTX 1080 Ti 11 GB | llama.cpp CUDA | 285 | 28.3 | | [link](bonsai/cuda-gtx1080ti-linux.md) |
| Ternary | Apple M5 Pro 64 GB | llama.cpp Metal | 130 | 26.5 | | [link](ternary-bonsai/mlx-m5-pro-macos.md) |
| Ternary | NVIDIA GeForce RTX 2060 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 415.8 | 25.71 | | [link](ternary-bonsai/cuda-rtx2060-12gb-linux.md) |
| Ternary | Apple M4 Pro 64 GB | MLX 2-bit | 120 | 24.8 | | [link](ternary-bonsai/mlx-m4-pro-64gb-macos.md) |
| Ternary | Apple M4 Pro 64 GB | llama.cpp Metal | 116 | 19.0 | slower on this HW | [link](ternary-bonsai/metal-m4-pro-64gb-macos.md) |
| Ternary | NVIDIA GeForce GTX 1080 Ti 11 GB | llama.cpp CUDA | 278 | 20.5 | | [link](ternary-bonsai/cuda-gtx1080ti-linux.md) |
| Ternary | Apple M4 Pro 64 GB | llama.cpp Metal | 116 | 19.0 | slower on this HW | [link](ternary-bonsai/metal-m4-pro-64gb-macos.md) |
| Ternary | Apple M4 24 GB | MLX 2-bit | 65.2 | 12.7 | | [link](ternary-bonsai/mlx-m4-24gb-macos.md) |
| Ternary | Apple M3 Pro 18 GB | llama.cpp Metal | 78.6 | 12.6 | | [link](ternary-bonsai/metal-m3-pro-macos.md) |
| Bonsai (1-bit) | Apple M5 (Air) 16 GB | llama.cpp Metal (upstream b11630) | 84.2 | 11.54 | | [link](bonsai/metal-m5-air-16gb-macos.md) |
| Ternary | Apple M5 (Air) 16 GB | llama.cpp Metal (upstream b11630, `Q2_g64`) | 80.2 | 6.68 | | [link](ternary-bonsai/metal-m5-air-16gb-macos.md) |
| Ternary | Apple M5 (Air) 16 GB | llama.cpp Metal (fork b9596, legacy `Q2_0`) | 34.1 | 6.06 | | [link](ternary-bonsai/metal-m5-air-16gb-macos.md) |

## 8B and smaller

Expand Down
3 changes: 3 additions & 0 deletions community-benchmarks/bonsai/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,10 +11,13 @@ Benchmark results submitted by the community running [Bonsai](https://huggingfac
| NVIDIA L40S 48 GB | llama.cpp CUDA | 2,937 | 107.5 | [link](cuda-l40s-27b-linux.md) |
| NVIDIA RTX 3090 24 GB | llama.cpp CUDA | 1,480 | 89.2 | [link](cuda-rtx3090-linux.md) |
| Apple M5 Max 48 GB | llama.cpp Metal | 796 | 63.9 | [link](metal-m5-max-48gb-macos.md) |
| NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 552.4 | 56.47 | [link](cuda-titanv-12gb-linux.md) |
| NVIDIA DGX Spark (GB10) | llama.cpp CUDA | 1,024 | 45.4 | [link](cuda-gb10-27b-linux.md) |
| NVIDIA RTX 2060 SUPER 8 GB | llama.cpp CUDA (WSL) | 494 | 32.7 | [link](cuda-rtx2060-linux.md) |
| NVIDIA GeForce RTX 2060 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 431.2 | 29.96 | [link](cuda-rtx2060-12gb-linux.md) |
| NVIDIA GeForce GTX 1080 Ti 11 GB | llama.cpp CUDA | 285 | 28.3 | [link](cuda-gtx1080ti-linux.md) |
| Apple M3 MacBook Air 16 GB | MLX 1-bit | 51.0 | 18.3 | [link](mlx-m3-air-macos.md) |
| Apple M5 (Air) 16 GB | llama.cpp Metal (upstream b11630) | 84.2 | 11.54 | [link](metal-m5-air-16gb-macos.md) |

### 8B and smaller

Expand Down
26 changes: 26 additions & 0 deletions community-benchmarks/bonsai/cuda-rtx2060-12gb-linux.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# NVIDIA GeForce RTX 2060 12 GB — CUDA — Bonsai-27B (1-bit)

## Summary

RTX 2060 12 GB (TU106, Turing sm_75; driver 580.178.04; ~$0.100/hr) measured with the PrismML llama.cpp fork `prism-b10777-e3de715`
(linux-cuda-12.4 assets) on a vast.ai pod, `llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 3`.

| file | pp512 (t/s) | tg128 (t/s) |
|------|------------:|------------:|
| `Q1_0` (3.53 GiB) | 431.23 ± 4.28 | 29.96 ± 0.05 |

## Configuration

- fork `prism-b10777-e3de715` CUDA-12.4
- Model: `prism-ml/Bonsai-27B-gguf` — `Bonsai-27B-Q1_0.gguf` sha256 `17ef842e47450caeb8eaa3ebfbbab5d2f2278b62b79be107985fb69a2f819aa0` (digest identical across the whole fleet).
- Rented pod destroyed after capture; ± = llama-bench dispersion over 3 reps.

## Notes

Same card as the ternary entry: 1-bit gives +3.7% prefill, +17% decode, half the bytes.

## Hardware

```
NVIDIA GeForce RTX 2060, 12288 MiB; driver 580.178.04
```
26 changes: 26 additions & 0 deletions community-benchmarks/bonsai/cuda-titanv-12gb-linux.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# NVIDIA TITAN V 12 GB — CUDA — Bonsai-27B (1-bit)

## Summary

TITAN V 12 GB (GV100, Volta sm_70, HBM2; driver 580.173.02; ~$0.114/hr) measured with the PrismML llama.cpp fork `prism-b10777-e3de715`
(linux-cuda-12.4 assets) on a vast.ai pod, `llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 3`.

| file | pp512 (t/s) | tg128 (t/s) |
|------|------------:|------------:|
| `Q1_0` (3.53 GiB) | 552.38 ± 0.51 | 56.47 ± 0.10 |

## Configuration

- fork `prism-b10777-e3de715` CUDA-12.4; upstream b11630 runs `Q1_0` too (fully merged)
- Model: `prism-ml/Bonsai-27B-gguf` — `Bonsai-27B-Q1_0.gguf` sha256 `17ef842e47450caeb8eaa3ebfbbab5d2f2278b62b79be107985fb69a2f819aa0` (digest identical across the whole fleet).
- Rented pod destroyed after capture; ± = llama-bench dispersion over 3 reps.

## Notes

3.53 GiB leaves ~8 GB for context on a 12 GB card; decode beats the same machine's ternary row (49.20 t/s, [../ternary-bonsai/cuda-titanv-12gb-linux.md]).

## Hardware

```
NVIDIA TITAN V, 12288 MiB; driver 580.173.02
```
28 changes: 28 additions & 0 deletions community-benchmarks/bonsai/metal-m5-air-16gb-macos.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
# Apple M5 (MacBook Air, 16 GB) — Metal — Bonsai-27B (1-bit)

## Summary

Base M5 Air, 16 GB unified. Upstream llama.cpp **b11630** (no fork needed — `Q1_0` is fully merged
upstream), `-ngl 99 -fa 1 -p 512 -n 128 -r 3`:

| file | pp512 (t/s) | tg128 (t/s) |
|------|------------:|------------:|
| `Q1_0` (3.53 GiB) | 84.18 ± 0.36 | 11.54 ± 0.08 |

Decode 1.7x the same laptop's ternary rows; the 27B sits in 3.53 GiB — twice over, with KV headroom.

## Configuration

- Command: `LD_LIBRARY_PATH=./bin/b11630 ./bin/b11630/llama-bench -m Bonsai-27B-Q1_0.gguf -ngl 99 -fa 1 -p 512 -n 128 -r 3` (build 6868a52cb).
- File: `prism-ml/Bonsai-27B-gguf` — `Bonsai-27B-Q1_0.gguf` sha256 `17ef842e47450caeb8eaa3ebfbbab5d2f2278b62b79be107985fb69a2f819aa0`.
- 27B only on this box (8B/4B/1.7B skipped: passive cooling makes long sweeps thermally noisy).

## Notes

- The whole point of 1-bit: a 27B-class model that fits *twice* in 16 GB with room for context, on upstream llama.cpp, in a $999 laptop.

## Hardware

```
Apple M5 (base), 16 GB unified, Metal4/tensor-API
```
25 changes: 16 additions & 9 deletions community-benchmarks/bonsai2/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,31 +19,38 @@ Benchmark results submitted by the community running
| `PQ2_0` | NVIDIA RTX 4090 24 GB | llama.cpp CUDA (Windows) | 3,285 | 84.9 | [link](cuda-rtx4090-windows.md) |
| `PQ2_0` | NVIDIA RTX 3090 Ti 24 GB | llama.cpp CUDA (Linux) | 1,552 | 81.6 | [link](cuda-rtx3090ti-linux.md) |
| `PTQ1_0` | NVIDIA RTX 3090 Ti 24 GB | llama.cpp CUDA (Linux) | 805 | 67.7 | [link](cuda-rtx3090ti-linux.md) |
| `PTQ1_0` | NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 562.2 | 51.25 | [link](cuda-titanv-12gb-linux.md) |
| `PTQ1_0` | NVIDIA RTX A4000 16 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 769.9 | 49.49 | [link](cuda-rtxa4000-16gb-linux.md) |
| `PQ2_0` (community MTP file, plain inference) | NVIDIA RTX 5070 Ti Laptop 12 GB | llama.cpp CUDA (Windows) | 1,135 | 49.0 | [link](cuda-rtx5070ti-laptop-windows.md) |
| `PTQ1_0` | NVIDIA RTX 5070 Ti Laptop 12 GB | llama.cpp CUDA (Windows) | 527 | 49.0 | [link](cuda-rtx5070ti-laptop-windows.md) |
| `PQ2_0` (MTP bundle, plain inference) | AMD Radeon RX 9070 XT 16 GB | llama.cpp ROCm/HIP (Proxmox) | 1,258 | 48.64 | [link](rocm-hip-rx9070xt-proxmox.md) |
| `PQ2_0` | AMD Radeon RX 9070 XT 16 GB | llama.cpp ROCm/HIP (Proxmox) | 1,259 | 48.53 | [link](rocm-hip-rx9070xt-proxmox.md) |
| `PTQ1_0` | NVIDIA RTX 5070 Ti Laptop 12 GB | llama.cpp CUDA (Windows) | 527 | 49.0 | [link](cuda-rtx5070ti-laptop-windows.md) |
| `PQ2_0` | AMD Radeon RX 7800 XT 16 GB | llama.cpp ROCm/HIP | 316.3 | 46.8 | [link](rocm-hip-rx7800xt-fedora44.md) |
| `PQ2_0` | NVIDIA TITAN V 12 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 528.2 | 46.66 | [link](cuda-titanv-12gb-linux.md) |
| `PQ2_0` | NVIDIA Tesla V100-SXM2 16 GB | llama.cpp CUDA (Windows) | 798 | 46.0 | [link](cuda-tesla-v100-windows.md) |
| `PTQ1_0` | Intel Arc B580 12 GB | llama.cpp SYCL (Windows, community fork Torchit1 arc-b580-v0.10.1) | 1,560 | 45.89 | [link](sycl-arc-b580-12gb-windows.md) |
| `PQ2_0` | Intel Arc B580 12 GB | llama.cpp SYCL (Windows, community fork Torchit1 arc-b580-v0.10.1) | 1,533 | 45.87 | [link](sycl-arc-b580-12gb-windows.md) |
| `PTQ1_0` (community mtp-lean graft, plain inference) | Intel Arc B580 12 GB | llama.cpp SYCL (Windows, community fork Torchit1 arc-b580-v0.10.1) | 1,562 | 45.45 | [link](sycl-arc-b580-12gb-windows.md) |
| `PQ2_0` | AMD Radeon RX 6800 XT 16 GB | llama.cpp ROCm/HIP (Windows) | 266.4 | 44.9 | [link](rocm-hip-rx6800xt-windows.md) |
| `PQ2_0` | NVIDIA RTX 3070 8 GB | llama.cpp CUDA (Windows) | 838.9 | 43.7 | [link](cuda-rtx3070-8gb-windows.md) |
| `PQ2_0` | NVIDIA RTX A4000 16 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 758.2 | 42.42 | [link](cuda-rtxa4000-16gb-linux.md) |
| `PTQ1_0` | Intel Arc B580 12 GB | llama.cpp Vulkan (Windows, PrismML fork prism-b10777) | 380 | 34.84 | [link](vulkan-arc-b580-12gb-windows.md) |
| `PTQ1_0` | NVIDIA Tesla V100-SXM2 16 GB | llama.cpp CUDA (Windows) | 852 | 34.3 | [link](cuda-tesla-v100-windows.md) |
| `PTQ1_0` | NVIDIA RTX 4070 Laptop 8 GB | llama.cpp CUDA (Windows) | 435.60 | 32.44 | [link](cuda-rtx4070-laptop-windows.md) |
| `Q2_0` (development) | Intel Arc B580 12 GB | llama.cpp Vulkan (Windows, PrismML fork prism-b10777) | 568 | 31.59 | [link](vulkan-arc-b580-12gb-windows.md) |
| `PQ2_0` | Intel Arc B580 12 GB | llama.cpp Vulkan (Windows, PrismML fork prism-b10777) | 559 | 28.59 | [link](vulkan-arc-b580-12gb-windows.md) |
| `2-bit` | Apple M3 Max 48 GB | MLX (mlx 0.32.2, mlx-vlm 0.6.17) | 165.9 | 27.1 | [link](mlx-m3-max-48gb-macos.md) |
| `PQ2_0` | Apple M3 Max 36 GB | llama.cpp Metal | 162.2 | 24.3 | [link](metal-m3-max-36gb-macos.md) |
| `PQ2_0` | AMD BC-250 16 GB UMA | llama.cpp Vulkan (Linux, local patches) | 142.58 | 24.11 | [link](vulkan-bc250-linux.md) |
| `PQ2_0` | Tesla P40 24 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 254.4 | 23.16 | [link](cuda-teslap40-24gb-linux.md) |
| `PTQ1_0` | Apple M3 Max 36 GB | llama.cpp Metal | 136.8 | 21.9 | [link](metal-m3-max-36gb-macos.md) |
| `PQ2_0` | Apple M4 Pro 64 GB | llama.cpp Metal | 126.9 | 20.5 | [link](metal-m4-pro-64gb-macos.md) |
| `PTQ1_0` | Apple M4 Pro 64 GB | llama.cpp Metal | 98.6 | 17.3 | [link](metal-m4-pro-64gb-macos.md) |
| `PTQ1_0` | Tesla P40 24 GB | llama.cpp CUDA 12.4 (Linux, fork b10777) | 139.9 | 19.86 | [link](cuda-teslap40-24gb-linux.md) |
| `2-bit` | Apple M4 Pro 64 GB | MLX (mlx_lm 0.31.3) | 97.9 | 18.9 | [link](metal-m4-pro-64gb-macos.md) |
| `PQ2_0` | Intel Arc B580 12 GB | llama.cpp Vulkan (Windows, PrismML fork prism-b10777) | 559 | 28.59 | [link](vulkan-arc-b580-12gb-windows.md) |
| `PTQ1_0` | Intel Arc B580 12 GB | llama.cpp Vulkan (Windows, PrismML fork prism-b10777) | 380 | 34.84 | [link](vulkan-arc-b580-12gb-windows.md) |
| `Q2_0` (development) | Intel Arc B580 12 GB | llama.cpp Vulkan (Windows, PrismML fork prism-b10777) | 568 | 31.59 | [link](vulkan-arc-b580-12gb-windows.md) |
| `PQ2_0` | Intel Arc B580 12 GB | llama.cpp SYCL (Windows, community fork Torchit1 arc-b580-v0.10.1) | 1,533 | 45.87 | [link](sycl-arc-b580-12gb-windows.md) |
| `PTQ1_0` | Intel Arc B580 12 GB | llama.cpp SYCL (Windows, community fork Torchit1 arc-b580-v0.10.1) | 1,560 | 45.89 | [link](sycl-arc-b580-12gb-windows.md) |
| `PTQ1_0` (community mtp-lean graft, plain inference) | Intel Arc B580 12 GB | llama.cpp SYCL (Windows, community fork Torchit1 arc-b580-v0.10.1) | 1,562 | 45.45 | [link](sycl-arc-b580-12gb-windows.md) |
| `PTQ1_0` | Apple M4 Pro 64 GB | llama.cpp Metal | 98.6 | 17.3 | [link](metal-m4-pro-64gb-macos.md) |
| `Q2_0` (development) | Intel Arc B580 12 GB | llama.cpp SYCL (Windows, community fork Torchit1 arc-b580-v0.10.1) | 770 | 10.26 | [link](sycl-arc-b580-12gb-windows.md) |

| `PQ2_0` | Apple M5 (Air) 16 GB | llama.cpp Metal (fork b10777) | 88.8 | 7.82 | [link](metal-m5-air-16gb-macos.md) |
| `PTQ1_0` | Apple M5 (Air) 16 GB | llama.cpp Metal (fork b10777) | 79.7 | 7.03 | [link](metal-m5-air-16gb-macos.md) |

## How to Submit

Expand Down
25 changes: 25 additions & 0 deletions community-benchmarks/bonsai2/cuda-rtxa4000-16gb-linux.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# NVIDIA RTX A4000 16 GB — CUDA — Bonsai 2 27B (Ternary-Bonsai-2)

## Summary

RTX A4000 16 GB (GA104, Ampere pro), driver 580.173.02, ~$0.116/hr on vast.ai. llama.cpp **PrismML fork `prism-b10777-e3de715`** (linux-cuda-12.4) on vast.ai,
`llama-bench -ngl 99 -fa 1 -p 512 -n 128 -r 3`. Both current bands measured on the same machine:

| Band | pp512 (t/s) | tg128 (t/s) |
|------|------------:|------------:|
| `PTQ1_0` (5.53 GiB) | 769.94 ± 22.99 | 49.49 ± 0.11 |
| `PQ2_0` (6.70 GiB) | 758.24 ± 19.84 | 42.42 ± 0.10 |

On this RTX A4000, `PTQ1_0` has higher measured means for both tests. This matches the TITAN V report, but contrasts with the existing RTX 3090 Ti report, where `PQ2_0` leads both columns. Hardware, binary builds, and runtime settings differ between submissions, so these comparisons do not isolate an architecture-level cause; select the band from measurements on the intended setup.

## Configuration

- Files: `prism-ml/Ternary-Bonsai-2-27B-gguf` — `PTQ1_0` sha256 `53107f530aa52eb00912263ab1ee29bd199261c87cd7b4ad4ca1318c1fe33ee3`; `PQ2_0` sha256 `3907dc1658db1f78a9826bf8d5bcb8dc65db0d466388937af57f2294fae62ec1` (identical digests across all machines in this sweep).
- Upstream b11630 cannot load either band (fork-only) and its ubuntu-cuda build additionally cannot start in the `vastai/pytorch` container (`GLIBC_2.38 not found`).
- Rented cloud pod, destroyed after capture. ± = llama-bench dispersion over 3 reps.

## Hardware

```
NVIDIA RTX A4000, 16376 MiB; driver 580.173.02; container vastai/pytorch:latest
```
Loading