Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,15 +23,15 @@ The 27B generation is a step change over the earlier 8B/4B/1.7B demos:

| `BONSAI_FAMILY` / `BONSAI_MODEL` | Weights | Notes |
|---|---|---|
| `bonsai2` / `27B` (**default**) | GGUF 5.9-7.2 GB + mmproj 0.63 GB; MLX 2-bit 8.6 GB | The current generation and what plain `./setup.sh` installs. Two GGUF bands, **both requiring this demo's binaries**: `PTQ1_0` (1.75 bpw, 5.9 GB, densely packed trits, smallest) and `PQ2_0` (2.13 bpw, 7.2 GB, faster prompt processing, what the scripts download). Q2_0 is supported by current official llama.cpp with Bonsai 2 integration. MLX uses native Hadamard-aware mlx-vlm 0.7.8 from `.venv-vlm` for one-shot generation and serving, not the fork in `.venv` |
| `bonsai2` / `27B` (**default**) | GGUF 5.9-7.2 GB + mmproj 0.63 GB; MLX 2-bit 8.6 GB | The current generation and what plain `./setup.sh` installs. Two GGUF bands, **both requiring this demo's binaries**: `PTQ1_0` (1.75 bpw, 5.9 GB, densely packed trits, smallest) and `PQ2_0` (2.13 bpw, 7.2 GB, faster prompt processing, what the scripts download). Q2_0 is supported by official llama.cpp b11556 or newer. MLX uses native Hadamard-aware mlx-vlm 0.7.8 from `.venv-vlm` for one-shot generation and serving, not the fork in `.venv` |
| `ternary` / `27B` (previous generation) | GGUF ~6.7-7.1 GB + mmproj 0.9 GB; MLX 2-bit ~7.9 GB | Superseded by `bonsai2`. Two GGUF formats since the mainline rebase (`prism-b10658+`): `PQ2_0` (group 128, 6.66 GiB, smallest/fastest where supported: CUDA, Metal, CPU, ROCm) and official `Q2_0` group 64 (`Ternary-Bonsai-27B-Q2_g64.gguf`, 7.05 GiB, adds Vulkan/SYCL); smaller sizes use `*-Q2_0_g64.gguf` naming. The scripts pick per backend. Legacy `*-Q2_0.gguf` files (no `g64`) only load on old `prism-v5` releases; new binaries refuse them with an error |
| `bonsai` / `27B` | GGUF Q1_0 ~3.5 GB + mmproj 0.9 GB; MLX 1-bit ~4.8 GB | Smallest and fastest; fits on a modern iPhone without offloading |
| `8B` / `4B` / `1.7B` (both families) | smaller | Text-only, no tools wiring, legacy tested flag set |

### Current runtime support

Current official llama.cpp supports Bonsai 2 in Q2_0 with the merged model and
Hadamard/sign-flip integration. PQ2_0 and PTQ1_0 still need our fork. Older engines
Official llama.cpp supports Bonsai 2 in Q2_0 from release b11556, which merged the
model and Hadamard/sign-flip integration. PQ2_0 and PTQ1_0 still need our fork. Older engines
may recognize Q2_0 but miss the transforms; check the build when diagnosing gibberish.
See [upstream support](README.md#upstream-support) instead of relying on old PR lists.

Expand Down
4 changes: 2 additions & 2 deletions BACKEND-SUPPORT.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,8 +89,8 @@ The demo setup scripts do not automatically select or install SYCL yet.

## Model files and upstream compatibility

- **Q2_0:** works with current official llama.cpp containing the merged Bonsai 2
model/Hadamard integration. Older bundled engines need updating; recognizing
- **Q2_0:** works with official llama.cpp b11556 or newer, which contains the merged
Bonsai 2 model/Hadamard integration. Older bundled engines need updating; recognizing
the type alone did not guarantee correct output in older builds.
- **PQ2_0 and PTQ1_0:** remain fork-specific. PQ2_0 is this demo's download default.
- **MTP/DFlash:** compatibility is checkpoint/runtime-specific; some paths still
Expand Down
2 changes: 1 addition & 1 deletion MODEL-FORMATS.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ and the fork's group-128 layout lives under its own name and id, PQ2_0 (142).

| Packing | Runtime | Trade-off |
|---|---|---|
| `Q2_0` | Current official llama.cpp with Bonsai 2 support, or our fork | Upstream format |
| `Q2_0` | Official llama.cpp b11556 or newer, or our fork | Upstream format |
| `PQ2_0` | PrismML fork | Faster prompt processing than PTQ1_0 on many devices |
| `PTQ1_0` | PrismML fork | Smallest packed files |

Expand Down
17 changes: 9 additions & 8 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ model's card for its specifications:
faster prompt processing and is what this demo downloads by default.
See [MODEL-FORMATS.md](MODEL-FORMATS.md).

**Official `Q2_0` works with current upstream llama.cpp containing Bonsai 2 support.**
**Official `Q2_0` works with upstream llama.cpp [b11556](https://github.com/ggml-org/llama.cpp/releases/tag/b11556) or newer.**
`PQ2_0` and `PTQ1_0` still require the [PrismML fork](https://github.com/PrismML-Eng/llama.cpp).
This demo downloads `PQ2_0` and the matching fork binaries by default.
All three packings contain the same model weights, with different memory/speed trade-offs.
Expand Down Expand Up @@ -155,7 +155,7 @@ See [community-benchmarks/](community-benchmarks/) for results on different hard

### Bonsai 2 (ternary, default)

Available in GGUF and MLX 2-bit formats. Use current official llama.cpp for `Q2_0`,
Available in GGUF and MLX 2-bit formats. Use official llama.cpp b11556 or newer for `Q2_0`,
or our fork for `PQ2_0` / `PTQ1_0`. Setup installs the fork for its default `PQ2_0` download.

| Model | GGUF | MLX (2-bit) |
Expand Down Expand Up @@ -201,12 +201,13 @@ Every launcher is configured through environment variables. The most common ones

## Upstream support

- **llama.cpp:** Bonsai 2 works in official `Q2_0` format with current upstream
builds containing the merged model/Hadamard integration. Use a recent version;
older engines may recognize the format but miss the required transforms.
- **PrismML fork:** still required for `PQ2_0` and `PTQ1_0`. Some MTP/DFlash
configurations and other experimental features may also need our fork; follow
the instructions for the specific checkpoint and runtime.
- **llama.cpp:** Bonsai 2 (2B, 4B, 9B and 27B) works in official `Q2_0` format on
upstream llama.cpp from release [b11556](https://github.com/ggml-org/llama.cpp/releases/tag/b11556) onward, which merged the
model/Hadamard integration. Older engines may recognize the format but miss the
required transforms.
- **PrismML fork:** still required for `PQ2_0` and `PTQ1_0`, and for the MTP and
DFlash2 drafters on the model cards. Other experimental features may also need
our fork; follow the instructions for the specific checkpoint and runtime.
- **MLX:** the 1-bit support and Hadamard changes have merged upstream. Support
should reach mlx-lm and other MLX engines as they integrate the model support
and ship updated runtimes. A core MLX merge alone does not make every engine
Expand Down
6 changes: 3 additions & 3 deletions community-benchmarks/bonsai2/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,9 +61,9 @@ Benchmark results submitted by the community running
The default family is Bonsai 2; setup downloads `PQ2_0` and the required fork binaries.
For `PTQ1_0`, download it from the model repository above and pass its path to
`llama-bench -m` (`BONSAI_MODEL` selects a size, not a file path).
The development [Q2_0 file](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf-dev/blob/main/Ternary-Bonsai-2-27B-Q2_0-prism-fork-required.gguf) is also welcome for benchmarking via `-m`.
It uses the official llama.cpp `Q2_0` format but **currently requires our fork**
for Bonsai 2's Hadamard transform support. Upstream support is pending our PRs.
The [Q2_0 file](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/blob/main/Ternary-Bonsai-2-27B-Q2_0.gguf) is also welcome for benchmarking via `-m`.
It uses the official llama.cpp `Q2_0` format and runs on our fork or on upstream
llama.cpp [b11556](https://github.com/ggml-org/llama.cpp/releases/tag/b11556) or newer.
2. Copy [TEMPLATE-llama-cpp.md](TEMPLATE-llama-cpp.md) to
`<backend>-<hardware>-<os>.md` here (lowercase, dashes). Keep tested formats in
the same machine report, with separate commands and raw results for each.
Expand Down
11 changes: 5 additions & 6 deletions community-benchmarks/bonsai2/TEMPLATE-llama-cpp.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,14 +54,13 @@ Use the same settings when comparing packings, and list any differences.

(Paste the exact command and raw results, or state that this packing was not tested.)

### Q2_0 (optional, development)
### Q2_0 (optional)

Use [Ternary-Bonsai-2-27B-Q2_0-prism-fork-required.gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf-dev/blob/main/Ternary-Bonsai-2-27B-Q2_0-prism-fork-required.gguf)
Use [Ternary-Bonsai-2-27B-Q2_0.gguf](https://huggingface.co/prism-ml/Ternary-Bonsai-2-27B-gguf/blob/main/Ternary-Bonsai-2-27B-Q2_0.gguf)
and pass its path to `-m` with the same benchmark settings.
This is the official llama.cpp `Q2_0` format, but Bonsai 2 currently requires our
fork's Hadamard transform support. Upstream support is pending our PRs; stock
llama.cpp may load the file but produce incorrect output without those changes.
Record the exact fork build and model revision or hash.
This is the official llama.cpp `Q2_0` format. It runs on our fork and on upstream
llama.cpp from release [b11556](https://github.com/ggml-org/llama.cpp/releases/tag/b11556) onward; older upstream builds load it but produce
incorrect output. Record the exact build (fork or upstream) and model revision or hash.

(Paste the exact command and raw results, or state that this format was not tested.)

Expand Down
Loading