Native Apple Silicon (Metal) support — scripts/mac-run.sh + docs/MAC.md

[soobrosa]

Summary

Adds a native Apple Silicon (Mac, Metal) run path alongside the existing NVIDIA/Docker deployment. The model is served by Homebrew's llama.cpp (Metal); the FastAPI app, Pi agent, and v5-nano embedder run in a local uv virtualenv.

Purely additive — zero application-code changes (4 files, +230/-0). The original Docker/NVIDIA flow and README are untouched. This works because the model is already decoupled behind the OpenAI-compatible LLAMA_URL, so the only Mac-specific concerns are which GGUF to use, the llama-server flags, and installing the deps the Docker image normally bundles.

What's included

• scripts/mac-run.sh — launches llama-server (Metal) + server.app, honouring the existing MODEL_FILE / CTX_SIZE / SPEC_ARGS / CHAT_TEMPLATE_FILE knobs with Mac defaults.
• docs/MAC.md — full setup guide plus a "what changed vs NVIDIA/Docker" table.
• README.md — a short "Run on Apple Silicon" section (NVIDIA path unchanged).
• .env.example — a brief Mac note.

Mac-specific choices (with rationale)

• Non-MTP GGUF (unsloth/Qwen3.6-35B-A3B-GGUF): the ...-MTP-GGUF variant ships an extra speculative-draft head (blk.40) that current Homebrew llama.cpp loads as a normal layer, failing with missing tensor 'blk.40.ssm_conv1d.weight'.
• SPEC_ARGS empty (no --spec-type draft-mtp): current Homebrew builds' --spec-type only has ngram-*. Left as an opt-in env var so it re-enables automatically once a Metal build supports it.
• --flash-attn on (build wants on|off|auto, not 1) and -ngl 999 (unified memory: all layers on Metal).
• torch installed explicitly (not in server/requirements.txt, which assumed the CUDA base image); embedder kept on CPU to leave Metal memory for the LLM.

Tested

M3 Pro / 36 GB, macOS, llama.cpp build 8890. Model loads in ~30s; an end-to-end job (Jina search/read, dataroom build, zip output, live dashboard) verified. ~22-25 GB wired; CTX_SIZE=65536 fits comfortably.

[hanxiao]

LGTM, will merge. Purely additive Mac path, nothing touches the NVIDIA flow, good.

One thing for whoever wants max perf on Mac: the non-MTP llama.cpp Metal path you picked is the safe default but prefill is the bottleneck. From my own benchmarks on this box:

• MTPLX (MTP depth=2, MLX): prefill ~430 t/s, decode ~54 t/s
• llama.cpp Metal (no MTP): prefill ~90 t/s, decode ~28 t/s

So ~5x slower prefill without MTP. For interactive / long-context dataroom jobs that prefill gap hurts a lot. If you want real speed on Apple Silicon, MTP via MTPLX (MLX) is the way, not llama.cpp.

BUT there's a hard ceiling you need to know: MTP on MLX crashes at long context. I tested it — it hangs at >86K tokens (~256K chars). And it's not an MTP bug: AR mode (no MTP) on MTPLX hangs at the exact same point, so it's an MLX kernel issue, not speculative decoding. Both modes die there.

Practical takeaway:

• MTPLX/MLX: soft-cap contextWindow at ~81920 tokens. Fast, but anything past ~86K hangs.
• llama.cpp Metal: slow prefill but stable well past that — clears 128K context without dying. That's exactly why your non-MTP path is the right safe fallback.
• For genuinely huge docs, push to GCP L4 (CUDA llama.cpp with --spec-type draft-mtp) where MTP + long context both work.

So I'd keep this PR as the stable default, and add MTPLX/MLX as an opt-in fast path with the 80K cap documented. Merging.

[soobrosa]

Thanks for the detailed feedback! I dug into all three claims on an M3 Pro / 36 GB. Summary: the speed claims hold up, but the "MLX hangs at >86K" blocker is wrong, it's a memory ceiling, not a kernel hang, and it's removable.

The 86K "hang" is a fp16-KV memory OOM. At default fp16 KV, MLX OOMs between ~78K and ~92K actual tokens on 36 GB, a genuine [METAL] kIOGPUCommandBufferCallbackErrorOutOfMemory, not a kernel hang (it moves with KV precision/size, exactly as OOM does). With --kv-bits 4 the same prompts clear ~92–113K. So it's the default-fp16-KV memory ceiling, removable with one flag, not a hard architectural blocker.

Your speed numbers check out. Prefill ~533 t/s vs llama.cpp ~90 → ~5.9x (matches your ~5x). Decode ~39.5 t/s, at or above llama.cpp's MTP ~37.5. Prefill is the metric the dataroom's compaction loop is bound by, so this is the win that matters.

MTP via MTPLX works once tooling matches the model contract. My first attempt was garbage purely due to a version mismatch (mtplx 0.3.7 vs a 0.1.0-preview model); mtplx==0.1.0rc3 fixes it. On the verified 27B I get ~1.4–2.1x decode at 51–71% draft acceptance (≈ your ~72%). Caveat: I couldn't find a verified 35B-A3B MTP build (the one matching build is publisher-unverified with a weak head, ~30% acceptance), so MTP on 35B-A3B is still pending a good artifact.

Correctness held up (it's a factual dataroom, so I gated on this): greedy parity 9/10 vs llama.cpp (sole diff a synonym), and --kv-bits 4 is greedy-lossless with 15/15 fact recall at 83.5K context.

Shipped as opt-in: #4. A BACKEND={llamacpp|mlx} knob in scripts/mac-run.sh. Default is unchanged llama.cpp (stable past 128K); BACKEND=mlx runs mlx_lm.server for the ~6x prefill, auto-capped to 75K. One gotcha I hit and documented: the stock mlx-lm server has no --kv-bits (ml-explore/mlx-lm#1043, still open), so the shipped MLX path runs fp16 KV / 75K cap. I'm upstreaming kv4 to that issue; once it lands, the dataroom flips on quantized KV and raises the cap to ~85K. So: llama.cpp stays the >128K stable default, MLX is the opt-in fast path, which matches your stance.