Running GLM-5.2 (744 B MoE, Q2_K_XL) tensor-parallel across three DGX Spark (GB10) nodes on a switchless RDMA triangle, via a patched llama.cpp.
This repo documents the work: the patches that add cross-host tensor parallelism to llama.cpp's RPC backend, the rationale behind each, and the benchmarks, profiling, and findings from making it fast. It's a documentation + patch-set repo — not a fork copy and not a turnkey deploy.
Code lives on the fork:
Enigmatic331/llama.cpp, branchglm-dsa-tp. Applypatches/onto an upstream llama.cpp checkout (git am patches/*.patch), or build that branch directly.
Serve a 744 B model that cannot fit on one node by splitting every weight tensor across three GB10 boxes and reducing partial results over RDMA each layer — i.e. tensor parallelism (TP=3), not just layer/pipeline splitting. Layer-split already worked (~7.5 tok/s, serial ceiling); the goal was real TP, which upstream llama.cpp did not support for this model's architecture.
Two things had to be built:
- Tensor-split support for the GLM-DSA architecture in llama.cpp (it was excluded from
-sm tensor). - A cross-host all-reduce for the RPC backend that performs well on this specific topology (switchless triangle, unified memory).
| Nodes | 3 × DGX Spark — GB10 Grace-Blackwell, 128 GB unified LPDDR5X (~273 GB/s), one iGPU each |
| Model | GLM-5.2, 744 B MoE — 79 layers (3 dense + 76 MoE), hidden 6144, 256 experts / 8 active + 1 shared, MLA attention + sparse indexer. UD-Q2_K_XL ≈ 2.55 bpw ≈ 237 GB |
| Fabric | Switchless QSFP triangle — every pair of nodes directly cabled (3 edges), 200 GbE per edge, RoCEv2/RDMA, each edge on its own /24. No switch. Directed port0 → port1 ring. |
| Roles | all three run ggml-rpc-server; one node also runs the llama-server client (the TP orchestrator) |
Because GB10 is unified memory (no discrete VRAM behind a PCIe BAR), the all-reduce writes partials directly into pinned host scratch that both the NIC and the GPU address — there's no separate host↔device staging copy, and traditional GPUDirect RDMA is both unavailable and unnecessary.
llama.cpp's RPC backend is wrapped by a meta-backend that, for each row-parallel weight, splits it across the 3 devices and inserts an all-reduce at the boundary where the partial sums must combine. Each layer's forward becomes: compute local shard → all-reduce over the RDMA triangle → continue. Decode adds MTP (multi-token prediction) speculative decode using GLM's built-in nextn head — no separate draft model.
patches/ 19 patches — the full GLM-DSA tensor-parallel work over upstream llama.cpp
docs/PATCHES.md each patch: what it does and why
docs/FINDINGS.md benchmarks, profiling, the bottleneck analysis, and the remaining lever
| Metric | Value |
|---|---|
| Prefill | ~188–197 tok/s |
| Decode (MTP) | 15–21 tok/s (content-dependent) |
| Decode (no MTP) | ~12.7 tok/s |
Decode throughput tracks MTP draft acceptance (≈0.9 on predictable text → ~21 tok/s; ≈0.3 on free-form prose → ~11). Full numbers, the per-stage profiling, and the bottleneck analysis are in docs/FINDINGS.md.
Headline finding: the interconnect is not the bottleneck. Prefill is GPU-matmul-bound; decode is communication-latency-bound — the client drives ~191 per-layer round-trips per token, and the GPUs idle at ~57 %. The one remaining lever is server-side sequence execution; its design is in the findings doc.
Research code, tied to this specific 3×GB10 switchless topology. Experimental paths are env-gated and default to the proven one.