Add blog post: Running a big LLM across multiple GPUs with vLLM - #265

Replaces the two-GPU A40 post. The old version simulated a 2-GPU split by
capping memory on larger cards, which meant the central claim was emulated
rather than measured. This version uses a model that genuinely does not fit:
Qwen3-235B-A22B-Instruct-2507-FP8, 221 GiB on disk, 2.3x too big for one
96GB card.

Reframed away from A40 specifics to the general question of how a big model
is spread over several GPUs and how inference works once it is. Byline is now
Shubham and Saiyam. Rewritten in plain English throughout, with every flag
and every line of the docker command explained.

All three splitting modes measured on the same 4 GPUs:
- TP=4:      17.14 ms/token, 503.68 tok/s at 32 concurrent, 621,392 KV tokens
- TP=4 + EP: 18.83 ms/token, 470.93 tok/s, 623,696 KV tokens
- PP=4:      21.19 ms/token, 296.48 tok/s, 555,680 KV tokens, best TTFT

Plus three real failure modes with their actual error text: an invalid
tensor-parallel size, a genuine CUDA OOM at 2 GPUs, and a DeepGEMM
"Unknown SF transformation" crash on sm_120 that needs VLLM_USE_DEEP_GEMM=0.

Adds Part 1 on downloading and on-disk storage (safetensors shards, the HF
cache blob layout, FP8 block scales) and Part 3 on what inference actually
does (prefill versus decode, continuous batching, why capacity is set by the
KV cache). Includes the disk-pressure hazard that evicted pods on our own
test node.

Four CSS animations replace the previous three, following the existing
series pattern: three-ways-to-split, tensor split across 4 GPUs, expert
routing, and memory fit on 1 / 2 / 4 GPUs.

Signed-off-by: Saiyam Pathak <saiyam911@gmail.com>

Signed-off-by: Saiyam Pathak <saiyam911@gmail.com>

…tion-focused and theory-focused readers each get a direct path through it

…aying for theory

The two-track structure asked every reader to route themselves through a
7,100-word post before doing anything. Readers who just need a big model
serving now get a single 4,500-word path with no detours, and the mechanics
move to a second post for readers who want them.

The largest section, Step 1, led with safetensors internals and shard-size
conventions and buried the disk trap that actually took down our Kubernetes
node; the disk check now opens the step. All twelve links into the deep dive
became plain-text forward references rather than links to a URL that does not
exist yet.

Adds a cake-layers diagram at the point in Step 3 where the reader has to pick
a split, since that decision is the one place the runbook genuinely needed a
visual. The three heavier animations go with part two.

Shubham's eight-step runbook is the right shape and stays as the spine.
But the cut removed both benchmark tables while keeping the verdict that
rests on them, so three headline percentages had nothing behind them, and
it deferred the post's central question to a part two we have not scoped.
This makes the post stand alone.

Restored, placed where the reader is making a decision rather than
front-loaded as theory:

- Step 3 gains the restaurant-kitchen analogy, what a tensor-parallel
  split costs (2 all-reduces per layer, 94 layers, 188 per token), and
  why a 235B MoE is only 22B of arithmetic. The three animations that
  were left orphaned by the cut now have homes here.
- Step 7 gains the prefill-versus-decode split, which is the only thing
  that explains why pipeline parallelism wins first-token latency while
  tensor parallelism wins throughput, then both measurement tables, the
  28.71 tok/s outlier we discarded and why, and the NVLink caveat.

The TP throughput figure was re-measured at --num-prompts 640 while PP
and EP were measured at 128, so the speed table now says so. TP came out
at 507.09 against 503.68 at the smaller scale, a 0.68% difference, so the
comparison holds at both.

Fixes: the startup-log section told readers to grep for "Available KV
cache memory", which vLLM 0.27.1 does not print and which appears nowhere
in our own pasted log. Model loading time said 45 seconds in prose and
48.5 in the log 120 lines below. "Those 10 GB boundaries" referred to a
figure removed with the shard section. Two flag-table rows promised tests
that had been deleted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

The previous cake-layers.png was an AI-generated illustration whose layer
axis had a duplicated tick: it read 1, 2, 3, 4, 4, 5, 6, 7, 8, 9, 10, so
eleven marks for ten layers. Text was baked into the raster, so it could
not be corrected without regenerating the whole image.

scripts/gen-multi-gpu-cake-diagram.mjs now emits it the way every other
diagram on the site is built, as hand-rolled SVG in the Excalidraw style
of gen-hami-diagrams.mjs, with the layer count as a constant so the axis
cannot drift from the labels again. 383 KB of raster becomes 18 KB of SVG
and a 116 KB PNG for the markdown reference.

The topology matrix was authored as a markdown table but wrapped in a
```bash fence, so the first technical artifact in the post rendered as
raw text: literal `| :---: |` separators and `**GPU0**` asterisks. It is
now a real table, with the legend trimmed to the codes that matter.

The glossary sentence was hardcoded in the markdown, but GlossaryNote
already renders it above the body for anything tagged vllm or llm, so
readers saw it twice. Removed the inline copy, which also drops an
absolute link the rest of the site writes as a relative one.

`HF_HOME` prose pointed at the `hub/` directory the tree diagram is
rooted at, while the Step 5 command correctly sets its parent. Following
the prose sends the libraries looking in `hub/hub/` and re-downloads
236 GB. Spelled out which one it is and why.

Also from the review:

- "the largest data-centre GPUs top out well below 236 GB" is false;
  B300 and MI355X both carry 288 GB. Narrowed to the box we ran on.
- "Powers of two are the safe habit" pointed at -tp 8, the exact
  configuration Step 3 spends a section arguing against on this model.
- The copy-pasteable benchmark carried --seed 42 with prefix caching on,
  the trap the post warns about 80 lines later. Added the flag.
- can_device_access_peer returns True on a SYS pair, which read as a
  contradiction. It reports capability, not speed; said so.
- PP's throughput deficit was blamed on too few requests in flight, but
  32 over 4 stages is 8 each, which the post's own analogy calls full.
  Reattributed to uneven stages, which the memory table already shows.
- The flags table described three flags the command does not set and
  omitted VLLM_USE_DEEP_GEMM, without which the server does not boot.
- Step 2 told readers to run the fit check before downloading while
  sitting after the download step.
- datePublished predated every pasted log by three days.
- The memory-fit animation printed the log string Step 6 now explains
  vLLM 0.27.1 does not emit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Shubham asked for the original image back. It was never removed, it was
redrawn as an SVG diagram, and the redraw was the wrong call: the reason
to touch it at all was a single bad glyph, not the illustration.

The left axis read 1, 2, 3, 4, 4, 5, 6, 7, 8, 9, 10, eleven labels for
ten layers, with the extra 4 struck through, plus a faint smudge above
the 10. Both are painted out against the surrounding background, so the
illustration is otherwise byte-for-byte the original.

Drops cake-layers.svg and scripts/gen-multi-gpu-cake-diagram.mjs, which
generated the replacement nobody wanted. Both are recoverable from
abb7e00 if the illustration ever needs a vector counterpart.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Three of the four gpu_worker.py lines were byte-identical to the first
apart from worker index and pid, so they cost 200 words to say nothing.
One line plus a sentence noting the other three match, which is the
actual point, since agreeing exactly is what proves the split is even.

Both benchmark dumps reprinted the same preamble and the same twenty
metric lines when the post only ever cites six of them. Kept those,
elided the rest.

43 min read to 41. No evidence removed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>