Add blog post: Running a big LLM across multiple GPUs with vLLM - #265
Draft
saiyam1814 wants to merge 8 commits into
Draft
Add blog post: Running a big LLM across multiple GPUs with vLLM#265saiyam1814 wants to merge 8 commits into
saiyam1814 wants to merge 8 commits into
Conversation
Answers a reader question about splitting a BF16 model across two A40s: how tensor parallelism partitions each layer, the memory math for whether it fits, and measured TP vs PP numbers on a pair of cards with no NVLink. All figures come from a real run (vLLM 0.27.1, Qwen3-32B BF16) on two RTX PRO 6000 cards held to a 40.47 GiB per-card budget to match a 45 GiB A40 at --gpu-memory-utilization 0.90. Adds three CSS animations following the existing series pattern: - two-gpu-tensor-split-animation: column/row-parallel split, one all-reduce - two-gpu-memory-fit-animation: 61.02 GiB against one card, then two - two-gpu-tp-vs-pp-animation: the TP/PP tradeoff plus measured results Marked draft: true pending review. Signed-off-by: Saiyam Pathak <saiyam911@gmail.com>
Deploying website with
|
| Latest commit: |
204e9fa
|
| Status: | ✅ Deploy successful! |
| Preview URL: | https://c9a309b9.website-dab.pages.dev |
| Branch Preview URL: | https://blog-two-gpu-vllm.website-dab.pages.dev |
Drafts are filtered out of production builds (INCLUDE_DRAFTS is false when NODE_ENV is production), so the preview deploy 404s on the post while the flag is set. Removing it makes the PR preview reviewable. The PR itself stays in draft, so nothing publishes until it is marked ready and merged. Signed-off-by: Saiyam Pathak <saiyam911@gmail.com>
Replaces the two-GPU A40 post. The old version simulated a 2-GPU split by capping memory on larger cards, which meant the central claim was emulated rather than measured. This version uses a model that genuinely does not fit: Qwen3-235B-A22B-Instruct-2507-FP8, 221 GiB on disk, 2.3x too big for one 96GB card. Reframed away from A40 specifics to the general question of how a big model is spread over several GPUs and how inference works once it is. Byline is now Shubham and Saiyam. Rewritten in plain English throughout, with every flag and every line of the docker command explained. All three splitting modes measured on the same 4 GPUs: - TP=4: 17.14 ms/token, 503.68 tok/s at 32 concurrent, 621,392 KV tokens - TP=4 + EP: 18.83 ms/token, 470.93 tok/s, 623,696 KV tokens - PP=4: 21.19 ms/token, 296.48 tok/s, 555,680 KV tokens, best TTFT Plus three real failure modes with their actual error text: an invalid tensor-parallel size, a genuine CUDA OOM at 2 GPUs, and a DeepGEMM "Unknown SF transformation" crash on sm_120 that needs VLLM_USE_DEEP_GEMM=0. Adds Part 1 on downloading and on-disk storage (safetensors shards, the HF cache blob layout, FP8 block scales) and Part 3 on what inference actually does (prefill versus decode, continuous batching, why capacity is set by the KV cache). Includes the disk-pressure hazard that evicted pods on our own test node. Four CSS animations replace the previous three, following the existing series pattern: three-ways-to-split, tensor split across 4 GPUs, expert routing, and memory fit on 1 / 2 / 4 GPUs. Signed-off-by: Saiyam Pathak <saiyam911@gmail.com>
saiyam1814
changed the title
A reader asked whether the shard count can be changed at download time and whether there is a standard. Adds two subsections to Part 1: shards are fixed by the publisher and recorded in model.safetensors.index.json, re-sharding is a local save_pretrained(max_shard_size=...) operation, and the Hub's <200GB recommendation plus 500GB hard limit explain why publishers land around 5-10GB. This model uses a 10GB cap: 23 shards of exactly 10.00 GB plus a 6.45 GB remainder. Also notes that shard count does not affect serving, because safetensors are memory-mapped. Signed-off-by: Saiyam Pathak <saiyam911@gmail.com>
Signed-off-by: Saiyam Pathak <saiyam911@gmail.com>
…tion-focused and theory-focused readers each get a direct path through it
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
A new blog post answering a reader question: someone with two A40s (45 GiB usable each) wanted to serve a BF16 model split across both cards, and asked how the weight splitting actually works.
The post covers:
Where the numbers come from
Everything measured on two RTX PRO 6000 Blackwell cards held to a 40.47 GiB per-card budget (
--gpu-memory-utilization 0.426) so the memory behaviour matches a 45 GiB A40 running at0.90. Model is Qwen3-32B in BF16, a 61.02 GiB checkpoint, on vLLM 0.27.1.Headline results:
Worth flagging for reviewers: vLLM's docs recommend pipeline parallelism over tensor parallelism on boxes without NVLink for "higher throughput", and that did not reproduce here. The post reports it as a miss rather than bending the result, and explains why the crossover may sit differently on Gen4 A40s.
New animations
Three CSS-only animations following the existing series pattern (no hooks, no
use client, theme variables,prefers-reduced-motionguards), registered inlib/markdown.js:{{two-gpu-tensor-split-animation}}the split and the all-reduce{{two-gpu-memory-fit-animation}}61.02 GiB against one card, then two{{two-gpu-tp-vs-pp-animation}}the tradeoff plus measured resultsVerified rendering in both light and dark mode.
Notes for review
draft: truehad to be removed so the post renders on the Cloudflare preview: drafts arefiltered out of production builds (
INCLUDE_DRAFTSis false whenNODE_ENVis production), sothe preview 404s while the flag is set. This PR stays in draft as the gate, so nothing
publishes until it is marked ready and merged. Re-add the flag if you would rather hold it after merge.
scripts/gen-two-gpu-vllm-cover.mjsand converted withrsvg-convert -w 1200 -h 630.