NVIDIA-NeMo · HuiyingLi · Aug 29, 2026 · Aug 28, 2026
@@ -21,6 +21,7 @@
 </div>
 
 ## 📣 News and Discussions
+- [08/28/2026][**Ox Alpha / GLM-5.3-Flash**](https://huggingface.co/zai-org/GLM-5.3-Flash) We now support fine-tuning Z.ai's 320B-A18B hybrid-attention MoE VLM with packed CP and EP. Check out the [MedPix EP72/CP2 recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/glm5_next/glm5_3_flash_medpix_packed2k_ep72_cp2_100steps.yaml) and [model coverage page](https://github.com/NVIDIA-NeMo/Automodel/blob/main/docs/model-coverage/vlm/thudm/glm-5-3-flash.mdx).
 - [08/26/2026][**Qwen3.8-Flash-Next**](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) We now support full-parameter fine-tuning for the 180B hybrid-attention Engram MoE with FlexAttention sparse QSA, packed (THD) sequences, and model-owned context parallelism. Check out the [HellaSwag EP64 recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/qwen/qwen3_8_flash_next_180b_hellaswag_ep64.yaml) and [model coverage page](https://github.com/NVIDIA-NeMo/Automodel/blob/main/docs/model-coverage/llm/qwen/qwen3-8-flash-next.mdx).
 - [08/21/2026][**DFlash 2**](https://github.com/NVIDIA-NeMo/Automodel/blob/main/docs/guides/speculative/dflash.mdx) We now support training DFlash 2 drafters: the DFlash block-diffusion backbone plus a two-tap dynamic convolution and a pairwise path selector for higher acceptance lengths. Check out the [Qwen3-8B recipe](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/speculative/dflash/qwen3_dflash2.yaml) and the [speculative decoding overview](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/speculative/README.md).
 - [08/14/2026][**Qwen3.8-27B**](https://huggingface.co/Qwen/Qwen3.8-27B) We now support full-parameter SFT and LoRA fine-tuning for Qwen's dense 27B vision-language model. Check out the [SFT](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/qwen3_8/qwen3_8_27b.yaml) and [LoRA](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/qwen3_8/qwen3_8_27b_lora.yaml) recipes, [fine-tuning guide](https://github.com/NVIDIA-NeMo/Automodel/blob/main/docs/guides/vlm/qwen3-8.mdx), and [model coverage page](https://github.com/NVIDIA-NeMo/Automodel/blob/main/docs/model-coverage/vlm/qwen/qwen3-8-vl.mdx).
@@ -182,7 +183,7 @@ Model Coverage and 🤗 Ecosystem compatibility
 - ✅ **Diffusion** - Pretraining and LoRA finetuning for image/video diffusion models (Qwen-Image, FLUX, Wan2.1, Wan2.2-T2V-A14B, Hunyuan).
 - ✅ **dLLM** - Discrete diffusion LM finetuning (LLaDA, LLaDA2, Nemotron-Labs-Diffusion, DiffusionGemma).
 - ✅ **Retrieval** - Bi-encoder and cross-encoder training with in-batch negative sampling.
-- ✅ **Extended MoE support** - GPT-OSS, Kimi K3, Qwen3 / Qwen3.5 / Qwen3.6 MoE, Qwen-next, MiniMax-M2.x, GLM-4.7 / GLM-5 / GLM-5.1 / GLM-5.2, DeepSeek V3.2 / V4 / V4-Flash, ERNIE 4.5, MiMo-V2-Flash, Ling 2.0, Hy3-preview.
+- ✅ **Extended MoE support** - GPT-OSS, Kimi K3, Qwen3 / Qwen3.5 / Qwen3.6 MoE, Qwen-next, MiniMax-M2.x, GLM-4.7 / GLM-5 / GLM-5.1 / GLM-5.2 / GLM-5.3-Flash, DeepSeek V3.2 / V4 / V4-Flash, ERNIE 4.5, MiMo-V2-Flash, Ling 2.0, Hy3-preview.
 
 Agentic Development and UX
 - ✅ **Agent-friendly skills** - Curated [`skills/`](https://github.com/NVIDIA-NeMo/Automodel/tree/main/skills) for common dev tasks (recipe runs, model onboarding, CI).
@@ -360,6 +361,7 @@ NeMo AutoModel provides native support for a wide range of models available on t
 | **LLM** | **Baichuan** | [`baichuan-inc/Baichuan2-7B-Chat`](https://huggingface.co/baichuan-inc/Baichuan2-7B-Chat) | [SFT](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/baichuan/baichuan_2_7b_squad.yaml), [PEFT](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/baichuan/baichuan_2_7b_squad_peft.yaml), [FP8](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/llm_finetune/baichuan/baichuan_2_7b_mock_fp8.yaml) |
 | **VLM** | **Gemma** | [`google/gemma-3-4b-it`](https://huggingface.co/google/gemma-3-4b-it) | [SFT](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/gemma3/gemma3_vl_4b_cord_v2.yaml), [PEFT](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/gemma3/gemma3_vl_4b_cord_v2_peft.yaml) |
 |  |  | [`google/gemma-3n-e4b-it`](https://huggingface.co/google/gemma-3n-e4b-it) | [SFT](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/gemma3n/gemma3n_vl_4b_medpix.yaml), [PEFT](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/gemma3n/gemma3n_vl_4b_medpix_peft.yaml) |
+| **VLM** | **Ox Alpha / GLM-5.3-Flash** | [`zai-org/GLM-5.3-Flash`](https://huggingface.co/zai-org/GLM-5.3-Flash) | [SFT](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/glm5_next/glm5_3_flash_medpix_packed2k_ep72_cp2_100steps.yaml) |
 | **VLM** | **North Micro Vision** | [`CohereLabs/North-Micro-Vision-Instruct`](https://huggingface.co/CohereLabs/North-Micro-Vision-Instruct) | [PEFT](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/cohere_micro_vision/north_micro_vision_rdr.yaml) |
 
 > [!NOTE]

@@ -4,38 +4,179 @@ description: "Fine-tune the GLM-5.3-Flash mixture-of-experts vision-language mod
 slug: model-coverage/vision-language-models/thudm/glm-5-3-flash
 ---
 
-[GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) is a mixture-of-experts vision-language model with a hybrid Kimi Delta Attention and Dynamic Sparse Attention language backbone.
+[GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash) is a
+320-billion-parameter, 18-billion-active vision-language model. Its language
+backbone combines Kimi Delta Attention (KDA), KPool-indexed DeepSeek Sparse
+Attention (DSA), Mixture-of-Experts (MoE) layers, and four-stream
+Manifold-Constrained Hyper-Connections (mHC).
 
 <Info>
 
 | | |
 |---|---|
 | **Task** | Image-Text-to-Text |
 | **Architecture** | `Glm5NextForConditionalGeneration` |
-| **Language Module** | Hybrid KDA / KPool-DSA MoE decoder |
+| **Parameters** | 320B total, 18B active |
+| **Decoder** | 45 layers: 34 KDA and 11 KPool-DSA layers |
+| **Experts** | 288 routed experts, top-8 routing, plus one shared expert |
+| **Context length** | 1,048,576 in the checkpoint configuration; validated here at 2,048 and 4,096 tokens |
 | **Training Precision** | BF16 after FP8 checkpoint dequantization |
 | **HF Org** | [zai-org](https://huggingface.co/zai-org) |
 
 </Info>
 
+## Architecture
+
+- A 24-block vision encoder and patch merger project image features into the
+  4,096-wide language hidden state.
+- The 45-layer decoder repeats three KDA layers followed by one KPool-DSA layer,
+  with a final KDA layer. Four persistent mHC streams wrap attention and
+  feed-forward sublayers.
+- The first three decoder layers use dense feed-forward networks. The remaining
+  42 layers use 288 routed experts with top-8 routing and one shared expert.
+- KPool-DSA selects up to 2,048 token positions through four-token pools before
+  running sparse latent attention over a shared 512-wide K/V representation.
+
 ## Supported Training Path
 
-NeMo AutoModel provides a native configuration, image processor, vision tower, language model, and Hugging Face state-dict adapter for GLM-5.3-Flash. The supported full-model initialization path loads the base checkpoint through distributed checkpointing; single-GPU full-checkpoint loading is not supported.
+NeMo AutoModel provides a native configuration, image processor, vision tower,
+language model, and Hugging Face state-dict adapter. The implementation runs on
+the repository's pinned `transformers==5.12.1`; users do not need Transformers
+5.16, a nightly Transformers build, or `trust_remote_code`.
 
-The current onboarding supports image training. Video inputs, tensor parallelism, and pipeline parallelism are not enabled for this model. The validated recipe uses FSDP2 with expert parallelism, contiguous packed context parallelism, and HybridEP dispatch.
+The supported base-checkpoint initialization path uses distributed checkpoint
+loading and dequantizes the released FP8 weights for BF16 training. Full-model
+single-GPU checkpoint loading and training are not supported.
+
+The current path supports image training with FSDP2, activation checkpointing,
+expert parallelism, and contiguous packed context parallelism. Tensor
+parallelism, pipeline parallelism, and video training are not enabled. An EP
+size must evenly divide the model's 288 routed experts and fit the available GPU
+memory; EP72 is the published validated topology, not a model requirement.
 
 ## Attention Backends
 
-- KDA layers use Flash Linear Attention kernels.
-- Sparse DSA layers support the SDPA reference path.
-- On SM90 or later, `backend.attn: cudnn` uses FlashMLA forward with cuDNN sparse-attention backward. This optional path requires compatible FlashMLA and cuDNN Frontend installations.
+- KDA layers use Flash Linear Attention (FLA) kernels. For CP, FLA carries the
+  recurrent state across contiguous sequence shards while preserving packed
+  document boundaries.
+- Sparse DSA layers support an SDPA numerical-reference path.
+- On SM90 or later, `backend.attn: cudnn` uses FlashMLA for the sparse forward
+  pass and cuDNN Frontend for backward. This is the backend selected by the
+  published recipe.
+
+## Validated Configuration
+
+| Area | Validated Configuration |
+|---|---|
+| Training | FSDP2 with whole-layer activation checkpointing; language backbone trainable; embeddings and vision tower frozen |
+| Parallelism | TP1 / PP1 / EP72 / CP2; HybridEP dispatch; four routed experts per GPU |
+| Workload | `mmoukouba/MedPix-VQA`, packed THD sequences of 2,048 tokens |
+| Batch | Local batch size 1, global batch size 144, four gradient-accumulation microsteps |
+| Scale | 9 nodes / 72 H100 GPUs |
+| Duration | 100 optimizer steps |
 
 ## Example Recipe
 
-- [Full SFT — MedPix, packed 2K, EP72 + CP2](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/glm5_next/glm5_3_flash_medpix_packed2k_ep72_cp2_100steps.yaml)
+| Recipe | Dataset | Description |
+|---|---|---|
+| [glm5_3_flash_medpix_packed2k_ep72_cp2_100steps.yaml](https://github.com/NVIDIA-NeMo/Automodel/blob/main/examples/vlm_finetune/glm5_next/glm5_3_flash_medpix_packed2k_ep72_cp2_100steps.yaml) | MedPix-VQA | Language-backbone SFT with packed 2K sequences, EP72, and CP2 |
+
+## Install and Run
+
+Clone and install NeMo AutoModel from source:
+
+```bash
+git clone https://github.com/NVIDIA-NeMo/Automodel.git
+cd Automodel
+uv sync --locked --all-groups --all-extras --extra vlm-media
+```
+
+The cuDNN backend also requires FlashMLA's `flash_mla_sparse_fwd`, which is not
+published as a complete source distribution on PyPI. Install the tested
+revision with its submodules:
+
+```bash
+git clone --recursive https://github.com/deepseek-ai/FlashMLA.git /tmp/FlashMLA
+git -C /tmp/FlashMLA checkout b7643bd54521f563b839b98289b5cd048c062ba2
+git -C /tmp/FlashMLA submodule update --init --recursive
+uv pip install --no-build-isolation /tmp/FlashMLA
+```
+
+The recipe requires 9 nodes with 8 GPUs per node. Launch it through the cluster
+launcher from inside the repository:
+
+```bash
+uv run automodel --nproc-per-node=8 examples/vlm_finetune/glm5_next/glm5_3_flash_medpix_packed2k_ep72_cp2_100steps.yaml
+```
+
+<Note>
+
+Use the [Slurm Launcher Guide](/job-launchers/slurm-cluster) to configure the
+multi-node launch. If FlashMLA or cuDNN Frontend is unavailable, change
+`model.backend.attn` to `sdpa`; KDA still requires the FLA dependency.
+
+</Note>
+
+## Numerical Validation
+
+### Hugging Face Logit Parity
+
+A four-layer model containing KDA, KDA, KDA, and DSA was compared with the
+Transformers reference at sequence length 4,096 over the full 154,880-token
+vocabulary. All 135 expected tensors loaded with no missing, unexpected, or
+shape-mismatched keys.
+
+| Metric | Result |
+|---|---:|
+| Mean KL, HF to NeMo AutoModel | 0.0002477953 |
+| P95 KL | 0.0005324535 |
+| Max KL | 0.0271428488 |
+| Logits cosine similarity | 0.9999251899 |
+| Top-1 token agreement | 98.33984375% |
+
+### Packed CP1 / CP8 Training Parity
+
+The full model trained for 100 steps with the same MedPix packed-2K workload,
+global batch size 144, EP144, TP1, PP1, and the SDPA DSA reference backend.
+
+| Run | Resources | Peak Logged Memory | Step 99 Loss | W&B |
+|---|---:|---:|---:|---|
+| CP1 | 18 nodes / 144 H100s | 38.89 GiB | 1.2344 | [xcu2nc2k](https://wandb.ai/Nemo-automodel/huiyingl_workspace/runs/xcu2nc2k) |
+| CP8 | 18 nodes / 144 H100s | 41.04 GiB | 1.2328 | [21yl4mj9](https://wandb.ai/Nemo-automodel/huiyingl_workspace/runs/21yl4mj9) |
+
+Across all 100 matched steps, the loss curves had mean absolute difference
+0.001879, RMSE 0.002800, and maximum absolute difference 0.0130.
+
+### cuDNN Sparse Attention Performance
+
+The SDPA and cuDNN backends were compared end to end with the published EP72 / CP2
+topology. Both trained for 100 steps on the same MedPix packed-2K workload; the
+throughput window covers steps 10 through 99.
+
+| Backend | Mean TPS | Median TPS | Mean / Peak Memory | Training Loop | W&B |
+|---|---:|---:|---:|---:|---|
+| SDPA | 8,378.15 | 8,417.33 | 57.160 / 57.68 GiB | 37:15 | [yn0z3ym3](https://wandb.ai/Nemo-automodel/huiyingl_workspace/runs/yn0z3ym3) |
+| cuDNN | 9,489.04 | 9,538.12 | 57.185 / 57.78 GiB | 33:18 | [kbp425qx](https://wandb.ai/Nemo-automodel/huiyingl_workspace/runs/kbp425qx) |
+
+The cuDNN path improved mean throughput by 13.26% and median throughput by
+13.32%, with 0.025 GiB/GPU higher mean steady-state memory. The two loss curves
+had mean absolute difference 0.002648, RMSE 0.003076, and final losses 1.2359
+for SDPA and 1.2337 for cuDNN.
+
+## Current Scope
 
-The recipe is sized for 9 nodes with 8 GPUs per node. It uses local batch size 1 and four gradient-accumulation microsteps to form a global batch of 144 packed samples.
+- Image training is supported; video training is not.
+- TP and PP are not supported for this model. Packed contiguous CP and EP are
+  supported with TP1 and PP1.
+- Full-model single-GPU checkpoint loading and training are not supported. Use
+  the distributed checkpoint initialization path.
+- The recipe freezes the embeddings and vision tower while training the
+  language backbone.
+- The cuDNN sparse-attention path requires SM90 or later, cuDNN Frontend, and
+  FlashMLA. Use `backend.attn: sdpa` as the portable reference path.
 
-## Hugging Face Model Card
+## References
 
 - [zai-org/GLM-5.3-Flash](https://huggingface.co/zai-org/GLM-5.3-Flash)
+- [GLM-5 technical report](https://arxiv.org/abs/2602.15763)
+- [FlashMLA](https://github.com/deepseek-ai/FlashMLA)