Fix Qwen3.5 VLM garbage output: gate the RMSNorm weight shift in sanitize#403
Merged
davidkoski merged 1 commit intoJul 13, 2026
Merged
Conversation
MLXVLM Qwen35.sanitize applied the RMSNorm `+1` weight offset unconditionally, while MLXLLM Qwen35 gates it behind `hasMTPWeights || hasUnsanitizedConv1d`. On a pre-converted MLX checkpoint (conv1d already sanitized, no MTP tensors) the weights are already shifted, so the VLM path double-shifted every layernorm and produced garbage tokens — while the same weights via the MLXLLM path decoded correctly. Mirror the LLM gate so the two paths agree. Adds sanitize unit tests covering both the pre-converted (no shift) and raw-checkpoint (shift) cases. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
GoodOlClint
force-pushed
the
pr/qwen35-vlm-sanitize-normshift
branch
from
b01c6b4 to
90b8b1b
Compare
davidkoski
approved these changes
Jul 13, 2026
davidkoski
left a comment
davidkoski
left a comment
Collaborator
There was a problem hiding this comment.
Changes look good, thank you!
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Proposed changes
MLXVLM/Models/Qwen35.sanitizeapplies the RMSNorm+1weight offset unconditionally, whereasMLXLLM/Models/Qwen35.sanitizegates it behindhasMTPWeights || hasUnsanitizedConv1d.On an already-converted MLX checkpoint (conv1d weights already sanitized to a trailing dim of 1, no MTP tensors), the norm weights have already been shifted at conversion time. The VLM path shifts them a second time, doubling every
input_layernorm/post_attention_layernorm/q_norm/k_norm/model.norm. That decoheres the model and produces garbage tokens.The same weights loaded through the
MLXLLMQwen35path (which gates the shift) decode correctly — which isolates the divergence tosanitize, not the model math or the shared gated-delta kernel. Because Qwen3.5 is natively multimodal, every Qwen3.5 checkpoint routes throughVLMModelFactory→ the affected path, so all Qwen3.5 generation (text and multimodal) is broken on pre-converted checkpoints.Fix
Mirror the
MLXLLMgate in the VLMsanitize: computeshouldShiftNormWeights = hasMTPWeights || hasUnsanitizedConv1dfrom the incoming weights and apply the+1only when true. This makes the VLM path reproduce theMLXLLMnumerics.Reproduction
VLMModelFactoryand greedily decode a trivial prompt → garbage before, coherent after.vision_configremoved route toLLMModelFactoryand were already coherent (the control).Tests
Adds two
Qwen35SanitizeTestscases: a pre-converted checkpoint (conv1d already sanitized) must not shift the norm weights; a raw checkpoint (unsanitized conv1d) must. Verified end-to-end on the previously-failing checkpoint (garbage → coherent).Checklist
pre-commit run --all-filesto format my code / installed pre-commit prior to committing changes🤖 Generated with Claude Code