Skip to content

fix(deepseek-v4): preserve grouped wo_a LoRA semantics - #2771

Open
zhoutong-hai wants to merge 1 commit into
radixark:mainfrom
zhoutong-hai:codex/deepseek-v4-grouped-wo-a
Open

fix(deepseek-v4): preserve grouped wo_a LoRA semantics#2771
zhoutong-hai wants to merge 1 commit into
radixark:mainfrom
zhoutong-hai:codex/deepseek-v4-grouped-wo-a

Conversation

@zhoutong-hai

@zhoutong-hai zhoutong-hai commented Aug 26, 2026

Copy link
Copy Markdown

Problem

DeepSeek-V4 wo_a is not a single dense projection. It stores G independent group weights in a flattened column-parallel matrix, and each input group must use only the matching output group.

A generic LoRALinear wrapper creates two problems:

  1. native forward reads self.wo_a.weight, but the wrapper owns the base module under to_wrap;
  2. applying the flattened LoRA projection to every input group produces a G × G cross-group result, while the base operator uses only the group diagonal.

Fix

Add a V4-specific grouped projection helper that:

  • unwraps the base weight from the adapter wrapper;
  • computes the native grouped base projection;
  • evaluates the Bridge adapter through its public adapter_forward contract;
  • reshapes the flattened delta and selects only matching input/output groups;
  • preserves the base-only path when the adapter is disabled; and
  • fails closed on incompatible base, input, or adapter-output layouts.

The helper is used only by V4 wo_a; other LoRA projections retain their generic wrappers.

Validation

  • Model-free numerical test compares against the exact effective grouped weight.
  • Gradients reach both LoRA factors.
  • Disabled-adapter and incompatible-layout cases are covered.
  • tests/fast/plugins/test_deepseek_v4_lora.py: 3 passed.
  • Ruff and diff checks pass.

Ordering

This PR should merge before #2772. That PR enables and wraps wo_a, and therefore depends on this grouped execution path.

Related rollout-engine execution PR: sgl-project/sglang#36579. V4 target registration and dimensions: sgl-project/sglang#36578.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant