[dinodeep]

Description

This PR integrates the updated support for TE MoE block for supporting DSv4. It does the following

• pipes hash routing through TE MoE block's moe(...) call so that hash routing layers can correctly use TE MoE block
• uses the expert counts returned from TE MoE block to support updating the routed bias via routed_bias_update_rate
• change the default initialization (not checkpointed) of DSv4's hash routing self.tid2eid from zero array to randomly initialized experts, preventing (1) a single token going to the same expert multiple times (2) having a balanced distribution of tokens to experts (3) prevent infinite gradients when all tokens go to the same expert

This is a useful change so that TE MoE block can work with DSv4 out of the box.

Tests

Tested by integrating a end-to-end train test to validate full execution.

Checklist

Before submitting this PR, please make sure (put X in square brackets):

• I have performed a self-review of my code. For an optional AI review, add the gemini-review label.
• I have necessary comments in my code, particularly in hard-to-understand areas.
• I have run end-to-end tests tests and provided workload links above if applicable.
• I have made or will make corresponding changes to the doc if needed, including adding new documentation pages to the relevant Table of Contents (toctree directive) as explained in our documentation.

[gemini-code-assist]

Code Review

This pull request adds support for hash routing and routed bias updates in TransformerEngine MoE blocks, particularly for DeepSeek-V4. It introduces the moe_from_routing path when hash routing is enabled, handles routed bias updates, and adds a corresponding integration test. However, a critical issue was identified where input_ids (which is required for hash routing) is not forwarded by the caller DeepSeekMoELayer in src/maxtext/models/deepseek.py to self.mlp_op or self.DeepSeekMoeBlock_0, which will cause a runtime ValueError.

[codecov]

Codecov Report

❌ Patch coverage is 31.81818% with 30 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
src/maxtext/layers/moe.py	30.00%	28 Missing ⚠️
src/maxtext/models/deepseek.py	50.00%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!