[bugfix] 恢复 checkpoint 时若 ZCH zch_size 已变更,提前抛出明确错误#600
Open
fooSynaptic wants to merge 1 commit into
Open
Conversation
Detect MCH buffer size mismatch from DCP metadata before loading, so continue_train/fine_tune with a changed zch_size raises a clear RuntimeError instead of an opaque validate_state AssertionError.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hi team,
我在
continue_train/fine_tune时也遇到了与 #176 相同的问题:修改特征的zch.zch_size后,checkpoint 仍能被 DCP 加载,但随后在MCHManagedCollisionModule.validate_state里以难懂的AssertionError失败。本 PR 针对这一场景,在restore_model的 DCP load 之前做兼容性检测,提前抛出可操作的RuntimeError,并补充文档与 GPU 集成回归测。Summary
restore_model入口、DCP 加载前,比对 checkpoint metadata 中_mch_sorted_raw_ids的 buffer size 与当前模型各 MCH 模块的_zch_size;不一致则 fail-fast,给出含模块前缀与 size 明细的错误信息。test_multi_tower_din_zch_finetune_zch_size_change_fails,覆盖与 issue 相同的zch_size变更 + fine_tune 路径;在docs/source/feature/zch.md注明continue_train/fine_tune不可修改zch_size。zch_size或换用 DynamicEmbedding。根因
与 #176 描述一致:
_output_segments_tensor为固定形状(1025,)的 buffer。修改zch_size不改变该 tensor 形状,DCP 仍会加载旧分区边界,最终在validate_state中以晦涩 assert 失败。Test plan
zch_size += 1→ fine_tune),RankIntegrationTest.test_multi_tower_din_zch_finetune_zch_size_change_fails通过(约 77s);fine_tune 阶段日志含ZCH (MCH) table size mismatch,符合预期本地复现(需 GPU)
预期在
zch_size += 1后的 fine_tune 阶段(与 issue 相同报错形态):Fixes #176