[release/1.3] cherry-pick: finalize gradient & mHC composite metrics for internal medicine monitors (#4870)(#4871) - #4879
Merged
Conversation
…tors (PaddlePaddle#4870) (cherry picked from commit 16e4ac9)
(cherry picked from commit 84f83cd)
risemeup1111
suggested changes
Aug 13, 2026
risemeup1111
left a comment
risemeup1111
left a comment
Contributor
There was a problem hiding this comment.
| 序号 | 位置 | 优先级 | 状态 |
|---|---|---|---|
| 1 | trainer_callback.py:1025 |  |
🚧 |
| 2 | trainer_callback.py:1034 | |
🚧 |
Powered by Nyanpasu with gpt-5.5 xhigh, please check the suggestions carefully.
| return | ||
|
|
||
| for monitor in self._monitor_dict.values(): | ||
| finalize = getattr(monitor, "finalize_composite_microbatch", None) |
Contributor
There was a problem hiding this comment.
mhc_health，所以 _monitor_dict 不会包含实现 finalize_composite_microbatch 的 monitor，这个 hook 实际不会执行；梯度累积时的 mHC composite 快照仍可能被后续 micro-batch 覆盖。请同步包含该 monitor/方法的子模块版本，并增加精确依赖下 gradient_accumulation_steps>1 的回归测试。
| return | ||
|
|
||
| for monitor in self._monitor_dict.values(): | ||
| finalize = getattr(monitor, "finalize_scaled_grad_metrics", None) |
Contributor
There was a problem hiding this comment.
llm-internal-medicine 子模块并未提供 finalize_scaled_grad_metrics，因此这里的 capability 检测会静默跳过，h_res_logits 梯度指标不会被还原为 unscaled 值。请同步包含该方法的子模块版本（或在本仓库提供实现），并补充精确依赖下启用 AMP monitor 的回归测试，确认回调实际命中。
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Merged in dev: #4870
Merged in dev: #4871
PR types
Bug fixes
PR changes
Others
Description
Cherry-pick #4870 and #4871 to
release/1.3.feat: support gradient metric finalization for internal medicine monitorsInternalMedicineCallback.on_optimizer_begin, which calls each monitor'sfinalize_scaled_grad_metrics(scaler)so gradient metrics are unscaled/finalized before the optimizer step.fix: finalize mHC composite metrics on substep endInternalMedicineCallback.on_substep_end, which calls each monitor'sfinalize_composite_microbatch()so composite mHC metrics are settled per microbatch instead of only at step end.Both hooks early-return when
self._setup_doneis False, and usegetattr(..., None)so monitors without these methods are unaffected.Scope:
paddleformers/trainer/trainer_callback.pyonly, +18 lines, no behavior change for users who don't enable the internal medicine monitors. Clean cherry-pick, no conflicts.