Skip to content

[release/1.3] cherry-pick: finalize gradient & mHC composite metrics for internal medicine monitors (#4870)(#4871) #4879

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
From00 merged 2 commits into from
Aug 13, 2026
Merged

[release/1.3] cherry-pick: finalize gradient & mHC composite metrics for internal medicine monitors (#4870)(#4871) #4879

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
9718e54
feat: support gradient metric finalization for internal medicine moni…
yuweivvv Aug 12, 2026
621fc02
fix: finalize mHC composite metrics on substep end (#4871)
yuweivvv Aug 12, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 18 additions & 0 deletions paddleformers/trainer/trainer_callback.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1017,6 +1017,24 @@ def on_step_begin(self, args, state, control, **kwargs):
if collect is not None:
collect()

def on_optimizer_begin(self, args, state, control, scaler=None, **kwargs):
if not self._setup_done:
return

for monitor in self._monitor_dict.values():
finalize = getattr(monitor, "finalize_scaled_grad_metrics", None)

@risemeup1111 risemeup1111 Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 当前 release/1.3 锁定的 llm-internal-medicine 子模块并未提供 finalize_scaled_grad_metrics，因此这里的 capability 检测会静默跳过，h_res_logits 梯度指标不会被还原为 unscaled 值。请同步包含该方法的子模块版本（或在本仓库提供实现），并补充精确依赖下启用 AMP monitor 的回归测试，确认回调实际命中。

if finalize is not None:
finalize(scaler)

def on_substep_end(self, args, state, control, **kwargs):
if not self._setup_done:
return

for monitor in self._monitor_dict.values():
finalize = getattr(monitor, "finalize_composite_microbatch", None)

@risemeup1111 risemeup1111 Aug 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 当前锁定依赖的 PaddleFleet monitor 注册表没有 mhc_health，所以 _monitor_dict 不会包含实现 finalize_composite_microbatch 的 monitor，这个 hook 实际不会执行；梯度累积时的 mHC composite 快照仍可能被后续 micro-batch 覆盖。请同步包含该 monitor/方法的子模块版本，并增加精确依赖下 gradient_accumulation_steps>1 的回归测试。

if finalize is not None:
finalize()

def on_step_end(self, args, state, control, **kwargs):
if not self._setup_done:
return
Expand Down
Loading