Skip to content

[Feat] Add Sampled-token OPD Support to RL Training #1977

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
YanhuiDua wants to merge 16 commits into
base: main
Choose a base branch
from
Open

[Feat] Add Sampled-token OPD Support to RL Training #1977

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
3a0f1e8
[Refactor] Move rollout filtering into agent loops
YanhuiDua Jul 22, 2026
5ea5763
[Feat] Add Teacher logprob client to AgentLoop
YanhuiDua Jul 22, 2026
e307f86
[Feat] Add PG-OPD token advantages
YanhuiDua Jul 22, 2026
8c6f4dd
[Feat] Propagate PG-OPD config to training agent loops
YanhuiDua Jul 22, 2026
f24a42c
[Fix] use old_logprobs instead of rollout_olgprobs to compute reverse kl
YanhuiDua Jul 27, 2026
961e243
add eval in opd config
YanhuiDua Jul 28, 2026
b6c002e
add lmdeploy backend
YanhuiDua Jul 28, 2026
ad31104
add OPDTeacherLaunchConfig, multi-teacher opd config and scripts
YanhuiDua Jul 29, 2026
b769e75
refactor cuda_visible_devices to global_device_rank to support multi-…
YanhuiDua Jul 31, 2026
8c02bad
fix mm input
YanhuiDua Jul 31, 2026
1d21b83
support teacher replicas with round-robin scoring
YanhuiDua Aug 3, 2026
d635683
fix when reward is None
YanhuiDua Aug 4, 2026
646d9a0
support prefix cache
YanhuiDua Aug 4, 2026
d5e8972
add log_level in OPDTeacherLaunchConfig
YanhuiDua Aug 4, 2026
c77eddc
add teacher session router
YanhuiDua Aug 6, 2026
0eea4a9
refactor to distillation loss and compute teacher logprob on training…
YanhuiDua Aug 21, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions examples/v1/config/reasoning_rl_qwen3p5vl_mtp_ep.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -291,10 +291,9 @@ def group_sample_filter_func(group_samples):
produce_strategy_config = AsyncProduceStrategyConfig(
over_sample_threshold=1,
enable_partial_rollout=1,
is_valid_sample_fn=group_sample_filter_func,
max_staleness=3,
)
# produce_strategy_config= SyncProduceStrategyConfig(is_valid_sample_fn=group_sample_filter_func)
# produce_strategy_config = SyncProduceStrategyConfig()

# 6. agent loop managers
agent_loop_config = SingleTurnAgentLoopConfig(
Expand All @@ -306,6 +305,7 @@ def group_sample_filter_func(group_samples):
task_name="train_task",
agent_loop_config=agent_loop_config,
judger_config=judger_config,
filter_func=group_sample_filter_func,
produce_strategy_config=produce_strategy_config,
sampler_config=SamplerConfig(dataloader_cfg=dataloader_cfg, prompt_repeat_k=prompt_repeat_k),
),
Expand Down
2 changes: 1 addition & 1 deletion examples/v1/config/rl_dapo_math_async_filter.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -157,13 +157,13 @@ def group_samples_filter_func(rollout_states):
enable_partial_rollout=True,
max_staleness=0,
tail_batch_trigger_size=256,
is_valid_sample_fn=group_samples_filter_func
)
agent_loop_manager_cfg = AgentLoopManagerConfig(
tasks=TaskSpecConfig(
task_name="train_task",
agent_loop_config=agent_loop_config,
judger_config=judger_config,
filter_func=group_samples_filter_func,
produce_strategy_config=produce_strategy_config,
sampler_config=sampler_config,
),
Expand Down
Loading