[megatron, vllm, recipe] feat: add Qwen2.5-1.5B GDPO training on Ascend

[RordChang]

What does this PR do?

Adds a canonical Qwen2.5-1.5B-Instruct GDPO training example for Ascend NPU, using Megatron/MBridge for the actor and reference workers and vLLM-Ascend for rollout.

• Adds examples/gdpo_trainer/run_qwen2_5_1_5b_megatron.sh with environment overrides, auto resume, timestamped logs, and a 100-step default.
• Connects the existing GDPO advantage estimator and reward manager to the RLLA accuracy_reward and format_reward components.
• Updates the concise GDPO README with the Megatron + vLLM-Ascend entry point.

Related task: verl-project/verl-ascend-recipe#18

Checklist Before Starting

• Search for similar PRs. Query: https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+GDPO+Megatron+Ascend
• Format the PR title as [{modules}] {type}: {description}.

Test

• pre-commit run --all-files --show-diff-on-failure --color=always: all hooks passed.
• CPU validation: 4 passed, 1 warning.
• bash -n examples/gdpo_trainer/run_qwen2_5_1_5b_megatron.sh: passed.

Experiment Results

Platform: 4 x Ascend 910B.

Metric	Result
Completed training steps	100/100
Reward mean, first 10 steps	-2.0765
Reward mean, last 10 steps	-0.3575
Reward change	+1.7189
Reward linear slope	+0.01421/step
Mean throughput per NPU	427.1 token/s/NPU
Mean total throughput	1708.4 token/s

Full redacted 100-step training log:
training_100step_sanitized.log

API and Usage Example

MODEL_PATH=/path/to/Qwen2.5-1.5B-Instruct \
DATA_ROOT=/path/to/data \
NPUS_PER_NODE=4 \
bash examples/gdpo_trainer/run_qwen2_5_1_5b_megatron.sh

DATA_ROOT should contain rlla_4k/train.parquet and rlla_4k/test.parquet. Additional Hydra overrides can be appended to the command.

Design & Code Changes

• run_qwen2_5_1_5b_megatron.sh composes the current Megatron model engine, MBridge conversion, vLLM rollout, GDPO estimator, and GDPO reward manager.
• The README change remains an entry-point-level addition; detailed results stay in the task issue.

Checklist Before Submitting

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always.
• Add / Update the documentation.
• Once the PR is ready for CI, send a message in the ci-request channel or the Feishu group.
• The recipe submodule is not changed, so no submodule reference update is required.

AI assistance disclosure: AI tools assisted with implementation, validation, and documentation preparation. The author reviewed the complete diff and takes responsibility for the submission.

[Copilot]

Pull request overview

Adds a canonical GDPO training entry point for Qwen2.5-1.5B on Ascend NPUs (Megatron/MBridge actor+ref, vLLM-Ascend rollout), and introduces CPU tests to validate the GDPO multi-reward wiring and RLLA reward outputs.

Changes:

• Add an Ascend-focused GDPO launcher script (run_qwen2_5_1_5b_megatron.sh) with environment defaults, logging, and Hydra overrides.
• Add CPU unit tests covering GDPO per-reward-dimension normalization, required reward keys, and registry wiring; also validates RLLA score breakdown.
• Update the GDPO trainer README to document the new Megatron + vLLM-Ascend entry point.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
tests/trainer/ppo/test_gdpo_on_cpu.py	New CPU tests validating GDPO advantage estimator behavior, registry wiring, and RLLA reward outputs.
examples/gdpo_trainer/run_qwen2_5_1_5b_megatron.sh	New Ascend NPU GDPO training script integrating Megatron/MBridge + vLLM-Ascend rollout + GDPO reward manager.
examples/gdpo_trainer/README.md	Documents the new Ascend entry point and updates dataset/environment notes.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[fangweii1]

What does this PR do?

Adds a canonical Qwen2.5-1.5B-Instruct GDPO training example for Ascend NPU, using Megatron/MBridge for the actor and reference workers and vLLM-Ascend for rollout.

• Adds examples/gdpo_trainer/run_qwen2_5_1_5b_megatron.sh with environment overrides, auto resume, timestamped logs, and a 100-step default.
• Connects the existing GDPO advantage estimator and reward manager to the RLLA accuracy_reward and format_reward components.
• Adds CPU tests for per-reward-dimension normalization, required reward components, registry wiring, and RLLA score outputs.
• Updates the concise GDPO README with the Megatron + vLLM-Ascend entry point.

Related task: verl-project/verl-ascend-recipe#18

Checklist Before Starting

• Search for similar PRs. Query: https://github.com/verl-project/verl/pulls?q=is%3Apr+is%3Aopen+GDPO+Megatron+Ascend
• Format the PR title as [{modules}] {type}: {description}.

Test

• pre-commit run --all-files --show-diff-on-failure --color=always: all hooks passed.
• pytest -q tests/trainer/ppo/test_gdpo_on_cpu.py: 4 passed, 1 warning.
• bash -n examples/gdpo_trainer/run_qwen2_5_1_5b_megatron.sh: passed.
• 100/100 training steps completed on 4 x Ascend 910B4.
• Mean reward over the first/last 10 steps: -2.0765 -> -0.3575 (+1.7189).
• Mean throughput: 427.1 token/s/NPU, 1708.4 token/s across four NPUs.

The sanitized full training log, metrics, and environment details are provided in the related task issue.

API and Usage Example

MODEL_PATH=/path/to/Qwen2.5-1.5B-Instruct \
DATA_ROOT=/path/to/data \
NPUS_PER_NODE=4 \
bash examples/gdpo_trainer/run_qwen2_5_1_5b_megatron.sh

DATA_ROOT should contain rlla_4k/train.parquet and rlla_4k/test.parquet. Additional Hydra overrides can be appended to the command.

Design & Code Changes

• run_qwen2_5_1_5b_megatron.sh composes the current Megatron model engine, MBridge conversion, vLLM rollout, GDPO estimator, and GDPO reward manager.
• test_gdpo_on_cpu.py verifies the multi-reward contract and the registered GDPO/RLLA integration without accelerator requirements.
• The README change remains an entry-point-level addition; detailed results stay in the task issue.

Checklist Before Submitting

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit install && pre-commit run --all-files --show-diff-on-failure --color=always.
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. test_gdpo_on_cpu.py is automatically collected by the existing CPU unit-test workflow; no workflow file change is required.
• Once the PR is ready for CI, send a message in the ci-request channel or the Feishu group.
• The recipe submodule is not changed, so no submodule reference update is required.

AI assistance disclosure: AI tools assisted with implementation and documentation preparation. The author reviewed the complete diff and takes responsibility for the submission.

请完善一下 1. 不需要提交test相关的代码 2. 提交实验数据

[RordChang]

@fangweii1 已按反馈完成修改：test 相关代码已删除，实验数据和完整脱敏训练日志链接已补充至 PR 正文。

[wucong25]

统一先提到verl-ascend-recipe：verl-ascend-recipe哈，此pr暂时关闭