[megatron, vllm, recipe] feat: add Qwen2.5-0.5B CISPO training on Ascend

[RordChang]

What does this PR do?

Adds a Qwen2.5-0.5B-Instruct CISPO training example using Megatron for actor/reference training and vLLM-Ascend for rollout on Ascend NPUs.

• Adds a canonical Megatron + vLLM-Ascend launcher with environment-variable overrides, automatic checkpoint resume, and timestamped logs.
• Connects the registered CISPO policy loss to GRPO advantages with lower/upper clipping of 10.0 / 0.2 and token-mean aggregation.
• Adds a concise entry point to the existing CISPO README and links to the full environment, logs, and 100-step results.

Related:
verl-project/verl-ascend-recipe#17

Checklist Before Starting

• Searched relevant open PRs:
  • CISPO Ascend
  • CISPO Megatron
  • PR [algo] feat: add cispo #4508 already provides the registered CISPO policy loss; no open PR covers this Qwen2.5-0.5B, Megatron, and vLLM-Ascend training combination.
• Formatted the title as [{modules}] {type}: {description}.

Test

• pre-commit run --all-files --show-diff-on-failure --color=always: passed.
• CISPO objective and registry CPU validation: 2 passed.
• bash -n examples/cispo_trainer/run_qwen2_5_0_5b_megatron.sh: passed.

Experiment Results

Platform: 4 x Ascend 910B.

Metric	Result
Completed training steps	100/100
Reward mean, first 10 steps	0.010156
Reward mean, last 10 steps	0.281250
Reward change	+0.271094
Reward linear slope	+0.00388862/step
Mean throughput per NPU	258.8282 token/s/NPU
Mean total throughput	1035.3128 token/s

Full redacted 100-step training log:
training_100step_sanitized.log

API and Usage Example

python3 examples/data_preprocess/gsm8k.py \
    --local_save_dir /path/to/data/gsm8k

MODEL_PATH=/path/to/Qwen2.5-0.5B-Instruct \
DATA_ROOT=/path/to/data \
NPUS_PER_NODE=4 \
TOTAL_TRAINING_STEPS=100 \
bash examples/cispo_trainer/run_qwen2_5_0_5b_megatron.sh

Additional Hydra overrides can be appended to the command.

Design & Code Changes

• The existing cispo policy loss remains the single implementation of the objective.
• The Megatron actor calls the policy loss through the common PPO loss path; actor/reference training uses TP=2 and BF16.
• vLLM-Ascend rollout uses TP=2, four responses per prompt, chunked prefill, prefix caching, and bucketed weight synchronization.
• The launcher exposes model, data, parallelism, batch size, checkpoint, and logging settings through environment variables.
• The existing CISPO README only gains the new entry point and a link to complete results.

Checklist Before Submitting

• Read the Contribute Guide.
• Applied pre-commit checks.
• Updated documentation.
• CPU validation results are reported above.
• Request CI after the PR is ready.
• Recipe submodule update: N/A; this PR does not modify the submodule.

AI assistance: OpenAI Codex was used for codebase inspection and preparation of the implementation, validation, and documentation. The submitter reviewed the changes and validation results.

[Copilot]

Pull request overview

Adds an Ascend-focused CISPO training example for Qwen2.5-0.5B-Instruct that uses Megatron for actor/reference training and vLLM-Ascend for rollout, and extends CPU unit coverage for CISPO policy-loss behavior and registration.

Changes:

• Added CPU tests covering CISPO clipped-ratio behavior, stop-gradient semantics, response masking, and registry wiring.
• Added a Megatron + vLLM-Ascend launcher script for Qwen2.5-0.5B-Instruct with environment-variable overrides and timestamped logging.
• Updated the CISPO trainer README with an entry point for the new Ascend recipe and a link to validated artifacts.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File	Description
tests/trainer/ppo/test_dynamic_policy_losses_on_cpu.py	Adds CPU unit tests for CISPO loss clipping + stop-gradient behavior and verifies the loss is registered.
examples/cispo_trainer/run_qwen2_5_0_5b_megatron.sh	Introduces an Ascend NPU training launcher (Megatron actor/ref + vLLM-Ascend rollout) with configurable env overrides.
examples/cispo_trainer/README.md	Documents the new Ascend launcher entry point and links to external validation artifacts.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[fangweii1]

What does this PR do?

Adds a Qwen2.5-0.5B-Instruct CISPO training example using Megatron for actor/reference training and vLLM-Ascend for rollout on Ascend NPUs.

• Adds a canonical Megatron + vLLM-Ascend launcher with environment-variable overrides, automatic checkpoint resume, and timestamped logs.
• Connects the registered CISPO policy loss to GRPO advantages with lower/upper clipping of 10.0 / 0.2 and token-mean aggregation.
• Adds CPU coverage for clipped-ratio behavior, stop-gradient semantics, response masking, and policy-loss registration.
• Adds a concise entry point to the existing CISPO README and links to the full environment, logs, and 100-step results.

Related: verl-project/verl-ascend-recipe#17

Checklist Before Starting

• Searched relevant open PRs:

  • CISPO Ascend
  • CISPO Megatron
  • PR [algo] feat: add cispo #4508 already provides the registered CISPO policy loss; no open PR covers this Qwen2.5-0.5B, Megatron, and vLLM-Ascend training combination.

• Formatted the title as [{modules}] {type}: {description}.

Test

• pre-commit run --all-files --show-diff-on-failure --color=always: passed

• CISPO objective and registry CPU tests: 2 passed

• bash -n examples/cispo_trainer/run_qwen2_5_0_5b_megatron.sh: passed

• 100-step training:

  • reward first/last 10-step mean: 0.010156 / 0.281250
  • reward linear slope: 0.00388862/step
  • mean total throughput on 4 × Ascend 910B4: 1035.3128 token/s

Training artifacts: verl-project/verl-ascend-recipe#17 (comment)

API and Usage Example

python3 examples/data_preprocess/gsm8k.py \
    --local_save_dir /path/to/data/gsm8k

MODEL_PATH=/path/to/Qwen2.5-0.5B-Instruct \
DATA_ROOT=/path/to/data \
NPUS_PER_NODE=4 \
TOTAL_TRAINING_STEPS=100 \
bash examples/cispo_trainer/run_qwen2_5_0_5b_megatron.sh

Additional Hydra overrides can be appended to the command.

Design & Code Changes

• The existing cispo policy loss remains the single implementation of the objective.
• The Megatron actor calls the policy loss through the common PPO loss path; actor/reference training uses TP=2 and BF16.
• vLLM-Ascend rollout uses TP=2, four responses per prompt, chunked prefill, prefix caching, and bucketed weight synchronization.
• The launcher exposes model, data, parallelism, batch size, checkpoint, and logging settings through environment variables.
• The existing CISPO README only gains the new entry point and a link to complete results.

Checklist Before Submitting

• Read the Contribute Guide.
• Applied pre-commit checks.
• Updated documentation.
• Added CPU tests to the existing policy-loss test suite.
• Request CI after the PR is ready.
• Recipe submodule update: N/A; this PR does not modify the submodule.

AI assistance: OpenAI Codex was used for codebase inspection and preparation of the implementation, tests, and documentation. The submitter reviewed the changes and validation results.

请完善一下 1. 不需要提交test相关的代码 2. 提交实验数据

[RordChang]

@fangweii1 已按反馈完成修改：test 相关代码已删除，实验数据和完整脱敏训练日志链接已补充至 PR 正文。

[wucong25]

统一先提到verl-ascend-recipe：verl-ascend-recipe哈，此pr暂时关闭