[training_utils, env, doc] feat: use Liger fused linear PPO kernel

[kolehma8]

What does this PR do?

Closes #7424.

Replaces verl's experimental fused linear PPO output-head implementation with Liger Kernel v0.8.2's LigerFusedLinearScaledCrossEntropyFunction when Liger is installed. The wrapper preserves log-probability sign, entropy output, 2D/3D shapes, and gradients; without Liger it keeps the existing chunked FusedLinearForPPOFunction unchanged.

This does not duplicate an existing PR: searches for issue-linked and fused-linear-PPO work found no open implementation PR.

AI assistance: GitHub Copilot CLI assisted with implementation, testing, benchmark automation, and PR drafting. The human submitter reviewed every changed line and supervised all validation.

Checklist Before Starting

• Search for similar PRs. Queries: #7424 in open PR bodies and fused linear PPO Liger.
• Format the PR title as [{modules}] {type}: {description}.

Test

• python -m pytest -q tests/utils/test_experimental_torch_functional_on_cpu.py — 4 passed.
• python -m pre_commit run --all-files --show-diff-on-failure --color=always — all hooks passed.
• uv lock --check --python 3.12 — 337 packages resolved; lock is current.
• H100 training benchmark — Qwen3-0.6B on GSM8K, GRPO/FSDP, 4 GPUs per run, 4 seeds per method (42, 101, 303, 404), 100 steps each, BF16 compute, dynamic batching and remove-padding packing, 48 trajectories/GPU (approximately 13–16K packed tokens/GPU), 32K token cap, and steps 11–100 used for timing statistics.

Four-seed actor update performance

Method	Seeds	Mean ms/token ± SD	Paired Liger-minus-Verl 95% CI	Liger reduction
Existing verl chunked fused PPO	4	0.02803 ± 0.00268	—	—
Liger CuTe DSL	4	0.02424 ± 0.00294	[-0.00434, -0.00325]	13.53%

Liger reduced actor update time/token on every seed.

Four-seed peak memory

Method	Actor max allocated (GiB) ± SD	Mean per-GPU peak used (GiB) ± SD
Existing verl chunked fused PPO	35.68 ± 0.06	39.63 ± 0.21
Liger CuTe DSL	33.70 ± 0.10	37.80 ± 0.11

Liger reduced actor allocated memory by 5.56% and physical peak memory by 4.64% versus the existing verl fused baseline.

Four-seed reward behavior

Method	Final-10-step reward mean ± SD
Existing verl chunked fused PPO	0.652 ± 0.014
Liger CuTe DSL	0.638 ± 0.022

The paired Liger-minus-Verl final-reward difference was -0.0141 with a 95% CI of [-0.0494, 0.0213]. The interval crosses zero, so these four seeds do not resolve a reward regression; both methods converge to the same reward range.

Benchmark Figures

Figures are attached below and are intentionally not committed to the repository.

Actor update time/token across seeds

GSM8K reward across seeds

Peak GPU memory across seeds

API and Usage Example

No configuration change is required for existing fused-kernel users:

actor_rollout_ref:
  model:
    use_fused_kernels: true
    fused_kernel_options:
      impl_backend: torch

With liger-kernel>=0.8.2, the torch backend uses Liger fused scaled linear cross entropy. If Liger is absent, it uses verl's existing chunked fallback.

Design & Code Changes

• Import Liger's public fused scaled linear cross-entropy operator optionally and surface broken/incompatible Liger installations instead of silently masking import failures.
• Flatten supported 3D hidden states and labels for Liger's 2D operator, negate NLL back to log-probabilities, and restore the original output shape.
• Preserve the original chunked custom autograd implementation and chunk_size behavior as the no-Liger fallback.
• Require liger-kernel>=0.8.2 in setup, development, FSDP, and Megatron dependency manifests; regenerate uv.lock.
• Add CPU tests for forward/backward parity, Liger dispatch arguments, shape restoration, label casting, and fallback chunking.
• Update installation, configuration, and performance documentation.

Checklist Before Submitting

• Read the Contribute Guide.
• Apply pre-commit checks: pre-commit run --all-files --show-diff-on-failure --color=always.
• Add / Update the documentation.
• Add unit or end-to-end test(s) to the CI workflow to cover all the code. The new tests/utils/test_experimental_torch_functional_on_cpu.py follows the CPU CI naming convention.
• Once the PR is ready for CI, send a message in the ci-request channel in the verl Slack workspace.
• If the PR is related to the recipe submodule, update its reference. Not applicable; this PR does not change recipe.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

jkolehma seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}