feat(ppo): add actor loss aggregation modes

[EazyReal]

Description

Stacked on #1416. The first two commits are that PR; please review the loss-aggregation commits. After #1416 lands, this rebases down to those commits.
See #1443 for alternative implementation with bigger blast ratio but more principled approach.

AReaL currently aggregates the actor policy-gradient loss as a global token mean. That makes token count the unit of weighting: longer responses, and prompt groups with more valid tokens, contribute more to the update. Some PPO/GRPO recipes instead want the update to average over responses, over prompt groups, or against a fixed response-length denominator, so the averaging unit needs to be explicit rather than hard-coded.

This PR adds actor.loss_aggregation for the actor pg_loss. For one train step, let B be the number of responses, G = gconfig.n_samples, P = B / G, M_i be the valid-token count for response i, and L = actor.loss_aggregation_divisor.

mode	step scalar	behavior
token_mean	sum_i,t loss_it * mask_it / sum_i M_i	Keeps the current behavior and remains the default.
seq_mean	(1 / B) * sum_i token_mean(response_i)	Gives each response one unit of weight.
prompt_mean	(1 / P) * sum_g token_mean(prompt_group_g)	Gives each prompt group of G responses one unit of weight.
constant	(1 / B) * sum_i (sum_t loss_it * mask_it / L)	Uses a positive fixed denominator per response for DrGRPO-style length-independent scaling.

The implementation fits the existing engine contract instead of adding a second reduction path. aggregate_pg_loss computes the local scalar for the selected unit, and _make_loss_weight_fn returns the matching unit count. The engine already reduces microbatches as sum(local_loss * local_weight) / sum(local_weight), so the global result is exact across packing and distributed workers as long as each mode reports the right weight.

Compared with the other OSS implementations checked for this change, this is the AReaL-shaped version of the same scalar contract: verl centralizes aggregation in agg_loss with global batch metadata; SkyRL pre-scales advantages before the worker sums policy losses; prime-rl stays intentionally narrow with global token mean; and slime's counterpart needs a CP-aware reducer because CP shards the token numerator while sample/prompt denominators are whole-step data. In AReaL, the existing train-engine local scalar + loss weight contract already carries the required numerator/denominator pair, so the PR keeps the change local to actor loss aggregation and loss-weight selection.

For prompt_mean, the group size is derived from the existing rollout configuration rather than exposed as another knob. The PR raises microbatch granularity and rollout.min_valid_group_size to that group size so prompt groups stay whole when they are averaged. For constant, actor.loss_aggregation_divisor must be positive and finite, and it is rejected for every other mode. CISPO remains limited to token_mean because the current CISPO implementation is token-level.

Before this PR, users could only get token-weighted actor loss without patching the loss function. After it, the configured mode directly names the unit being averaged, while the default training objective is unchanged.

The focused tests cover per-mode values, packed and padded parity, denom_mask, the loss/weight pairing invariant used by the engine reduction, config derivations for prompt groups, constant-divisor validation, and CISPO rejection outside token_mean. The CLI reference is regenerated for both English and Chinese; the Chinese page has translated descriptions for the new loss-aggregation options.

Tested with:

• uv run pytest -q tests/test_prompt_mean_loss.py tests/test_cispo_loss.py
• uv run ruff check areal/api/cli_args.py areal/trainer/ppo/actor.py areal/utils/functional/functional.py areal/utils/functional/__init__.py tests/test_prompt_mean_loss.py tests/test_cispo_loss.py
• uv run ruff format --check areal/api/cli_args.py areal/trainer/ppo/actor.py areal/utils/functional/functional.py areal/utils/functional/__init__.py tests/test_prompt_mean_loss.py tests/test_cispo_loss.py
• uv run python docs/generate_cli_docs.py
• uv run pre-commit run --all-files

Related Issue

Closes #1423. Stacked on #1416.

Type of Change

• ✨ New feature

Checklist

• I have read the Contributing Guide
• Pre-commit hooks pass (pre-commit run --all-files)
• Relevant tests pass; new tests added for new functionality
• Documentation updated (CLI reference regenerated)
• Branch is up to date with main
• Self-reviewed via /review-pr command
• This PR was created by a coding agent via /create-pr
• This PR is a breaking change

[gemini-code-assist]

Code Review

This pull request introduces support for different loss aggregation levels (token, sequence, and prompt mean) to reduce per-token policy-gradient loss, alongside a min_valid_group_size parameter to handle under-filled rollout groups. The normalization, remote inference engine, and PPO actor loss calculations are updated accordingly, supported by new tests and documentation. The review feedback suggests addressing a potential device mismatch in the packed loss aggregation path, adding a validation check to ensure min_valid_group_size does not exceed n_samples, and translating the newly added parameter descriptions in the Chinese CLI reference.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[EazyReal]

Superseded by #1443. The pre-superseded #1417 head was e3b97045ee4a259f53a051b5643355b21907f01b; the replacement implementation is #1443 at 43294f4a7e3d6e09fe510c0bcf4c33eaca4bc53a. Closing the older implementation so review can focus on the explicit loss-reduction contract.

[EazyReal]

Final replacement: #1546. It keeps the strongest part of this conservative design, the existing loss_fn plus loss_weight_fn engine seam, while replacing positional fixed-size prompt grouping with explicit ragged trajectory-group metadata and correcting filtered-unit denominator behavior. #1443 is also closed, leaving #1546 as the sole public review target. The broader engine abstraction is archived separately at EazyReal#4.