[rollout] feat: re-evaluate sticky sessions against replica load

[ralovets]

What does this PR do?

GlobalRequestLoadBalancer pins all turns of a conversation to one replica for prefix-cache reuse, but never re-evaluates that choice. Long conversations can therefore remain concentrated on a few replicas while others go idle.

This PR adds an opt-in affinity_break_margin. On a sticky hit, the request stays pinned while its replica is within margin in-flight requests of the least-loaded replica; otherwise, the conversation is re-pinned to the least-loaded replica. The default is null, so existing behavior is unchanged. The option is ignored under full_determinism to preserve deterministic routing.

PR #6533 addresses the same problem with a global dual-threshold imbalance gate. This PR instead evaluates each sticky session against a per-session tolerance band, allowing sessions already near the minimum to remain pinned. In #6533, aggressive thresholds caused frequent migrations and regressed its long-context workload, while conservative settings were near baseline. Here, margin=2 improved all paired workloads below. The workloads are not directly comparable, but together the results suggest that migration selectivity is important.

Checklist Before Starting

• Searched for similar PRs: https://github.com/verl-project/verl/pulls?q=is%3Apr+GlobalRequestLoadBalancer
• PR title follows the [rollout] feat: ... format.

Test

CPU unit tests:

python -m pytest tests/workers/rollout/test_sticky_affinity_on_cpu.py -q
20 passed in 5.49s

The 20 tests cover unchanged default behavior, margin boundaries, re-pinning, counters, invalid values, removed replicas, and full_determinism.

End-to-end paired A/B tests used Qwen2.5-32B on DAPO-Math-17k, 8 H200 nodes per arm, 16 vLLM replicas, a frozen policy, and affinity_break_margin=2. The result is the paired median change in timing_s/generate_async.

approximate context	baseline prefix hit rate	paired steps	result
2.2K	8.7%	39	-3.8%
8.7K	5.3%	9	-8.6%
19.8K	1.6%	17	-10.1%
8.7K, large KV pool	99.7%	39	-5.4%

All four results have p <= 0.004. With the large KV pool, prefix reuse remained high (99.7% to 97.6%) while generation improved by 5.4%. Replica starvation and queue skew also decreased. These tests do not cover real tool-calling latency, instruct-tuned models, or contexts beyond approximately 20K tokens.

API and Usage Example

actor_rollout_ref:
  rollout:
    # null (default): always keep affinity
    # 0: move unless already on a least-loaded replica
    # positive values: tolerate bounded load skew
    affinity_break_margin: 2

get_status() exposes affinity_kept and affinity_broken counters.

Design & Code Changes

• Add the sticky-affinity load check and re-pinning logic to GlobalRequestLoadBalancer.
• Add RolloutConfig.affinity_break_margin and generated trainer configuration entries.
• Add CPU-only unit coverage for existing and bounded-affinity behavior.

Checklist Before Submitting

• Read the Contribute Guide.
• Full pre-commit suite passes.
• Add or update documentation.
• Added CPU unit tests.
• Request CI in the ci-request channel.

AI assistance was used for the implementation, benchmark harness, and PR editing. Every changed line was reviewed by me, and I ran the tests and A/B experiments reported above.

[CLAassistant]

All committers have signed the CLA.