Train from inline multimodal rollouts

[eligotts]

Summary

• retain inline image URLs through rollout collection, truncation, packing, and transport instead of shipping preprocessed pixel tensors
• return vLLM's expanded prompt IDs for exact logical/expanded token accounting
• materialize multimodal inputs lazily in the trainer through a small Qwen-VL / Kimi K2.5 model-family adapter registry
• split multimodal packing bins across trainer workers and leave the existing multimodal SFT path unchanged
• pin Renderers PR length rewards #133 at bd7c91e and Verifiers PR fix(ring_attn): make FA3 causal/window_size kwargs version-agnostic #2417 at 6720736e1

vLLM 0.28 alignment

Prime now delegates content-parts parsing, raw-media resolution, multimodal engine-input construction, DP routing, max-token defaulting, and usage reporting to the vLLM 0.28 implementation on main. Its serving subclass retains only expanded prompt IDs pending vLLM #53187, compact routed-expert responses, and the KV-transfer bridge pending a release containing vLLM #42644.

Cache-salt forwarding through content-parts remains a small upstream vLLM follow-up; Prime no longer carries a copied endpoint implementation for it.

Stack

• Renderers: Add raw multimodal content parts support renderers#133
• Verifiers: Keep multimodal content inline in v1 rollouts verifiers#2417
• vLLM content-parts support: [Frontend] Add content_parts to /inference/v1/generate for raw multim… vllm-project/vllm#51478
• vLLM prompt metadata: [Frontend] Return prompt metadata from /inference/v1/generate vllm-project/vllm#53187

Validation

• uv lock --check
• uv run pytest -q tests/unit/orchestrator/test_algorithms.py tests/unit/orchestrator/test_batch.py tests/unit/orchestrator/test_qwen3_vl_e2e.py (50 passed)
• targeted Ruff check passed

---

Note

Medium Risk
Changes the multimodal training data path (transport, packing, lazy image decode, and model-family forward policy) and inference response shape; misaligned placeholder lengths or refs would fail at materialize/forward time rather than in the old encoded-tensor path.

Overview
Multimodal RL no longer ships preprocessed pixel tensors over the batch transport. Samples carry mm_refs (base64 image URLs plus offset/length in the vLLM-expanded prompt) with mm_token_type_ids, built in the orchestrator from inline image_url parts and image-token runs. Truncation and packing keep whole placeholder blocks and remap refs when sequences are concatenated, without requiring matching mm_kwargs shapes to co-pack.

The trainer materializes vision inputs at forward time via a small prime_rl.multimodal registry (Qwen-VL / Kimi K2.5 adapters) and materialize_mm_refs, with ForwardPolicy driving position IDs and context-parallel deferral. forward() accepts mm_forward_policy while the existing SFT eager-processor path stays on kwargs-only behavior.

vLLM PrimeRlServingTokens now surfaces prompt_token_ids from the engine stream (expanded prompt after multimodal placeholders) alongside the existing routed-experts and KV-transfer bridges. Tests and the Qwen3-VL e2e check were updated for raw content_parts / expanded prompt IDs instead of a features payload round-trip.

^{Reviewed by Cursor Bugbot for commit 577123e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ab4ca01. Configure here.}

[cursor]

Missing KL mismatch validation table for new models

Medium Severity

This PR introduces new custom multimodal adapters (QwenVLAdapter for qwen3_vl/qwen3_vl_moe/qwen3_5/qwen3_5_moe and KimiK25Adapter for kimi_k25) with distinct ForwardPolicy configurations that change how position_ids, mm_token_type_ids, and context parallelism behave during the model forward pass. Per project rules, any PR introducing a new custom model must include a table showing mean KL mismatch across 20 steps on a math environment with batch_size=64, with all entries below 0.015. No such table is present in the PR description.

Additional Locations (2)

• src/prime_rl/multimodal/qwen_vl.py#L9-L16
• src/prime_rl/multimodal/kimi_k25.py#L9-L12

 

^{Triggered by project rule: BugBot Instructions}

^{Reviewed by Cursor Bugbot for commit ab4ca01. Configure here.}