Train from inline multimodal rollouts - #3320
Conversation
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
# Conflicts: # src/prime_rl/inference/vllm/serving_tokens.py # tests/unit/inference/test_serving_tokens.py
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit ab4ca01. Configure here.
| try: | ||
| return _BY_MODEL_TYPE[model_type] | ||
| except KeyError as exc: | ||
| raise NotImplementedError(f"Raw image training is not implemented for model type {model_type!r}") from exc |
There was a problem hiding this comment.
Missing KL mismatch validation table for new models
Medium Severity
This PR introduces new custom multimodal adapters (QwenVLAdapter for qwen3_vl/qwen3_vl_moe/qwen3_5/qwen3_5_moe and KimiK25Adapter for kimi_k25) with distinct ForwardPolicy configurations that change how position_ids, mm_token_type_ids, and context parallelism behave during the model forward pass. Per project rules, any PR introducing a new custom model must include a table showing mean KL mismatch across 20 steps on a math environment with batch_size=64, with all entries below 0.015. No such table is present in the PR description.
Additional Locations (2)
Triggered by project rule: BugBot Instructions
Reviewed by Cursor Bugbot for commit ab4ca01. Configure here.
1 participant
Summary
d1bb692and Verifiers PR fix(ring_attn): make FA3 causal/window_size kwargs version-agnostic #2417 at0c5a08b41vLLM 0.28 alignment
Prime now delegates content-parts parsing, raw-media resolution, multimodal engine-input construction, DP routing, max-token defaulting, and usage reporting to the vLLM 0.28 implementation on
main. Its serving subclass retains expanded prompt IDs and authoritative multimodal placeholder ranges pending vLLM #53187, compact routed-expert responses, and the KV-transfer bridge pending a release containing vLLM #42644.Cache-salt forwarding through content-parts remains a small upstream vLLM follow-up; Prime no longer carries a copied endpoint implementation for it.
Stack
Validation
uv lock --checkuv run pytest -q tests/unit/orchestrator/test_algorithms.py tests/unit/orchestrator/test_batch.py tests/unit/orchestrator/test_qwen3_vl_e2e.py(50 passed)Note
Medium Risk
Touches the full multimodal RL data path (transport, packing, inference response shape, and train-time image decode/processor alignment); misaligned placeholder lengths fail hard at materialize time.
Overview
Multimodal RL no longer ships preprocessed pixel tensors from orchestrator to trainer. Training samples and micro-batches carry
MMRefs(base64 image URLs plus offset/length for vLLM-expanded placeholder runs) andmm_token_type_ids, with pixels built only on the trainer via a newprime_rl.multimodaladapter registry (Qwen-VL / Kimi K2.5) andmaterialize_mm_refs.The orchestrator
trace_to_samplespath scrapes inlineimage_urlparts from branch messages and pairs them with image-token runs instead of concatenating renderermm_kwargs. Batch truncation, validation, and packing are updated for refs (whole-image cuts, relaxed MM packing, ref offset shifting when bins merge);EncodedTensoris removed from transport types.vLLM
PrimeRlServingTokensaddsprompt_token_idsandmm_placeholderson generate responses (plus prompt-id capture) so rollouts can align logical vs expanded prompts.forward()gains optionalmm_forward_policyfor per-family behavior (e.g. Qwen MRoPE / context-parallel deferral). Tests and the Qwen3-VL e2e check now expect rawcontent_partsand expanded prompt metadata instead of afeaturespayload.Reviewed by Cursor Bugbot for commit af60d8a. Bugbot is set up for automated code reviews on this repo. Configure here.
Authoritative multimodal ranges
mm_placeholdersfrom the engine input and return them besideprompt_token_ids.uv lock --check; 52 focused Prime tests passed against vLLM 0.28.0; Ruff and pre-push hooks passed.