Cooperative Multi-LLM Reinforcement Learning (CoMLRL) is an open-source library for training multiple LLMs to collaborate using Multi-Agent Reinforcement Learning (MARL). It provides implementations of various MARL algorithms for LLM collaboration and support for different environments and benchmarks.

Installation and first run

Install a PyTorch build compatible with your machine, then run an example directly with Python. No job scheduler or W&B account is required.

git clone https://github.com/OpenMLRL/CoMLRL.git
cd CoMLRL
pip install -e .
python examples/story-len-ratio.py

Memory requirements depend on the model, number of agents, batch size, and response length. The examples leave W&B logging disabled by default; enable it explicitly (add --wandb-project or --wandb-entity to choose where runs are logged):

python examples/story-len-ratio.py --wandb

For use in your own Python programs, CoMLRL is also available via PyPI or conda-forge:

pip install comlrl
# Or:
conda install -c conda-forge comlrl

Features

• Cooperative trainers for LLM collaboration:

  • Multi-Agent REINFORCE: Critic-free policy gradient methods, including MAREINFORCE, MAGRPO, MARLOO, MAREMAX.
    • Aligned individual response joint with joint_mode='aligned'.
    • Cartesian-product joint responses with joint_mode='cross'.
  • Multi-Agent Actor-Critic: Actor-Critic methods, including IAC and MAAC.
    • Independent actor-critic (separate critic or value-head over LLM backbone).
    • Centralized critic over joint prompts with separate actors.
  • Multi-Agent Preference Learning: MADPO, MARLHF, and their iterative variants.
    • Direct joint preference optimization or learned joint rewards followed by RL.
    • Decentralized actors or a centralized actor generating a complete joint response.

• Environments that simulate real-world tasks for training and evaluating LLM collaboration:

  • Writing: Multiple LLM agents collaborate on processing articles.
    • TLDR - Summarizing Reddit posts.
    • ArXiv - Expanding abstracts into introductions.
  • Coding: Generate code solutions for programming problems.
    • MBPP - Mostly basic python problems.
    • HumanEval - Handwritten evaluation problems.
    • CoopHumanEval - HumanEval with cooperative nature.
    • ClassEval - Complete class-level code based on attributes and docstrings.
  • Minecraft: Collaborative building tasks in Minecraft.
    • StrBuild - Building structures based on string blueprints.
    • HouseBuild - Constructing houses from given blueprints while defending against spider attacks.
  • Travel: Collaborative travel planning under user constraints.
    • TravelPlanner - Planning multi-day trip itineraries.
  • BFCL v4: Collaborative function calling with tools.
    • Native Parallel - Making parallel function calls.

Usage

Quick start by training two Qwen-2.5 agents on Reddit prompts with MAGRPO. This toy reward encourages a 3:1 output-length ratio; it does not measure summary quality:

from datasets import load_dataset
from transformers import AutoTokenizer
from comlrl.trainers.reinforce import MAGRPOConfig, MAGRPOTrainer

# Load dataset and tokenizer
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen2.5-0.5B")
dataset = load_dataset("trl-lib/tldr", split="train").select(range(128))

# Initialize trainer and start training
trainer = MAGRPOTrainer(
    agent_model="Qwen/Qwen2.5-0.5B",
    num_agents=2,
    tokenizer=tokenizer,
    train_dataset=dataset,
    reward_func=lambda a, b: [
        -abs(len(second) / max(len(first), 1) - 3.0)
        for first, second in zip(a, b)
    ],
    formatters=[lambda example: example["prompt"]] * 2,
    args=MAGRPOConfig(num_turns=1),
)
trainer.train()

Contributing

We thank the gracious help of all contributors:

Shuo Liu 🤔 🚧 💻 📖

Tianle Chen 🚧 💻 🐛

Ryan Amiri 🚧 💻 🐛

Zeyu Liang 📖 🐛

_{🤔: Foundational Ideas; 🚧: Maintenance; 💻: Code; 📖: Documentation; 🐛: Bug Report.}

For new contributors, please see contributing guidelines on setting up a dev environment.

Sponsorship

CoMLRL was developed using substantial computational resources. Its growth has been made possible by the generous support of the following organizations and institutions.

We welcome computational sponsorship to support the continued development of CoMLRL. If you are interested in supporting this project, please contact us.

Citation

Please cite the following papers if you find this library useful in your research:

@inproceedings{liu2026llm,
  title={LLM Collaboration with Multi-Agent Reinforcement Learning},
  author={Liu, Shuo and Liang, Zeyu and Lyu, Xueguang and Amato, Christopher},
  booktitle={Proceedings of the AAAI Conference on Artificial Intelligence},
  volume={40},
  number={38},
  pages={32150--32158},
  year={2026}
}

@inproceedings{liu2026learning,
  title   = {Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic},
  author  = {Liu, Shuo and Chen, Tianle and Amiri, Ryan and Amato, Christopher},
  booktitle={Proceedings of the International Conference on Machine Learning.},
  year    = {2026}
}