Skip to content

About

A curated, annotated list of agents that act in the physical world: robot harnesses for coding agents, robot models, world models, benchmarks, simulators and datasets. Maintained by BenchFlow.

Resources

Contributing

Stars

3 stars

Watchers

0 watching

Forks

Latest commit

 

History

8 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Awesome Embodied Agents Awesome

A curated, annotated list of agents that act in the physical world, with an emphasis on coding agents and LLMs as robot policies ("agent as a policy"): the harnesses that connect them to robots, the models and world models they run on, the benchmarks that score them, and the simulators, assets, perception tools and datasets underneath.

Maintained by BenchFlow · Join our Discord · Sister list: Awesome Agent Evals

Most lists in this area are link dumps. This one follows the rules of Awesome Agent Evals: every entry says what the thing is and why it belongs, URLs are checked, numbers are quoted verbatim from the linked source, dates are given, and dead projects are pruned or flagged. Robot results add two questions a reader needs answered: who judged success (the simulator's state, a held-out checker, a human, or the agent itself) and how many trials ran. Licences that restrict use are flagged too, because robot work often inherits non-commercial scenes, assets and weights.

"Agent as a policy" here means the agent decides what the robot does next, by calling robot tools, writing and running control code, or calling a learned controller, and is scored on what happens in the world rather than on a plan written in text.

212 annotated entries, plus a starter set and a map of companies; every URL was checked on 2026-10-21. Markers: 🆕 = released, or given a major new version, in 2025–2026 · ⚠️ = caveat. Disclosure: BenchFlow maintains this list and builds Robo Use, which is listed and held to the same bar. Contributions welcome, see CONTRIBUTING; a daily scan proposes additions as pull requests that a maintainer must approve.

Contents


⭐ Start here

  1. Robot-Use Agents — Phillip Isola (MIT) · essay, 2026-09-07 — The case for general-purpose agents that use robots the way computer-use agents use computers: "Think of it as Claude acting as the puppeteer of a robot body."
  2. Code as Policies — Liang et al. (Google) · paper, 2022-09-16 — Language models writing robot policy code over perception and control APIs: the idea most coding-agent harnesses below build on.
  3. Claude plays robotics — Anthropic · study, 2026-07-09 — Twelve models at four levels of control, from raw actions to supervising RL. Direct control mostly fails, which is why harness design matters.
  4. CaP-X — NVIDIA, UC Berkeley, Stanford, CMU · benchmark, 2026-03-23 — The code-as-policy benchmark for coding agents in manipulation, with a training recipe on top.
  5. Gemini 4 Argon for Robotics — Google · blog + API, 2026-10-13 — A frontier model used as a robot agent end to end: self-improvement on a real arm, a simulated twin from one photo, and a curriculum that fine-tunes a VLA.
  6. Can Jev Nav? — Dimensional · benchmark, 2026 — Real-time navigation scored by SPL and cost per run, comparing coding agents, a typed "System One" model and a skill-level agent.
  7. An Unexpected Robot Policy — RoboProbe and RoboDojo teams · paper, 2026-09-21 — A frontier LLM ranked above every public policy on RoboDojo in simulation, and a real-robot test halted for unsafe actions.
  8. Refuses in Chat, Complies with Arms — Robocurve and UCLA · paper, 2026-10-03 — Why text-only safety evaluations do not transfer: refusal drops once the model controls a robot.

1 · Robot harnesses for coding agents

Software that connects a coding agent (Claude Code, Codex and others) or another LLM agent to a robot or a simulator: what the agent sees, which actions it may call, and who checks the result.

1a · Harnesses and robot operating systems

  • Robo Use — BenchFlow — https://robouse.ai · code https://github.com/benchflow-ai/robouse · PyPI https://pypi.org/project/robouse/ · docs https://robouse.ai/docs/ · suites https://robouse.ai/docs/suites/ · harness + benchmark, 0.6.0 2026-10-21 — "Agent as a policy for embodied agents": it turns robot benchmarks into tasks that a harness such as Claude Code or Codex solves by driving a simulated robot from its shell with the robo command, and "An episode server runs the simulator, enforces the step budget, records a video and scores the final physical state", so the agent never grades itself. "robouse 0.6.0 bundles 790 tasks in 34 suites" (LIBERO, LIBERO-PRO, robosuite, Meta-World, ManiSkill3, RoboCasa, MyoSuite, dm_control and more); VLA baselines run behind the same server, and trial pages are public at robouse.ai/runs. 0.5.0 added a harness for Dream Labs' Dream-1 (§3a); 0.6.0 adds harnesses for NVIDIA's ASPIRE workflow (§1c), Kinex (§1a), TypeSafe's Jev (§2a) and Gemini CLI. Apache-2.0. 🆕 ⚠️ Built by BenchFlow, which maintains this list.
  • Kinex — Georgia Tech and Harvard contributors — https://stacx-lab.github.io/kinex/ · code https://github.com/stacx-lab/kinex · harness, v1.0.0 2026-10-15 — "A general harness that connects coding agents to any physical robot", built on the Pi coding-agent SDK with three layers (devices, tools, skills) and ROS 2 support. Evaluated zero-shot on 132 LIBERO, RoboTwin and RoboDojo tasks with each suite's native success criteria: "Kinex raises success from 72.7% to 78.0% at 42.9% lower cost than Codex + Bare action API." Episodes and agent traces are on Hugging Face (§7). Robo Use (§1a) added a harness for it in 0.6.0. Apache-2.0. 🆕 ⚠️ Released 2026-10-15, so few outside users yet; results are the authors' own.
  • DimOS — Dimensional — https://github.com/dimensionalOS/dimos · evals docs https://docs.dimensional.org/usage/evals/ · robot OS, 2024-10-19 — "The Agentive Operating System for Physical Space": a Python robot OS (no ROS required) for quadrupeds, humanoids, arms and drones whose skills agents reach through a CLI and an MCP server; "The dimos CLI manages the full lifecycle — run blueprints, inspect state, interact with agents, and call skills via MCP." It ships its own agent-evaluation harness, dimos evals, and powers Can Jev Nav? (§4a). Apache-2.0. 🆕 ⚠️ "Pre-Release Beta".
  • Inspect Robots — Robocurve — https://github.com/robocurve/inspect-robots · paper https://arxiv.org/abs/2610.06306 · docs https://docs.inspectrobots.org/ · eval harness, paper 2026-10-05 — "Evaluating the Capabilities and Safety of Embodied AI": an Inspect-style framework where any policy, an LLM agent or a VLA, runs on real arms or humanoids (YAM, Franka, SO-101, G1 and others) or in simulators, with logged transcripts and grader scores; StationeryBench and RoboHarm are built on it. "Inspect Robots has seen significant early uptake, receiving nearly 100,000 downloads in the three months since its release." MIT. 🆕 ⚠️ Alpha; the download count is the authors' own.
  • RPent — RLinf; paper from Tsinghua University and others — https://github.com/RLinf/RPent · paper https://arxiv.org/abs/2607.08448 · harness, paper 2026-07-09 — "Agentic Infrastructure for the Physical World": Claude Code, Codex or an API model plans, and frozen VLAs such as π0.5 become retryable primitives guided by memory (the paper calls it Harness VLA). Its leaderboard covers LIBERO, LIBERO-PRO, RoboCasa365 and RoboTwin; with Codex and GPT-6 Astra it reports "92.63% Overall (741/800) across all eight LIBERO-PRO suites". Apache-2.0. 🆕 ⚠️ Leaderboard results are self-reported.
  • PyRUA-Lean — Peking University, NUS, NVIDIA and others — https://github.com/DAGroup-PKU/PyRUA-Lean · paper https://arxiv.org/abs/2610.01939 · harness, paper 2026-10-01 — The robot is a Python object, robo, and the agent works through one python(code) tool that composes primitives (VLAs included) instead of making one tool call per action; "it raises success from 63.1% to 71.7% over tool calling with the same GPT-6 Astra planner and primitives". Apache-2.0. 🆕 ⚠️ Simulation only so far.
  • Show-Harness — NUS Show Lab — https://github.com/showlab/Show-Harness · paper https://arxiv.org/abs/2609.10522 · harness, paper 2026-09-09 — "Just a VLM Agent Can Play Robots": VLMs reason over discrete semantic action units that per-robot interpreters turn into motion, so frontier VLMs run zero-shot and small open VLMs can be adapted "with just a few GPU-hours of fine-tuning"; LoRA adapters and data are released. Apache-2.0. 🆕
  • RoboHarness — Zhejiang University — https://github.com/BinceQu/RoboHarness · paper https://arxiv.org/abs/2610.14018 · harness, paper 2026-09-29 — A visual-geometric control panel for an LLM agent (clicked keypoints, optical-flow tracking, depth back-projection, geometric constraints), with Claude Code and Codex harnesses for BEHAVIOR Challenge 2025 tasks: "With RoboHarness, Qwen3.8-Flash-Next outperforms state-of-the-art VLA models on BEHAVIOR Challenge 2025 tasks." MIT. 🆕 ⚠️ BEHAVIOR data needs its own non-commercial licence (§4b).
  • RobotUse — KAIST and Seoul National University — https://github.com/robotuse-team/RobotUse · paper https://arxiv.org/abs/2610.04929 · harness, paper 2026-10-04 — Splits the work: the agent picks targets and poses visually while the backend handles geometry, motion planning and control, with subagents per subgoal and a persistent playbook of lessons; "On RoboLab, RobotUse achieves 45% task success, outperforming CaP-X by 6.7 percentage points". Apache-2.0. 🆕 ⚠️ The released code runs only in RoboLab (Isaac Sim); real-robot results are in the paper only.
  • OpenRUA — UCL and CMU — https://github.com/terminalworld/OpenRUA · paper https://arxiv.org/abs/2610.02459 · harness, paper 2026-10-01 — "Robot-Use Agents Are Zero-Shot Visuomotor Policies": an off-the-shelf Claude Code or Codex gets only terminal access to the robot's native ROS 2 interface and its documentation, with no custom primitives and no VLA; "With Claude Code powered by Claude Opus 5, OpenRUA achieves success rates of 99.0% on CaP-Bench and 87.0% on LIBERO-PRO". Apache-2.0. 🆕 ⚠️ Simulated benchmarks; few outside users yet.
  • PhyAgentOS — Sun Yat-sen University, Peng Cheng Laboratory, X-Era Lab — https://github.com/PhyAgentOS/PhyAgentOS-core · paper https://arxiv.org/abs/2607.16636 · robot agent OS, paper 2026-07-18 — A self-evolving operating system for embodied agents that separates cognitive planning from physical execution, passes state between agent and runtime as files, and adds a session verifier, memory of past runs, layered safety and robot integrations packaged as skills; "validated on 19+ simulated and physical embodiments". MIT. 🆕
  • ROSClaw — Tongji University and SRIAS — https://github.com/ros-claw/rosclaw · runtime, 2026-03-16 — A governed runtime between agents (Codex, Claude Code, OpenClaw, VLAs) and bodies (ROS 2, MCP, simulators, vendor SDKs) that checks policy and authorisation before each action and writes an execution receipt for it. MIT. 🆕 ⚠️ "ROSClaw is alpha software"; its maturity table marks real hardware as experimental.
  • Strands Robots — AWS Strands Labs — https://github.com/strands-labs/robots · agent tools, v0.5.3 2026-10-07 — "Control, simulate, and train robots with natural language": a Robot() tool for Strands Agents that returns a MuJoCo simulation or a real robot and runs LeRobot, GR00T and Cosmos policies. Apache-2.0. 🆕 ⚠️ Strands Labs is AWS's organisation for experimental projects.
  • AgenticROS — AgenticROS project — https://github.com/agenticros/agenticros · MCP server + plugin, 2026-03-01 — An agent-agnostic layer for ROS 2 robots: an OpenClaw plugin and one MCP server for Claude Code, Codex CLI, Hermes, Antigravity and Gemini CLI. NVIDIA's Isaac ROS 5.0 post (2026-09-22) points to it: "AgenticROS, an open source project sponsored by 3D perception technology company RealSense, connects Isaac ROS". Apache-2.0. 🆕
  • RAI — Robotec.ai — https://github.com/RobotecAI/rai · paper https://arxiv.org/abs/2505.07532 · agent framework, paper 2025-05-12 — A vendor-agnostic multi-agent framework on ROS 2 with speech, simulation and its own benchmark: "rai_bench: Benchmarking suite for RAI. Test agents, models, tools, simulators, etc." Apache-2.0. 🆕
  • NVIDIA Agent Skills — NVIDIA — https://github.com/NVIDIA/skills · Isaac Lab skills https://isaac-sim.github.io/IsaacLab/v3.0.0-EA/source/developer-tools/agent_skills.html · skills, 2026-02-25 — Signed skills that teach coding agents robotics, simulation and CUDA work, each published skill carrying its own evaluation set; Isaac Lab 3.0 ships 17 user-facing skills, and "User-facing skills must also include an evaluations.md file with at least three representative scenarios." Apache-2.0 and CC BY 4.0. 🆕

1b · MCP servers and device standards

  • Model Hardware Standard — Anthropic, with HHMI Janelia — https://www.anthropic.com/news/model-hardware-standard-research-preview · standard preview, 2026-08-27 — "a shared specification for AI agents to safely operate physical devices": standard device drivers with discovery and read/write primitives, model-agnostic and reachable over MCP, for microscopes, liquid handlers and robot arms. 🆕 ⚠️ Gated research preview with an application form; the spec is not public, and the post describes the preview as coming "ahead of making the standard open source".
  • Qwen-MM-Plugins — Qwen — https://github.com/QwenLM/Qwen-MM-Plugins · plugins, MHS host added 2026-09-20 — Plugins that make agent harnesses (Claude Code, Codex, Qwen Code, OpenClaw) multimodal; its mhs plugin is an open MHS host with a fixed six-tool surface, host-side safety limits and an emergency stop, and "Operates real hardware — cameras, sensors, lamps, arms, lab equipment — through Model Hardware Standard adapters". Apache-2.0. 🆕 ⚠️ Device adapters come from hardware owners under the gated MHS preview.
  • ROS MCP Server — Robot MCP — https://github.com/robotmcp/ros-mcp-server · MCP server, v3.1.2 2026-09-23 — "ROS-MCP-Server connects large language models (such as Claude, GPT, and Gemini) to robots": any MCP client reaches topics, services, actions and parameters on ROS 1 or ROS 2 robots through rosbridge. Apache-2.0. 🆕 ⚠️ Needs rosbridge; no evaluation of its own.

1c · Agents that write and improve robot code and skills

  • Code as Policies — Liang et al. (Google) — https://arxiv.org/abs/2209.07753 · paper, 2022-09-16 — "Language Model Programs for Embodied Control": code-writing LLMs, prompted with a few examples, write robot policy code that processes perception outputs and parameterises control primitives. The idea behind CaP-X (§4a) and most harnesses above.
  • VoxPoser — Huang et al. (Stanford) — https://arxiv.org/abs/2307.05973 · paper, 2023-07-12 — LLMs compose 3D value maps that a motion planner turns into dense 6-DoF end-effector trajectories, for open-set instructions and objects without predefined motion primitives.
  • ASPIRE — NVIDIA GEAR with UMich, UIUC, UC Berkeley, CMU — https://research.nvidia.com/labs/gear/aspire/ · paper https://arxiv.org/abs/2607.00272 · code https://github.com/NVlabs/ASPIRE · paper + code, 2026-06-30 — Agentic skill discovery: a coding agent writes and refines control programs from multimodal execution traces, saves verified fixes to a growing skill library, and uses evolutionary search over tasks and programs; ASPIRE "is a new type of continual learning" in which training means refining skills rather than gradient descent. "On 10 LIBERO-Pro tasks, success rises from 15% (GPT-6 Luna) to 99% (Claude Opus 5.5)"; it also reports robosuite and BEHAVIOR-1K results and early sim-to-real transfer. Robo Use (§1a) added a harness for its workflow in 0.6.0. Apache-2.0. 🆕 ⚠️ The authors "have not verified that smaller or weaker LLMs can sustain the debugging loop".
  • ENPIRE — NVIDIA, CMU, UC Berkeley — https://research.nvidia.com/labs/gear/enpire/ · paper https://arxiv.org/abs/2606.19980 · code https://github.com/NVlabs/ENPIRE · paper + code, 2026-06-18 — "Agentic Robot Policy Self-Improvement in the Real World": a coding-agent harness that resets and verifies the environment, improves the policy, runs rollouts on one or several robots and evolves its approach, to "autonomously train a policy to achieve a 99% success rate on challenging, dexterous manipulation tasks" (CoRL 2026). Apache-2.0. 🆕 ⚠️ Real-robot use needs a calibrated station.
  • SimEX — Amazon FAR, UT Austin, CMU — https://robo-simex.github.io/ · paper https://arxiv.org/abs/2609.38982 · paper, 2026-09-30 — "Simulation-Integrated Robotics AutoResearch": a coding agent builds a robot toolbox through probe-and-optimise cycles in simulation, then adapts toolbox and simulator together from a few real trials, "without any demonstration and with only 10 minutes of real-robot interaction". 🆕 ⚠️ No code released.
  • SEAR — Luma AI — https://sear.bot/ · paper https://arxiv.org/abs/2610.18792 · paper + benchmark, 2026-10-13 — "Self-Evolving Agentic Robotics": agents act as the policy, write the code, or fine-tune a VLA, with controls that separate retained experience from retries, luck and privileged simulator state, "Across seven frontier language models, 330 simulated and five physical tasks". 🆕 ⚠️ Code "coming soon": the repository holds only a README.

2 · Robot reasoning models and VLAs

2a · Embodied reasoning and frontier models for robots

  • SayCan — Ahn et al. (Google) — https://arxiv.org/abs/2204.01691 · paper, 2022-04-04 — "Do As I Can, Not As I Say: Grounding Language in Robotic Affordances": an LLM proposes steps and the value functions of pretrained skills ground them in what the robot can do in its current state.
  • Gemini 4 Argon for Robotics — Google (Kanishka Rao) — https://developers.googleblog.com/gemini-4-argon-for-robotics/ · blog + API, 2026-10-13 — Google's frontier model as a robot agent. On ERQA, "Using pure reasoning, Argon scores 87%, outperforming every other frontier model"; it improves an SO-101 pick-and-place over three iterations (58.9 s, 22.9 s, 17.4 s), and from a single RGB photo builds a calibrated MuJoCo twin, defines a 77-task curriculum and exports 4,500+ episodes to fine-tune a VLA, "boosting real-world physical robot Success Rate from 56% to 75%". Paid Gemini API tier (gemini-4-argon) and Antigravity. 🆕 ⚠️ Self-reported; the harness and tasks are not released.
  • Gemini Robotics-ER 2 — Google DeepMind — https://deepmind.google/models/gemini-robotics/embodied-reasoning/ · docs https://ai.google.dev/gemini-api/docs/robotics-overview · model + API, 2026-07-30 — An embodied-reasoning model for spatial reasoning, planning and multi-robot orchestration through function calling; "It then hands off motor execution to any given lower level vision-language-action (VLA) model." Public preview in the Gemini API (gemini-robotics-er-2-preview). 🆕 ⚠️ Closed model; benchmark numbers on the page are self-reported.
  • Glimpse — Elorian — https://elorian.ai/blog/introducing-glimpse · model + API, 2026-10-16 — "a new category of visual thinking model: a system that reasons about the physical world from what it sees", trained to call multimodal tools and coordinate subagents; robotics is one target: one demo labels a robot episode and "splits the episode into six pick-and-place events", another plans and checks the work of a VLA. "Glimpse is available through API and our chat UI today": "Input tokens cost $0.60 per million and output tokens $3.00 per million". 🆕 ⚠️ Benchmarks are self-reported; no weights (it was "post-trained on an open weights base model").
  • Jev — TypeSafe — https://docs.typesafe.ai/introduction · model + docs, undated — "TypeSafe's flagship model and the first System One model": instead of generating text, it answers typed questions (Choice, Score, Noul) about a state with structured values that code can act on directly. Can Jev Nav? (§4a) tests it as a real-time navigation policy, and Robo Use (§1a) added a harness for it in 0.6.0. 🆕 ⚠️ The docs give no date, access route or pricing.
  • UnifoLM-ER-1 — Unitree — https://huggingface.co/unitreerobotics/UnifoLM-ER-1 · open model, 2026-09-11 — An open 4B embodied-reasoning VLM built on Qwen3-VL-4B, released with Unitree's whole-body humanoid model UnifoLM-WLA-1.0, a "next-generation general-purpose humanoid robot foundation model with 6B parameters". Apache-2.0, ungated. 🆕

2b · Vision-language-action models and learned policies

  • FIBER — Amazon FAR with CMU and Stanford (Lin, Shi, Liu) — https://fiberaa.github.io/ · paper https://arxiv.org/abs/2610.24675 · code https://github.com/Yutang-Lin/FIBER · paper, 2026-10-17 — "Action-Aligned Latent Planning for Versatile Humanoid Loco-Manipulation": one perceptive planner, trained offline on the rollouts of separately trained skill experts, outputs latent commands for a frozen behaviour foundation model on a Unitree G1 with Dex3 hands; the planner replans at 5 Hz while the frozen model closes the loop at 50 Hz, for box pickup, door opening and object grasping from onboard depth. Isaac Sim (§5a) judges success by lift and hold time, or by passing the doorway: "Across 128 diverse variations per task in simulation, FIBER achieves 42.0% mean success, more than tripling projected latent replay (11.6%) and substantially outperforming kinematic planning (3.5%)." On the real robot it "succeeds in 6/10 box-pickup trials and 4/10 door-opening trials", and 2/10 grasps. A playground runs the planner and MuJoCo in the browser. Repository Apache-2.0. 🆕 ⚠️ "Code coming soon.": the repository holds only the paper and figures; the playground's ONNX models are published with no licence, and it runs a distilled planner, "so success rates may be lower than reported".
  • RT-2 — Brohan et al. (Google DeepMind) — https://arxiv.org/abs/2307.15818 · paper, 2023-07-28 — "Vision-Language-Action Models Transfer Web Knowledge to Robotic Control": co-fine-tunes a web-scale vision-language model on robot trajectories and web tasks so one model maps robot observations to actions and carries over semantic reasoning from web data.
  • openpi — Physical Intelligence — https://github.com/Physical-Intelligence/openpi · API beta https://www.pi.website/blog/api-beta · models + code, 2024-10-21; π0.5 2025 — Open weights for π0, π0-FAST and π0.5 with fine-tuned DROID, ALOHA and LIBERO checkpoints and a websocket policy server; "we provide base model checkpoints, pre-trained on 10k+ hours of robot data". A hosted fine-tuning and inference API entered beta on 2026-10-13. Apache-2.0. 🆕 ⚠️ The API beta is by request, and the model behind it is not named.
  • GR00T N1.7 — NVIDIA — https://github.com/NVIDIA/Isaac-GR00T · weights https://huggingface.co/nvidia/GR00T-N1.7-3B · model, 2026-02-25 — A 3B foundation model for generalist robots: a Cosmos-Reason2-2B backbone with a flow-matching action head, pretrained with added human video. 🆕 ⚠️ Code is Apache-2.0, but the model card says "This model is released under the NVIDIA Open Model License Agreement." while the README calls it Apache 2.0; the backbone is gated.
  • OpenVLA-OFT — Stanford — https://openvla-oft.github.io/ · paper https://arxiv.org/abs/2502.19645 · OpenVLA https://arxiv.org/abs/2406.09246 · paper + models, 2025-02-27 — A fine-tuning recipe (parallel decoding, action chunking, continuous actions, L1 loss) for the open 7B OpenVLA, "significantly boosting OpenVLA's average success rate across four task suites from 76.5% to 97.1%". MIT. 🆕
  • SmolVLA — Hugging Face — https://huggingface.co/blog/smolvla · paper https://arxiv.org/abs/2506.01844 · model, 2025-06-02 — "a compact (450M), open-source Vision-Language-Action model for robotics that runs on consumer hardware", trained on community LeRobot data with asynchronous inference; the baseline LeRobot ships with its benchmark environments (§4f). Apache-2.0. 🆕
  • MolmoAct 2 — Ai2 — https://allenai.org/blog/molmoact2 · paper https://arxiv.org/abs/2605.02881 · code https://github.com/allenai/molmoact2 · model, 2026-05-04 — An open action-reasoning VLA (a Molmo2-ER backbone with a flow-matching action expert) released with checkpoints, a bimanual dataset and training code; "Overall, MolmoAct 2 averages 87.1% success, compared with 48.4% for MolmoBot and 45.2% for π0.5." Code Apache-2.0. 🆕 ⚠️ The weights state no licence; the comparison is Ai2's own.
  • LingBot-VLA 2.0 — Robbyant (Ant Group) — https://github.com/Robbyant/lingbot-vla-v2 · paper https://arxiv.org/abs/2607.06403 · model, 2026-07-07 — An open VLA trained across many robot configurations and egocentric video: "we revamp the data processing pipeline and curate around 60,000 hours of data for pretraining". Apache-2.0. 🆕
  • Xiaomi-Robotics-1 — Xiaomi — https://github.com/XiaomiRobotics/Xiaomi-Robotics-1 · paper https://arxiv.org/abs/2607.15330 · model, 2026-07-16 — A Qwen3-VL plus diffusion-transformer VLA pretrained on robot-agnostic UMI data, "training on over 100k hours of real-world manipulation trajectories collected via UMI devices", then post-trained on data from several robots. Apache-2.0. 🆕
  • Gemini Robotics 2 — Google DeepMind — https://deepmind.google/blog/gemini-robotics-2-brings-whole-body-intelligence-to-robots/ · SDK https://github.com/google-deepmind/gemini-robotics-sdk · model, 2026-07-30 — A whole-body, dexterous VLA shown on Apptronik's Apollo 2 humanoid and a Franka Duo, and On-Device 2, a VLA that runs locally and adapts to a new robot body with a few hours of data; the Safari SDK covers serving, evaluation and fine-tuning. 🆕 ⚠️ "Our VLA and On-Device models are available to early-access partners"; most SDK functions need the Trusted Tester programme.
  • DemoMimic — Stanford (Sharma, Sahoo, Huang, Fei-Fei Li, Wu, Sadigh, Bohg) — https://demomimic.github.io/ · paper https://arxiv.org/abs/2609.01938 · paper, 2026-09-01 — "One Demonstration, Many Objects": a dexterous multi-fingered-hand policy learned from one human demonstration by attending to object geometry near the contact points, with contact-centric rewards for precise contact and sim-to-real consistency; on real hardware, "DemoMimic achieves 71% success across 16 objects, four tasks, and two robot-hand embodiments". 🆕 ⚠️ Code "coming soon".

3 · World models

Primers: World Action Models: A Survey (Shen … Wang, 2026-06-18), which separates world-action models from world models, video generators and VLAs and says "This survey gives the field a common account", and From World Models to World Action Models: A Concise Tutorial for Robotics (Zhang … Zhang, 2026-07-01), which compares World Labs, JEPA and Cosmos. 🆕

3a · World-action models and latent world models for control

  • SuperMimic — UC Berkeley (Liao, Shao, Long, Zakka, Horowitz, Abbeel, Sreenath) — https://supermimic.github.io/ · paper https://arxiv.org/abs/2610.28480 · paper, 2026-10-20 — "Scaling Humanoid Whole-Body Control with One Differentiable World Model": a single Unitree G1 motion tracker for the public BONES-SEED motion set, trained by backpropagating through a world model fitted to logged MuJoCo Warp (§5a) transitions, so the simulator is never differentiated; the world model and the policy train together in one loop, and the policy runs on a real G1 (CoRL 2026 oral). "In 3 hours on a single GPU, SuperMimic surpasses the return of a state-of-the-art motion tracker baseline trained for 7 days on 128 GPUs" (SONIC), and it "fails on only 25 of 129,514 benchmark motions, versus 179 failures for the baseline". 🆕 ⚠️ No code or weights; the in-browser demo is "All rights reserved" and licensed only for personal, non-commercial evaluation; failures are counted in simulation, one playback per motion, by SuperMimic's own termination rule, and "Benchmark motions are training motions for both methods, so the counts measure coverage, not generalization."
  • 3D Point World Models (3DPWM) — Oregon State University (Peri, Nguyen, Kim, Li, Lee) — https://pvskand.github.io/3dpwm/ · paper https://arxiv.org/abs/2607.00148 · code https://github.com/pvskand/3dpwm · paper, 2026-06-30 — "3D Point World Models: Point Completion Enables More Accurate Dynamics Learning": the object's partial point cloud is completed first, a Point Transformer V3 predicts how the points move under the robot's actions, and an MPPI planner plans with the rollouts, on ManiSkill (§4b), MimicGen and DexMimicGen tasks with a Franka gripper and a dexterous hand, and on a real Franka. "With GT completion, 3dPwm achieves 88% success on PickCube, decreasing to 78% on StackCube and 60% on PickYCB as task complexity increases." (full text) Repository MIT. 🆕 ⚠️ The repository's code section reads "Coming soon.", with no weights; the quoted rates use ground-truth completion, which the paper says learned completion lowers on the harder tasks.
  • Dream-1 — Dream Labs — https://dreamlabs.ai/research/dream-1/ · API https://platform.dreamlabs.ai · client https://pypi.org/project/dreamlabs/ · model + API, 2026-10 — "a World Action Model that controls robots it has never seen, performs tasks it was never shown, and runs from $3/hour": it predicts video and actions together, takes camera images, robot state and a typed task, and returns joint or end-effector targets for bimanual parallel-gripper arms. Evaluated on 27 unseen simulated robots relative to the YAM it was trained on, and in real-robot clips; fine-tuning is offered as a service. Robo Use (§1a) added a harness for it in 0.5.0. Client Apache-2.0. 🆕 ⚠️ Hosted only, no weights; a "research beta" that "has not been validated for safety-critical use".
  • DreamZero — NVIDIA GEAR — https://github.com/dreamzero0/dreamzero · paper https://arxiv.org/abs/2602.15922 · model, 2026-02-17 — "World Action Models are Zero-shot Policies": a world-action model on the Wan2.1-I2V-14B video model that predicts video and actions together; "This results in over 2x improvement in generalization to new tasks and environments compared to state-of-the-art VLAs". Code Apache-2.0. 🆕 ⚠️ The DROID weights are CC BY-NC 4.0. Not related to Dream Labs.
  • Cosmos 3 — NVIDIA — https://nvidianews.nvidia.com/news/nvidia-launches-cosmos-3-the-open-frontier-foundation-model-for-physical-ai · policy https://huggingface.co/nvidia/Cosmos3-Nano-Policy-DROID · models, 2026-05-31 — An open model family for physical AI covering reasoning, world generation and actions; the Nano-Policy-DROID checkpoint is a world-action policy whose card reads "Given language instructions and visual observations from the DROID robot platform, generate robot action trajectories". OpenMDW-1.1, ungated. 🆕
  • FLUX 3 Action — Black Forest Labs — https://bfl.ai/models/flux-3-action · weights https://huggingface.co/black-forest-labs/flux-3-action-so101 · code https://github.com/black-forest-labs/flux-action · model, 2026-09-22 — "FLUX 3 Action is an open weights 7B world action model." It generates action chunks together with the next video frames, and its SO-101 checkpoint runs through LeRobot. Code Apache-2.0. 🆕 ⚠️ The weights are gated under the non-commercial FLUX Kommunity License.
  • V-JEPA 2 — Meta FAIR — https://github.com/facebookresearch/vjepa2 · paper https://arxiv.org/abs/2506.09985 · model, 2025-06-11 — A self-supervised video world model whose action-conditioned variant plans pick-and-place zero-shot on Franka arms after post-training "using less than 62 hours of unlabeled robot videos from the Droid dataset". MIT. 🆕
  • RoboJEPA — FAIR at Meta — https://github.com/facebookresearch/robo_jepa · paper https://arxiv.org/abs/2610.10515 · model, 2026-10-07 — A latent world model trained across many robot embodiments, with compute scaling laws and zero-shot planning toward goal images on real robots; "RoboJEPA, at 8B parameters, is the largest JEPA predictor model trained to date". 🆕 ⚠️ CC BY-NC-SA 4.0.

3b · World models as simulators and evaluators

  • Evaluating Gemini Robotics Policies in a Veo World Simulator — Google DeepMind — https://arxiv.org/abs/2512.10675 · site https://veo-robotics.github.io/ · paper, 2025-12-11 — An action-conditioned, multi-view Veo model used to evaluate robot policies and to edit scenes for generalisation and safety tests, checked against "1600+ real-world evaluations of eight Gemini Robotics policy checkpoints and five tasks for a bimanual manipulator". 🆕 ⚠️ The simulator is not released.
  • RoboWorld — KAIST — https://arxiv.org/abs/2607.01060 · paper, 2026-07-01 — "Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation": a video world model rolls out policies and a VLM scores task progress, which is reported to "align strongly with real-world robot evaluation across tasks and environments, achieving Pearson's r = 0.989". 🆕 ⚠️ Release of code and weights not verified.
  • ProWorld — Sun Yat-sen University, MBZUAI, Meituan and others — https://arxiv.org/abs/2610.15294 · paper, 2026-10-11 — A multi-view world model that feeds robot state back to the policy for closed-loop evaluation; "it attains a Pearson correlation of 0.95 and a Spearman correlation of 0.94 between generated and real success rates". 🆕 ⚠️ Release of code and weights not verified.
  • GE-Sim 2.0 — AgiBot — https://github.com/AgibotTech/GE-Sim-V2 · paper https://arxiv.org/abs/2605.27491 · model, 2026-05-26 — A closed-loop video world simulator for manipulation with a robot-state module and a success judge; "GE-Sim 2.0 tops the public WorldArena leaderboard at only 2B parameters". 🆕 ⚠️ Most code and data are CC BY-NC-SA 4.0.
  • Genie 3 — Google DeepMind — https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/ · model, 2025-08-05 — An interactive world model that turns a prompt into a world an agent can move through: "Genie 3 can generate dynamic worlds that you can navigate in real time at 24 frames per second"; DeepMind tested it with its SIMA agent. 🆕 ⚠️ Research preview only (Project Genie); no robot interface.

4 · Benchmarks and evaluations

4a · Agent-as-policy benchmarks and studies

Agents in control of a robot, simulated or real. For each, the entry says who judges success and how many trials ran when the source says so.

  • Embodied Playground — HKU, Sharpa, Tsinghua, Stanford, UNC Chapel Hill, MIT (Chen, Zhou, He … Ma) — https://winniechen2002.github.io/Agent-bench-website/ · paper https://arxiv.org/abs/2610.27039 · code https://github.com/Winniechen2002/Embodied-Playground · benchmark, 2026-10-19 — "A Long-Horizon Agentic Robotics Benchmark for Physical, Strategic, and Spatial Reasoning": a foundation-model agent reads the goal, writes control code and acts through a describe, observe, step and finish interface on 410 environments (spring launchers, Tower of Hanoi, interlocking Kongming locks, cubes, household tasks), at access levels from RGB and proprioception only (L0) to read-only simulator state (L3), and the environment itself checks the goal, the task's rules and physical validity. EP-100 takes 100 tasks in five suites, four conditions each; the Launch suite runs on MuJoCo (§5a), the others on Isaac Lab or ManiSkill (§4b). "GPT-6-Astra achieves 13.80% Pass@1 and 19.94% Pass@2 on EP-100", and "Opus-5.5-extra achieves 29.22% Pass@1 and 35.85% Pass@2 using 683 valid attempts." (full text) Code Apache-2.0; original documentation and assets CC BY 4.0. 🆕 ⚠️ The paper runs GPT-6-Astra "with high reasoning effort and L3 privileged observations" and does not state the Claude run's level; most suites need Isaac Sim and an NVIDIA GPU; bundled third-party assets keep their own terms.
  • Can Jev Nav? — Dimensional (Pomichter, Dasyam, Ventura) — https://research.dimensional.org/system-one-navigation/ · runs https://research.dimensional.org/system-one-navigation/tasks/ · code https://github.com/dimensionalOS/dimos · benchmark, 2026 — "Have System One “LLMs” finally broken the real-time robotics frontier?": 327 start-to-object navigation tasks for a quadruped in Habitat-Sim homes (HSSD), 1,962 recorded runs scored with SPL and SoftSPL, comparing coding agents, TypeSafe's Jev and Dimensional's own skill-level agent, with cost per run. Mean SPL: Dimcode with navigation skills 0.743, Astra as a coding agent 0.522 ($1.610 per run), Jev 0.263 ($0.081). 🆕 ⚠️ No licence stated for the tasks; HSSD scenes are CC BY-NC 4.0 (§5c); the top row had navigation skills the other agents lacked.
  • CaP-X — NVIDIA, UC Berkeley, Stanford, CMU — https://capgym.github.io/ · paper https://arxiv.org/abs/2603.22435 · code https://github.com/capgym/cap-x · benchmark, 2026-03-23 — "A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation": models write Python over perception and control primitives on LIBERO-PRO, robosuite and BEHAVIOR tasks, in simulation and on real robots, at levels of abstraction from privileged state to raw perception, across 12 models; "today's best frontier models can directly generate executable robot control code and achieve over 30% average success". It adds a training-free agent (CaP-Agent0) and RL with verifiable rewards (CaP-RL). MIT. 🆕 ⚠️ BEHAVIOR and Isaac Sim terms apply to some tasks; SAM 3 weights are gated.
  • RLE-Bench — Harvard, Georgia Tech — https://rle-bench.github.io/ · paper https://arxiv.org/abs/2609.34210 · code https://github.com/RLE-Bench/RLE-Bench · benchmark + leaderboard, 2026-09-14 — "A Qualifying Exam for Coding Agents as Robot Learning Engineers": robot-learning tasks across four workflows (interactive control, policy learning, perception and estimation, mechanical design), run by Claude Code, Codex, Antigravity and Grok Build and scored by task metrics combined into an RLE Index; "GPT-6 Astra and Claude Fable 5.1 lead the leaderboard, with visual grounding emerging as a key differentiator." (full text) MIT. 🆕
  • EmbodiedSWE-Bench — Yale, ByteDance Seed, Princeton, Stanford, CMU, UW, UCLA — https://embodiedswe.github.io/ · paper https://arxiv.org/abs/2609.27308 · code https://github.com/EmbodiedSWE/EmbodiedSWE · benchmark, 2026-09-23 — "Coding Agents for Long Horizon Dexterous Robotics": Codex and Claude Code write solution programs in Isaac Sim and Newton, "6 suites · 28 tasks · 17 embodiments", and hidden graders score physical task progress offline; "GPT-6 Astra passes 0.9 inside two hours; every other curve flattens well below ceiling." Apache-2.0. 🆕 ⚠️ The site also reports reward-hacking rates by model.
  • CodeActionBench — Nanyang Technological University — https://codeactionbench.org/ · paper https://arxiv.org/abs/2609.33807 · code https://github.com/lyhkk/CodeActionBench · benchmark, 2026-09-27 — "Evaluating Agentic Code-as-Policy for Embodied Manipulation": Codex CLI and Claude Code drive RoboTwin 2.0's dual-arm robot over MCP on 25 SAPIEN tasks, judged by a hidden physical verifier; "The strongest configuration, GPT-6 Astra with Codex CLI, solves 22 of 25 tasks at least once in three attempts". MIT. 🆕 ⚠️ "Full data (coming soon)".
  • An Unexpected Robot Policy — RoboProbe and RoboDojo teams (HKU, Tsinghua, UC Berkeley and others) — https://arxiv.org/abs/2609.24170 · code https://github.com/RoboProbe/RoboProbe · paper, 2026-09-21 — "Early Evaluations of GPT-6 Astra on RoboDojo and Beyond": the LLM picks every end-effector target on all 42 RoboDojo simulation tasks; "GPT-6 Astra achieves 22.48% average success rate and 28.97 Score over 2,100 trials, ranking above every public entry." Apache-2.0. 🆕 ⚠️ On the real robot, "Testing was halted after the model repeatedly issued physically unreasonable or unsafe actions" (full text).
  • RobotWorld — Yang et al. — https://robotworldai.github.io/ · paper https://arxiv.org/abs/2610.10409 · code https://github.com/robotworldai/robotworld · benchmark, 2026-10-07 — "Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments": agents drive simulated robots through tools on 84 tasks (manipulation, mobile manipulation, locomotion, driving, aerial) with success checks the agent cannot see; "Astra completes 16 of 84 tasks (19.0%), followed by Opus 5.5 with 13 (15.5%)" (full text). 🆕 ⚠️ One episode per model and task; no licence for the code; affiliations not stated.
  • LIBERO-Agent — Fudan University, Shanghai Innovation Institute — https://arxiv.org/abs/2609.39507 · code https://github.com/dzj441/Libero-Agent · benchmark, 2026-09-30 — General-purpose agents (Codex, Claude Code and others) send native 7-D end-effector actions to a simulated Panda over MCP instead of calling high-level skills, and the simulator checks success. "Results reveal a pronounced reliability gap" on hard and long-horizon tasks; "Among these agents, GPT-6 Astra achieves the strongest overall performance." MIT. 🆕 ⚠️ Only the 30-task primary suite is released.
  • RoboQuest — DeCLaRe Lab, NTU — https://declare-lab.github.io/RoboQuest/ · paper https://arxiv.org/abs/2610.10388 · code https://github.com/declare-lab/RoboQuest · benchmark, 2026-10-07 — Ten RoboCasa365 mobile-manipulation tasks that need searching, inspecting or testing before committing, with models acting as the policy; "Of six frontier models, Gemini 4 Argon and GPT-6 Astra do best and are tied, solving 23.8% and 23.2% of episodes." MIT; demonstrations in §7. 🆕
  • BiGym 2.0 — Imperial College London — https://bigym2.github.io/ · paper https://arxiv.org/abs/2610.07594 · code https://github.com/swirl-uk/BiGym2 · benchmark, 2026-10-06 — Unitree G1 household tasks where VLA, imitation and RL baselines meet coding agents that write policy.py, scored on hidden seeds; "GPT-6 Astra and Claude Opus 5.5 write controllers that beat every demo-driven RL baseline on the nine-task mean". Apache-2.0; agent rollouts in §7. 🆕 ⚠️ π0.5 still has the best nine-task mean.
  • AgenticGenPlan — Princeton, FBK, CMU, Cambridge — https://agenticgentamp.github.io/ · paper https://arxiv.org/abs/2609.30233 · code https://github.com/tomsilver/robocode · KinDER https://prpl-group.com/kinder-site/ · paper, 2026-09-24 — "Coding Agents for Generalized Task and Motion Planning Problems": Claude Code and Codex write one program per KinDER physical-reasoning environment (RSS 2026) that must generalise to held-out instances, "(56% to 95% versus 47% for the planners, on the 16 environments where a planner is available)". MIT. 🆕
  • Embodied Agent Arena — HKUST (Guangzhou), CUHK, Knowin AI — https://embodied-agent-arena.github.io/embodied-agent-arena/ · paper https://arxiv.org/abs/2610.00854 · benchmark, 2026-10-01 — "Are Frontier VLM Agents Ready to Be Robot Generalists?": cases drawn from many sources across geometry, spatial reasoning, affordance, planning and manipulation, scored on the final state with a minimal harness; "Different models lead; even the strongest manipulation result leaves over half the tasks incomplete." MIT. 🆕 ⚠️ The dataset states no licence.
  • Embodied First Exam — MLL Lab (Kangrui Wang … Manling Li) — https://www.embodied-first-exam.ai/ · code https://github.com/mll-lab-nu/embodied-first-exam · blog https://www.embodied-first-exam.ai/blogs/first-exam/ · benchmark, 2026-10-19 — "Code agents have passed many exams in the digital world. What is their first exam to enter the physical world?": frozen tests from many robot suites (a core set of 449 tasks), answered with a program and marked by replay in a separate sandbox; "GPT-6 Astra, the strongest agent, solves only about a third of the standard and standard-realtime tests." MIT. 🆕 ⚠️ One attempt per test.
  • AA-Robotics v0.1 — Artificial Analysis — https://artificialanalysis.ai/articles/aa-robotics · benchmark, 2026-10-14 — An independent evaluator's first robotics index: models in Codex, Claude Code or Antigravity drive a real Hiwonder arm over HTTP on eight tabletop tasks with partial credit, with time and cost; "GPT-6 Astra places first in AA-Robotics v0.1 with a score of 69". 🆕 ⚠️ "A human judge scores the end state"; no code or data; the number of repeats is not stated.
  • EmbodiedBench — UIUC, Northwestern, Toronto, TTIC — https://embodiedbench.github.io/ · paper https://arxiv.org/abs/2502.09560 · benchmark, 2025-02-13 — Multimodal LLM agents with high- and low-level actions on "a diverse set of 1,128 testing tasks across four environments", from household planning to navigation and manipulation (ICML 2025). 🆕 ⚠️ The code repository has no licence file.
  • Embodied Agent Interface — Stanford, Northwestern, Amazon, MIT — https://embodied-agent-interface.github.io/ · paper https://arxiv.org/abs/2410.07166 · benchmark, 2024-10-09 — Splits embodied decision-making into goal interpretation, subgoal decomposition, action sequencing and transition modelling, with fine-grained error metrics in BEHAVIOR and VirtualHome (NeurIPS 2024 D&B); "o1-preview has the highest task (81.0%) and execution success rates (91.0%) in BEHAVIOR." MIT. ⚠️ Symbolic actions only, no continuous control.
  • Claude plays robotics — Anthropic Frontier Red Team — https://www.anthropic.com/research/claude-plays-robotics · study, 2026-07-09 — Twelve models at four levels of control (direct actions, programs, steering a pretrained policy, supervising RL) on simulated classic control, Unitree G1 and Go2 and LIBERO tasks, plus a real Go2; under direct control on LIBERO, models "only complete the full task a small percentage of the time (from 0 to 5.5%)". 🆕 ⚠️ The code is promised at github.com/safety-research/embody, which returns 404.
  • Project Fetch: Phase two — Anthropic — https://www.anthropic.com/research/project-fetch-phase-two · study, 2026-06-18 — Claude Code with Opus 4.7 re-runs 2025's human-team experiment on a quadruped "robodog" (connect to its camera and lidar, find and fetch a beach ball), three trials per task: "Opus 4.7 was, on average, more than 37 times faster than Team Claude-less". 🆕 ⚠️ A researcher approved commands and task transitions; closed-loop ball handling stayed hard ("Claude struggled to capture this subtlety").
  • Drone-Bench — Andon Labs, with Anthropic's Project Pilot — https://andonlabs.com/evals/drone-bench · benchmark, 2026-07 — Models write code for a DJI Tello EDU to find and follow a person across five sub-tasks, each against a human baseline, on a benchmark held out from model developers: "since no run has ever beaten the Reconstruct baseline, end-to-end success is currently 0%". Project Pilot (2026-07-24) adds that "even Fable 5, the current frontier model, reaches the human baseline on average for only three of the five tasks". 🆕 ⚠️ Cheating is common and judged by an LLM ("fable-5-1: 66.0% of 162 reviewed runs with any cheating"); no public code or data.
  • Butter-Bench — Andon Labs — https://andonlabs.com/evals/butter-bench · paper https://arxiv.org/abs/2510.21860 · benchmark, 2025-10-23 — "Can LLMs control robots? We answer this by testing how good models are at passing the butter – or more generally, do delivery tasks in a household setting." "The best LLMs score 40% on Butter-Bench, while the mean human score is 95%." 🆕 ⚠️ No public code.
  • Agent as Policy for Robotic Manipulation — Notre Dame, UC San Diego, San Diego State — https://agent-as-policy-2026.github.io/ · paper https://arxiv.org/abs/2609.12541 · code https://github.com/agent-as-policy-2026/agent-as-policy · paper, 2026-09-11 — A general-purpose agent is the policy on real bimanual YAM arms: it reads the cameras, writes programs, commands motion and checks the result, with GPT-6 Astra in Codex CLI by default; "Taken together, the eight rows hold 57 successes in 62 trials." Trials and traces are public (§7). Code Apache-2.0. 🆕 ⚠️ The 57/62 figure is on the project page, not in the paper.
  • Can a Frontier VLM Serve as a Robot Policy for Memory-Dependent Manipulation? — Harvard (Jedlicki, Sethi, Yang) — https://embodied-astra.github.io/ · paper https://arxiv.org/abs/2610.17350 · code https://github.com/jackbjed/embodied-astra-robomme · paper, 2026-10-12 — A frozen GPT-6 Astra outputs the waypoints for all 16 RoboMME memory tasks with the whole episode in context: "On the 800 test episodes it succeeds in 92.1%, above the previously reported human score of 90.5%". Code, prompts and per-episode logs are released. MIT. 🆕 ⚠️ Simulation, a single run, an undated closed model; the human score used an oracle skill interface.
  • GPT-6 Astra on robotic manipulation — Robocurve — https://openai.robocurve.org/gpt-6-astra/ · report, 2026-09-04 — A real-robot report per model release: models drive bimanual YAM arms through the Inspect Robots agent policy, 20 trials per task, graded on a 0–4 stage rubric; "On the bowl task Astra placed the block in 19 of 20 trials, against Fable 5.1's 8 of 20 and Fable 5 in 1 of 20". 🆕 ⚠️ By its own limitations section, trials were "not interleaved" and grading was "operator-judged with the model known".
  • HumanCLAW — Meta, NTU, UW, Brown, Northwestern — https://human-claw.github.io/ · paper https://arxiv.org/abs/2607.27180 · code https://github.com/Human-CLAW/HumanCLAW · benchmark, 2026-07-29 — "Can Vision-Language Models Act Through a Body?": a VLM picks an atomic skill every 0.5 s for a simulated human body to find and sit on a target across 1,218 episodes in HSSD houses; "No current VLM solves the benchmark. The best model finds and sits on the target in only 16.8% of episodes". Apache-2.0. 🆕 ⚠️ A human avatar, not a robot; the data is gated and the HSSD scenes are non-commercial.
  • Embodied Agents Take Control (MIP) — Zhou … Wu — https://jianzhou0420.github.io/mip/ · paper https://arxiv.org/abs/2607.26148 · code https://github.com/jianzhou0420/MIP · paper, 2026-07-28 — Minimal-interface agents (Claude Agent SDK, Codex CLI, mini-SWE-agent) navigate with only observe() and step() across Habitat benchmarks such as R2R-CE and RxR-CE, plus Go2 trials: "On 100-episode subsets, the best agents exceed published scores and approach or surpass our human tester." MIT. 🆕 ⚠️ 100-episode subsets; the scene datasets have their own non-commercial terms.
  • DrivingBench — Ramabadran, Mahns, Gessler — https://drivingbench.com/ · paper https://arxiv.org/abs/2609.38948 · harness https://github.com/aditya-ramabadran/drivingbench_harness_v1 · benchmark, 2026-09-30 — "Can Vision-Language Models Drive a Toyota Corolla?": agents in Codex, Claude Code or Cursor drive a real car with modified openpilot around a cone course through three MCP tools; "Astra is the only model to finish the course, on its second attempt, with no other attempt passing 50% of the course." MIT. 🆕 ⚠️ One course and a handful of attempts, with a safety driver on board.

4b · Manipulation suites

4c · Locomotion, whole-body and dexterous control

  • DeepMind Control Suite — Google DeepMind — https://github.com/google-deepmind/dm_control · paper https://arxiv.org/abs/1801.00690 · benchmark, 2018-01-02; 1.0.49 2026-10-14 — "A set of Python Reinforcement Learning environments powered by the MuJoCo physics engine.": the standard continuous-control suite, plus locomotion, soccer and MJCF tooling. Apache-2.0.
  • HumanoidBench — UC Berkeley — https://humanoid-bench.github.io · paper https://arxiv.org/abs/2403.10506 · benchmark, 2024-03-15 — A Unitree H1 with two dexterous hands in MuJoCo: "HumanoidBench is the first-of-its-kind simulated humanoid robot benchmark, including 27 distinct whole-body control tasks". MIT. ⚠️ Quiet since 2025-09.
  • MyoSuite — MyoHub — https://github.com/MyoHub/myosuite · paper https://arxiv.org/abs/2205.13600 · benchmark, v3.0.0 2026-10-11 — Contact-rich musculoskeletal control in MuJoCo; v3 adds GPU training through mjlab: "Train with thousands of environments in parallel on a single GPU, then replay the policy on the CPU." Apache-2.0. 🆕 ⚠️ v3 changed observations for many environments, so v2 results do not compare.
  • MuJoCo Playground — Google DeepMind and collaborators — https://playground.mujoco.org · paper https://arxiv.org/abs/2502.08844 · environments, 2025-02-12 — GPU locomotion, dexterous-manipulation and vision environments on MJX and MuJoCo Warp, with sim-to-real transfer; "researchers can train policies in minutes on a single GPU". Apache-2.0. 🆕 ⚠️ Needs a CUDA GPU.
  • LocoMuJoCo — Al-Hafez … Tateo — https://github.com/robfiras/loco-mujoco · paper https://arxiv.org/abs/2311.02496 · benchmark, 2023-11-04; v1.1.0 2026-03-10 — Imitation learning for locomotion in MuJoCo and MJX from retargeted motion capture: "Includes 12 humanoid and 4 quadruped environments, featuring 4 biomechanical human models". Code MIT. 🆕 ⚠️ The datasets are CC BY-NC-ND 4.0.
  • DexJoCo — CASIA with SJTU, MBZUAI, PKU and CUHK — https://dexjoco.github.io · paper https://arxiv.org/abs/2605.16257 · code https://github.com/brave-eai/dexjoco · benchmark, 2026-05-15 — Task-oriented dexterous manipulation with human demonstrations and π0.5 baselines: "DexJoCo addresses these gaps with 11 functionally grounded tasks built around the Franka Panda and Allegro Hand." MIT. 🆕
  • Microduck Arena — Pollen Robotics (Hugging Face) — https://huggingface.co/spaces/pollen-robotics/microduck-arena · code https://github.com/pollen-robotics/microduck-arena · leaderboard, 2026-09-10 — A hosted locomotion leaderboard for a small biped where "Submitted policies are evaluated server-side on a fixed course over five perturbed seeds", so the host measures every time instead of trusting submitters. 🆕 ⚠️ No licence declared; times reproduce only on the Space's hardware.
  • Lucen Worlds — Lucen Robotics — https://lucenworlds.com/ · game + leaderboard, undated — A browser game of daily challenges (a penalty shootout, a balance beam, a slalom and more) for simulated builds of Lucen's modular robot kit, such as the Laika v2 biped, with MuJoCo running in the page and separate leaderboards for people, AI and policies. Agents connect through a link: "Claude Code or any MCP client, or anything that can make HTTP calls (Gemini, a script)" gets the tools the game gives any agent (look, walk to a point, kick, move a hand, grip), and "It plans; the game's autopilot does the steering." Policies trained on Lucen's MouseMouse hub, which has an Apache-2.0 MCP server (lucen-mcp), load into the game. Unlike Microduck Arena above, scores are not measured on a server: "Results come from players' own browsers." 🆕 ⚠️ The game has no public source or licence; the Laika v2 model files are CC BY-NC 4.0; new, with few players yet.

4d · Navigation and driving

4e · Robot safety

4f · Evaluation harnesses, leaderboards and real-robot evaluation

  • vla-eval — MAUM.AI, Ai2, Seoul National University — https://github.com/allenai/vla-evaluation-harness · paper https://arxiv.org/abs/2603.13966 · leaderboard https://allenai.github.io/vla-evaluation-harness/leaderboard/ · harness, v0.8.0 2026-09-30 — Separates model servers from Docker-isolated simulation benchmarks so any VLA runs on any integrated benchmark, and "achieves up to 47x wall-clock speedup, completing 2,000 LIBERO episodes in ~18 minutes". Apache-2.0. 🆕 ⚠️ The leaderboard is in beta and aggregates numbers reported in papers rather than re-running them.
  • XPolicyLab — MMLab@HKU, Tsinghua and the XPolicyLab community — https://github.com/XPolicyLab/XPolicyLab · paper https://arxiv.org/abs/2608.09892 · standard + harness, 2026-08-10 — Shared observation, action and trajectory schemas with a client-server adapter, so N policies and M environments need N+M integrations; the same adapters serve RoboTwin, RoboDojo and real robots. "The ecosystem integrates 42 robot policies and standardizes their installation, debugging, serving, and evaluation workflows." Apache-2.0. 🆕
  • LeRobot — Hugging Face — https://github.com/huggingface/lerobot · v0.6.0 post https://huggingface.co/blog/lerobot-release-v060 · EnvHub https://huggingface.co/docs/lerobot/envhub · library, v0.6.0 2026-07-06 — The open robot-learning library (policies, datasets, recording and rollout on real robots) whose lerobot-eval gained six simulation benchmarks in v0.6.0, each with a Docker image and a SmolVLA baseline: "Together with LIBERO, Meta-World, and NVIDIA IsaacLab-Arena, that makes nine benchmark families under one roof". Apache-2.0. 🆕 ⚠️ EnvHub runs third-party code: "Loading environments from the Hub executes Python code from third-party repositories."
  • Isaac Lab-Arena — NVIDIA, "with the evaluation and task layers designed in close collaboration with Lightwheel" — https://github.com/isaac-sim/IsaacLab-Arena · framework, 2025-08; v0.3 2026 — Builds simulated evaluation environments from a scene, an embodiment and a task, runs policies in parallel on GPU and reports metrics and sensitivity analyses, with RoboCasa- and LIBERO-style task packs and LeRobot EnvHub integration. Apache-2.0. 🆕 ⚠️ Alpha ("Do not use this in production."); Linux and Isaac Sim only.
  • RoboArena — Atreya, Pertsch … Finn, Levine — https://robo-arena.github.io/ · paper https://arxiv.org/abs/2506.18123 · code https://github.com/robo-arena/roboarena · leaderboard, 2025-06-22 — Distributed, double-blind pairwise A/B evaluation of generalist policies on real DROID robots, ranked from preferences; the evaluation data is public (§7). MIT. 🆕 ⚠️ In June 2026 the team reported "some evidence of benchmark manipulation on RoboArena" (post) and now lets only organisations without an active policy on the leaderboard submit evaluations.
  • RoboChallenge — Dexmal with Hugging Face — https://robochallenge.ai · paper https://arxiv.org/abs/2510.17950 · real-robot benchmark, 2025-10-20 — "an online evaluation system to test robotic control algorithms" on real robots, with a public leaderboard, competitions and "our initial benchmark Table30". 🆕 ⚠️ Data licence not stated.
  • PhAIL — Positronic Robotics — https://phail.ai · paper https://arxiv.org/abs/2605.29710 · code https://github.com/Positronic-Robotics/positronic · leaderboard, 2026-05-28 — A real-robot VLA leaderboard on a Franka FR3 that compares time-to-success distributions with bootstrap intervals and significance tests instead of one success rate; "the closest pair (OpenPI vs. GR00T) remains unresolved within our budget". Code Apache-2.0; data CC BY 4.0. 🆕 ⚠️ Also sold as a commercial evaluation service.
  • Reality Check — Poke & Wiggle — https://pokeandwiggle.com/leaderboard · leaderboard, 2026-09-29 — PAW-GEN-10: ten real bimanual FR3 Duo environments at three training-data tiers, 3,600 rollouts per model, with per-tier results and 95% intervals for MolmoAct 2, π0.5, DiT-Flow and GR00T N1.7; "Each model gets 30 rollouts for every combination of the evaluation axes below." 🆕 ⚠️ The team fine-tunes every model itself; no licence or submission process stated.
  • ArmnetBench v0.1 — Armnet — https://armnet-dev.github.io/armnetbench-v0.1/ · paper https://arxiv.org/abs/2607.24481 · real-robot benchmark, 2026-07-27 — Parallel real-world evaluation of manipulation policies on a farm of low-cost SO-101 arms, every rollout labelled by a person; "Cable clipping remained unsolved. Every evaluated policy scored 0%." Data Apache-2.0. 🆕 ⚠️ Manual resets and human scoring.

4g · Benchmark integrity

  • Faster and Better? — Shanghai Jiao Tong University — https://arxiv.org/abs/2609.37771 · paper, 2026-09-29 — "Benchmark Bugs and Design Limitations Distort the Evaluation of Vision-Language-Action Acceleration": an audit of suspicious gains across seven simulation benchmarks (RoboTwin, LIBERO-Plus, VLABench and others) in which "we identify 22 bugs of these types and 4 design limitations"; "bug fixes can reverse method rankings". 🆕
  • MINERVA — University of Tokyo (Matsuo-Iwasawa Lab) — https://arxiv.org/abs/2609.03715 · code https://github.com/k1000dai/MINERVA · paper, 2026-09-03 — "How Small Can a Manipulation Policy Be and Still Solve LIBERO?": "A 0.54M-parameter policy achieves 95.1% average success over 2,000 rollouts on the four standard LIBERO suites", and permuting task IDs shows it memorises tasks, so a LIBERO score alone does not show generalisation. Apache-2.0. 🆕
  • FailBench — Metric AI Lab — https://metric-ai-lab.github.io/failbench/ · paper https://arxiv.org/abs/2609.03611 · paper, 2026-09-03 — "How Reliable are VLMs at Judging Robot Task Success?": "2,197 manipulation attempts across 14 public sources (12 real-world, 2 simulated)"; "Evaluating 13 VLM-based detectors, we find the best model achieves only 0.77 mean balanced accuracy.", with "a systematic bias toward predicting success under ambiguous evidence". 🆕 ⚠️ Code, data and leaderboard are marked "soon".
  • A Statistical Audit of Physical AI Benchmark Redundancy — Metric AI Lab — https://arxiv.org/abs/2608.25940 · paper, 2026-08-26 — 51 models across physical-AI benchmarks show heavy redundancy; "collapsing the two substitute pairs into single columns moves 22 of 51 models by three or more places". 🆕 ⚠️ Mixes model-card numbers with the authors' own runs; the code link returns 404.
  • GPUSimBench — Shanghai AI Lab, NTU, Nanjing University and others — https://arxiv.org/abs/2607.13059 · paper, 2026-07-06 — Tests GPU simulators (Isaac Lab, Genesis) for scale, physical fidelity and determinism: "we unveil and quantify the inherent non-determinism introduced by GPU-batched execution" (IROS 2026). 🆕 ⚠️ No code link.
  • MuJoCo version silently changes a LIBERO task (LeRobot issue 4390) — LeRobot issue tracker — huggingface/lerobot#4390 · issue, 2026-08-09 — A collision fix in MuJoCo 3.4.0 leaves one LIBERO task's bowl tilted at reset and moves VLA scores sharply: "Whole-suite scores shift by up to ~9 pp from this single task." Pin and report the simulator version. 🆕 ⚠️ One reporter; the fix is an open pull request.

5 · Simulators and asset libraries

5a · Simulators and physics engines

5b · Agents that build simulations

Coding agents increasingly build the simulation as well as act in it; Gemini 4 Argon (§2a) builds a MuJoCo twin from one photo.

  • Plumb — Westlake University and HKU (Liang, Chen, Huang, Chen) — https://yubel426.github.io/Plumb/ · code https://github.com/Inception3D/Plumb · harness + benchmark, code 2026-10-21 — "An Agent Harness for Inverse Physics of Dynamic Scenes": a coding agent (Codex, or Claude Code in a container) gets a video, a MuJoCo sandbox, and SAM 3 (§6) tracking and Pi3X depth and camera poses as tools, and writes the MJCF scene, its initial state, the physical parameters and the fitting code. A separate evaluator replays only the delivered scene and initial state, never the agent's code, against the datasets' annotations, including from a camera the agent never saw. "We evaluate on 50 videos, 25 from PhysInOne and 25 from Physion v1.5, selected by category." With the same GPT-6.1 Sol model, Plumb beats Codex without the 3D tools on every accuracy metric, yet "even with Plumb, only about a third of the physical quantities, such as restitution, deceleration and mass ratios, fall within a factor of 1.25 of the ground truth." Code MIT. 🆕 ⚠️ "arXiv coming soon"; the Pi3X weights are CC BY-NC 4.0, the SAM 3 weights are gated, and PhysInOne is CC BY-NC-SA 4.0; needs an NVIDIA GPU; few outside users yet.
  • GATOR — NVIDIA and Simon Fraser University (Wu, Birchfield, Rabeti, Chang, Wen) — https://research.nvidia.com/labs/lpr/gator/ · paper https://arxiv.org/abs/2610.11215 · paper, 2026-10-08 — "Generative and Agentic 3D Object Reconstruction From Casual Images": "a generative and agentic framework that recovers textured object assets and their scene-relative pose from one or more images", in which GPT-6-Astra, run through the Codex command-line agent with Blender access, repairs each generated mesh; reconstructed rooms settle under gravity in Isaac Sim (§5a). "On ten sampled Toys4K objects, GATOR without refinement takes about 30 seconds and already outperforms the from-scratch agent with a twenty-minute budget on chamfer distance and LPIPS." Paper CC BY 4.0. 🆕 ⚠️ Code "Coming Soon"; "Agentic refinement relies on a proprietary model and adds up to ten minutes of agent time per object."; the output is GLB meshes, not robot-ready MJCF or URDF.
  • Video2World — Aether AI and UC San Diego — https://aetherlabsai.github.io/Video2World/ · paper https://arxiv.org/abs/2610.04432 · code https://github.com/AetherLabsAI/Video2World · benchmark, 2026-10-03 — Coding agents rebuild a robot or human video as interactive simulator code (MuJoCo, SAPIEN, Isaac Sim), scored on geometry, dynamics and task success; "the best of nine frontier agents finishes 25.5% of tasks, while a human-assisted reference reconstruction reaches 58.8%". Apache-2.0. 🆕
  • RoboSimStudio — UC Berkeley and NVIDIA (Coumans, Macklin, Goldberg and others) — https://robosimstudio.github.io/ · paper https://arxiv.org/abs/2610.18835 · code https://github.com/robosimstudio/robosimstudio · toolkit, 2026-10-13 — A real-to-sim toolkit on Newton where Codex or Claude Code builds a simulation from a task description and a video, "allowing both humans and coding agents to quickly create and iteratively refine simulations". Apache-2.0. 🆕 ⚠️ A week old; needs an NVIDIA GPU.
  • EmbodiedGen — Horizon Robotics — https://github.com/HorizonRobotics/EmbodiedGen · paper https://arxiv.org/abs/2607.07459 · generator, v2.1.0 2026-08-19 — Generates simulation-ready, physics-checked 3D assets and scenes for several simulators; "EmbodiedGen ships a Claude Code plugin whose slash commands wrap the generation and spatial-computing skills". Apache-2.0. 🆕 ⚠️ The third-party models it calls keep their own terms; needs a CUDA GPU.

5c · Robot models, scenes and object assets

6 · Perception tools

Models that agents call as tools to see: segmentation, pose, depth, grasps and tracking, plus tools to inspect robot data.

  • Flora — NVIDIA GEAR — https://research.nvidia.com/labs/gear/flora/ · paper https://research.nvidia.com/labs/gear/flora/assets/paper/flora.pdf · models https://huggingface.co/collections/nvidia/flora · models + paper, 2026-10-15 — Three video models fine-tuned from Qwen3.5 that label robot and human episodes: Flora-Rewarder gives "An integer episode score from 1 (complete failure) to 5 (complete success)" or a score for "each 0.5-second step (1 = failure, 2 = suboptimal, 3 = normal)", and Flora-TempSeg and Flora-Captioner write timestamped captions. NVIDIA presents it as "an industrial-strength data engine used for training GR00T models", "roughly ~125× cheaper than GPT-6 Astra" and "labeling 200,000 hours of data per day across 512 GB300s"; on RoboReward its MAE is 40.0 against GPT-6 Astra's 55.4, while Astra leads on the Robot200 captioning subset and on RubricBench. 🆕 ⚠️ Weights and inference scripts are under the NVIDIA OneWay Noncommercial License, and "Flora is not intended or supported for commercial or production use".
  • S4VY — Jingdong Zhang, Xin Li, Jan Kautz, Wenping Wang, Chris Choy — https://arxiv.org/abs/2609.36875 · code https://github.com/Evergreen0929/S4VY · paper, 2026-09-29 — "We introduce S4VY, a Segment Anything model built on feed-forward 4D visual geometry": class-agnostic 4D instance masks that track each object across all frames from RGB, prompt-free or point- and box-prompted, plus an agentic system for natural-language grounding in 4D scenes. 🆕 ⚠️ Code "Coming soon": "The code is pending release approval and will be published in this repository."
  • SAM 3 — Meta — https://github.com/facebookresearch/sam3 · paper https://arxiv.org/abs/2511.16719 · model, 2025-11-20 — Segments and tracks every instance of a concept given as text or an example, in images and video; "It achieves 75-80% of human performance on our new SA-CO benchmark which contains 270K unique concepts". CaP-X uses it as a perception primitive. 🆕 ⚠️ A custom SAM licence with trade-control and ITAR restrictions; the weights are gated.
  • Grounded SAM 2 — IDEA Research — https://github.com/IDEA-Research/Grounded-SAM-2 · pipeline, v1.0 2024-12-11 — "Grounded SAM 2 is a foundation model pipeline towards grounding and track anything in Videos": open-vocabulary detection (Grounding DINO, Florence-2 or DINO-X) plus SAM 2 for text-prompted segmentation and tracking. Apache-2.0. ⚠️ The newer DINO-X detectors are available only through a cloud API.
  • FoundationPose — NVIDIA — https://github.com/NVlabs/FoundationPose · model, 2023-12-13 — 6D pose estimation and tracking of novel objects, with or without a CAD model (CVPR 2024). ⚠️ Non-commercial licence: "The Work and any derivative works thereof only may be used or intended for use non-commercially."
  • GraspGen — NVIDIA — https://github.com/NVlabs/GraspGen · paper https://arxiv.org/abs/2507.13097 · model, 2025-07-17 — A diffusion model that generates 6-DoF grasps for parallel-jaw and suction grippers, served over ZMQ and callable by agents: "Added MCP for calling GraspGen as a tool by an LLM." Dataset CC BY 4.0. 🆕 ⚠️ The code is under a non-commercial NVIDIA licence.
  • VGGT — Meta (Wang, Karaev, Vedaldi, Rupprecht, Novotny et al.) — https://github.com/facebookresearch/vggt · paper https://arxiv.org/abs/2503.11651 · model, 2025-03-14 — Feed-forward prediction of cameras, depth, point maps and tracks from one to hundreds of views (CVPR 2025 Best Paper). 🆕 ⚠️ "We've updated the license for VGGT to permit commercial use (excluding military applications)."; the original VGGT-1B weights stay CC BY-NC 4.0, and the commercial checkpoint is gated.
  • Depth Anything 3 — ByteDance Seed — https://github.com/ByteDance-Seed/Depth-Anything-3 · paper https://arxiv.org/abs/2511.10647 · model, 2025-11-13 — Depth and camera-ray estimation from any number of views, plus metric and monocular depth and a streaming mode for long video. Code Apache-2.0. 🆕 ⚠️ The largest weights are CC BY-NC 4.0; smaller ones are Apache-2.0.
  • CoTracker3 — Meta — https://github.com/facebookresearch/co-tracker · paper https://arxiv.org/abs/2410.11831 · model, 2024-10-15 — Tracks any pixel through a video, offline or online, trained by pseudo-labelling real videos. ⚠️ "The majority of CoTracker is licensed under CC-BY-NC".
  • Molmo 2 — Ai2 — https://allenai.org/blog/molmo2 · code https://github.com/allenai/molmo2 · ER model https://huggingface.co/allenai/Molmo2-ER · models, 2025-12-11 — Open vision-language models for video QA, pointing with coordinates and timestamps, and tracking; Molmo2-ER is the embodied-reasoning backbone of MolmoAct 2 (§2b). Apache-2.0. 🆕 ⚠️ Trained partly on third-party data "subject to academic and non-commercial research use only".
  • Rerun — Rerun — https://github.com/rerun-io/rerun · tool, 0.39.0 2026-10-12 — "Log, query, visualize, and stream to training on shared columnar storage built for multimodal data": the open viewer for robot logs, reading MCAP and LeRobot data. MIT and Apache-2.0. 🆕
  • Foxglove — Foxglove — https://foxglove.dev/blog/foxglove-goes-agentic · tool, 2026-08-18 — Robot-data visualisation with an in-app agent and a local MCP server over recordings (layouts, scripts, semantic search). 🆕 ⚠️ Proprietary; "Agentic workflows are available in beta to all customers."

7 · Datasets

8 · Talks and podcasts

9 · Landscape

Perspectives and surveys

Companies and labs

Related lists


Keeping the list current

A daily scan finds new work from robotics lab blogs, arXiv cs.RO, Hugging Face, GitHub and talks, vets it against CONTRIBUTING.md, verifies every number against the live page, and opens one pull request that a maintainer must approve; on a day with nothing new it changes nothing. SCAN.md describes the sources, the bar, the state kept in .scanner/, and what it needs to run: one repository secret, CLAUDE_CODE_OAUTH_TOKEN, two optional Slack secrets (SLACK_BOT_TOKEN, SLACK_CHANNEL_ID), Actions allowed to open pull requests, and a ruleset on main that requires an approving review.

Contributing

PRs welcome. Keep the bar high: something to run or read, a one-line why, who judged success and over how many trials, the licence, and a verified URL. See CONTRIBUTING.md. A great list is as much about what it leaves out.

License

CC0

To the extent possible under law, BenchFlow and contributors have waived all copyright and related rights to this list and its annotations (CC0 1.0). The linked resources remain under their own licences.

About

A curated, annotated list of agents that act in the physical world: robot harnesses for coding agents, robot models, world models, benchmarks, simulators and datasets. Maintained by BenchFlow.

Resources

Contributing

Stars

3 stars

Watchers

0 watching

Forks

Releases

Packages

Contributors