A curated, annotated list of agents that act in the physical world, with an emphasis on coding agents and LLMs as robot policies ("agent as a policy"): the harnesses that connect them to robots, the models and world models they run on, the benchmarks that score them, and the simulators, assets, perception tools and datasets underneath.
Maintained by BenchFlow · Join our Discord · Sister list: Awesome Agent Evals
Most lists in this area are link dumps. This one follows the rules of Awesome Agent Evals: every entry says what the thing is and why it belongs, URLs are checked, numbers are quoted verbatim from the linked source, dates are given, and dead projects are pruned or flagged. Robot results add two questions a reader needs answered: who judged success (the simulator's state, a held-out checker, a human, or the agent itself) and how many trials ran. Licences that restrict use are flagged too, because robot work often inherits non-commercial scenes, assets and weights.
"Agent as a policy" here means the agent decides what the robot does next, by calling robot tools, writing and running control code, or calling a learned controller, and is scored on what happens in the world rather than on a plan written in text.
212 annotated entries, plus a starter set and a map of companies; every URL was checked on 2026-10-21. Markers: 🆕 = released, or given a major new version, in 2025–2026 ·
- ⭐ Start here
- 1 · Robot harnesses for coding agents
- 2 · Robot reasoning models and VLAs
- 3 · World models
- 4 · Benchmarks and evaluations
- 5 · Simulators and asset libraries
- 6 · Perception tools
- 7 · Datasets
- 8 · Talks and podcasts
- 9 · Landscape
- Keeping the list current
- Contributing
- License
- Robot-Use Agents — Phillip Isola (MIT) · essay, 2026-09-07 — The case for general-purpose agents that use robots the way computer-use agents use computers: "Think of it as Claude acting as the puppeteer of a robot body."
- Code as Policies — Liang et al. (Google) · paper, 2022-09-16 — Language models writing robot policy code over perception and control APIs: the idea most coding-agent harnesses below build on.
- Claude plays robotics — Anthropic · study, 2026-07-09 — Twelve models at four levels of control, from raw actions to supervising RL. Direct control mostly fails, which is why harness design matters.
- CaP-X — NVIDIA, UC Berkeley, Stanford, CMU · benchmark, 2026-03-23 — The code-as-policy benchmark for coding agents in manipulation, with a training recipe on top.
- Gemini 4 Argon for Robotics — Google · blog + API, 2026-10-13 — A frontier model used as a robot agent end to end: self-improvement on a real arm, a simulated twin from one photo, and a curriculum that fine-tunes a VLA.
- Can Jev Nav? — Dimensional · benchmark, 2026 — Real-time navigation scored by SPL and cost per run, comparing coding agents, a typed "System One" model and a skill-level agent.
- An Unexpected Robot Policy — RoboProbe and RoboDojo teams · paper, 2026-09-21 — A frontier LLM ranked above every public policy on RoboDojo in simulation, and a real-robot test halted for unsafe actions.
- Refuses in Chat, Complies with Arms — Robocurve and UCLA · paper, 2026-10-03 — Why text-only safety evaluations do not transfer: refusal drops once the model controls a robot.
Software that connects a coding agent (Claude Code, Codex and others) or another LLM agent to a robot or a simulator: what the agent sees, which actions it may call, and who checks the result.
- Robo Use — BenchFlow — https://robouse.ai · code https://github.com/benchflow-ai/robouse · PyPI https://pypi.org/project/robouse/ · docs https://robouse.ai/docs/ · suites https://robouse.ai/docs/suites/ · harness + benchmark, 0.6.0 2026-10-21 — "Agent as a policy for embodied agents": it turns robot benchmarks into tasks that a harness such as Claude Code or Codex solves by driving a simulated robot from its shell with the
robocommand, and "An episode server runs the simulator, enforces the step budget, records a video and scores the final physical state", so the agent never grades itself. "robouse 0.6.0 bundles 790 tasks in 34 suites" (LIBERO, LIBERO-PRO, robosuite, Meta-World, ManiSkill3, RoboCasa, MyoSuite, dm_control and more); VLA baselines run behind the same server, and trial pages are public at robouse.ai/runs. 0.5.0 added a harness for Dream Labs' Dream-1 (§3a); 0.6.0 adds harnesses for NVIDIA's ASPIRE workflow (§1c), Kinex (§1a), TypeSafe's Jev (§2a) and Gemini CLI. Apache-2.0. 🆕⚠️ Built by BenchFlow, which maintains this list. - Kinex — Georgia Tech and Harvard contributors — https://stacx-lab.github.io/kinex/ · code https://github.com/stacx-lab/kinex · harness, v1.0.0 2026-10-15 — "A general harness that connects coding agents to any physical robot", built on the Pi coding-agent SDK with three layers (devices, tools, skills) and ROS 2 support. Evaluated zero-shot on 132 LIBERO, RoboTwin and RoboDojo tasks with each suite's native success criteria: "Kinex raises success from 72.7% to 78.0% at 42.9% lower cost than Codex + Bare action API." Episodes and agent traces are on Hugging Face (§7). Robo Use (§1a) added a harness for it in 0.6.0. Apache-2.0. 🆕
⚠️ Released 2026-10-15, so few outside users yet; results are the authors' own. - DimOS — Dimensional — https://github.com/dimensionalOS/dimos · evals docs https://docs.dimensional.org/usage/evals/ · robot OS, 2024-10-19 — "The Agentive Operating System for Physical Space": a Python robot OS (no ROS required) for quadrupeds, humanoids, arms and drones whose skills agents reach through a CLI and an MCP server; "The dimos CLI manages the full lifecycle — run blueprints, inspect state, interact with agents, and call skills via MCP." It ships its own agent-evaluation harness,
dimos evals, and powers Can Jev Nav? (§4a). Apache-2.0. 🆕⚠️ "Pre-Release Beta". - Inspect Robots — Robocurve — https://github.com/robocurve/inspect-robots · paper https://arxiv.org/abs/2610.06306 · docs https://docs.inspectrobots.org/ · eval harness, paper 2026-10-05 — "Evaluating the Capabilities and Safety of Embodied AI": an Inspect-style framework where any policy, an LLM agent or a VLA, runs on real arms or humanoids (YAM, Franka, SO-101, G1 and others) or in simulators, with logged transcripts and grader scores; StationeryBench and RoboHarm are built on it. "Inspect Robots has seen significant early uptake, receiving nearly 100,000 downloads in the three months since its release." MIT. 🆕
⚠️ Alpha; the download count is the authors' own. - RPent — RLinf; paper from Tsinghua University and others — https://github.com/RLinf/RPent · paper https://arxiv.org/abs/2607.08448 · harness, paper 2026-07-09 — "Agentic Infrastructure for the Physical World": Claude Code, Codex or an API model plans, and frozen VLAs such as π0.5 become retryable primitives guided by memory (the paper calls it Harness VLA). Its leaderboard covers LIBERO, LIBERO-PRO, RoboCasa365 and RoboTwin; with Codex and GPT-6 Astra it reports "92.63% Overall (741/800) across all eight LIBERO-PRO suites". Apache-2.0. 🆕
⚠️ Leaderboard results are self-reported. - PyRUA-Lean — Peking University, NUS, NVIDIA and others — https://github.com/DAGroup-PKU/PyRUA-Lean · paper https://arxiv.org/abs/2610.01939 · harness, paper 2026-10-01 — The robot is a Python object,
robo, and the agent works through onepython(code)tool that composes primitives (VLAs included) instead of making one tool call per action; "it raises success from 63.1% to 71.7% over tool calling with the same GPT-6 Astra planner and primitives". Apache-2.0. 🆕⚠️ Simulation only so far. - Show-Harness — NUS Show Lab — https://github.com/showlab/Show-Harness · paper https://arxiv.org/abs/2609.10522 · harness, paper 2026-09-09 — "Just a VLM Agent Can Play Robots": VLMs reason over discrete semantic action units that per-robot interpreters turn into motion, so frontier VLMs run zero-shot and small open VLMs can be adapted "with just a few GPU-hours of fine-tuning"; LoRA adapters and data are released. Apache-2.0. 🆕
- RoboHarness — Zhejiang University — https://github.com/BinceQu/RoboHarness · paper https://arxiv.org/abs/2610.14018 · harness, paper 2026-09-29 — A visual-geometric control panel for an LLM agent (clicked keypoints, optical-flow tracking, depth back-projection, geometric constraints), with Claude Code and Codex harnesses for BEHAVIOR Challenge 2025 tasks: "With RoboHarness, Qwen3.8-Flash-Next outperforms state-of-the-art VLA models on BEHAVIOR Challenge 2025 tasks." MIT. 🆕
⚠️ BEHAVIOR data needs its own non-commercial licence (§4b). - RobotUse — KAIST and Seoul National University — https://github.com/robotuse-team/RobotUse · paper https://arxiv.org/abs/2610.04929 · harness, paper 2026-10-04 — Splits the work: the agent picks targets and poses visually while the backend handles geometry, motion planning and control, with subagents per subgoal and a persistent playbook of lessons; "On RoboLab, RobotUse achieves 45% task success, outperforming CaP-X by 6.7 percentage points". Apache-2.0. 🆕
⚠️ The released code runs only in RoboLab (Isaac Sim); real-robot results are in the paper only. - OpenRUA — UCL and CMU — https://github.com/terminalworld/OpenRUA · paper https://arxiv.org/abs/2610.02459 · harness, paper 2026-10-01 — "Robot-Use Agents Are Zero-Shot Visuomotor Policies": an off-the-shelf Claude Code or Codex gets only terminal access to the robot's native ROS 2 interface and its documentation, with no custom primitives and no VLA; "With Claude Code powered by Claude Opus 5, OpenRUA achieves success rates of 99.0% on CaP-Bench and 87.0% on LIBERO-PRO". Apache-2.0. 🆕
⚠️ Simulated benchmarks; few outside users yet. - PhyAgentOS — Sun Yat-sen University, Peng Cheng Laboratory, X-Era Lab — https://github.com/PhyAgentOS/PhyAgentOS-core · paper https://arxiv.org/abs/2607.16636 · robot agent OS, paper 2026-07-18 — A self-evolving operating system for embodied agents that separates cognitive planning from physical execution, passes state between agent and runtime as files, and adds a session verifier, memory of past runs, layered safety and robot integrations packaged as skills; "validated on 19+ simulated and physical embodiments". MIT. 🆕
- ROSClaw — Tongji University and SRIAS — https://github.com/ros-claw/rosclaw · runtime, 2026-03-16 — A governed runtime between agents (Codex, Claude Code, OpenClaw, VLAs) and bodies (ROS 2, MCP, simulators, vendor SDKs) that checks policy and authorisation before each action and writes an execution receipt for it. MIT. 🆕
⚠️ "ROSClaw is alpha software"; its maturity table marks real hardware as experimental. - Strands Robots — AWS Strands Labs — https://github.com/strands-labs/robots · agent tools, v0.5.3 2026-10-07 — "Control, simulate, and train robots with natural language": a
Robot()tool for Strands Agents that returns a MuJoCo simulation or a real robot and runs LeRobot, GR00T and Cosmos policies. Apache-2.0. 🆕⚠️ Strands Labs is AWS's organisation for experimental projects. - AgenticROS — AgenticROS project — https://github.com/agenticros/agenticros · MCP server + plugin, 2026-03-01 — An agent-agnostic layer for ROS 2 robots: an OpenClaw plugin and one MCP server for Claude Code, Codex CLI, Hermes, Antigravity and Gemini CLI. NVIDIA's Isaac ROS 5.0 post (2026-09-22) points to it: "AgenticROS, an open source project sponsored by 3D perception technology company RealSense, connects Isaac ROS". Apache-2.0. 🆕
- RAI — Robotec.ai — https://github.com/RobotecAI/rai · paper https://arxiv.org/abs/2505.07532 · agent framework, paper 2025-05-12 — A vendor-agnostic multi-agent framework on ROS 2 with speech, simulation and its own benchmark: "rai_bench: Benchmarking suite for RAI. Test agents, models, tools, simulators, etc." Apache-2.0. 🆕
- NVIDIA Agent Skills — NVIDIA — https://github.com/NVIDIA/skills · Isaac Lab skills https://isaac-sim.github.io/IsaacLab/v3.0.0-EA/source/developer-tools/agent_skills.html · skills, 2026-02-25 — Signed skills that teach coding agents robotics, simulation and CUDA work, each published skill carrying its own evaluation set; Isaac Lab 3.0 ships 17 user-facing skills, and "User-facing skills must also include an evaluations.md file with at least three representative scenarios." Apache-2.0 and CC BY 4.0. 🆕
- Model Hardware Standard — Anthropic, with HHMI Janelia — https://www.anthropic.com/news/model-hardware-standard-research-preview · standard preview, 2026-08-27 — "a shared specification for AI agents to safely operate physical devices": standard device drivers with discovery and read/write primitives, model-agnostic and reachable over MCP, for microscopes, liquid handlers and robot arms. 🆕
⚠️ Gated research preview with an application form; the spec is not public, and the post describes the preview as coming "ahead of making the standard open source". - Qwen-MM-Plugins — Qwen — https://github.com/QwenLM/Qwen-MM-Plugins · plugins, MHS host added 2026-09-20 — Plugins that make agent harnesses (Claude Code, Codex, Qwen Code, OpenClaw) multimodal; its
mhsplugin is an open MHS host with a fixed six-tool surface, host-side safety limits and an emergency stop, and "Operates real hardware — cameras, sensors, lamps, arms, lab equipment — through Model Hardware Standard adapters". Apache-2.0. 🆕⚠️ Device adapters come from hardware owners under the gated MHS preview. - ROS MCP Server — Robot MCP — https://github.com/robotmcp/ros-mcp-server · MCP server, v3.1.2 2026-09-23 — "ROS-MCP-Server connects large language models (such as Claude, GPT, and Gemini) to robots": any MCP client reaches topics, services, actions and parameters on ROS 1 or ROS 2 robots through rosbridge. Apache-2.0. 🆕
⚠️ Needs rosbridge; no evaluation of its own.
- Code as Policies — Liang et al. (Google) — https://arxiv.org/abs/2209.07753 · paper, 2022-09-16 — "Language Model Programs for Embodied Control": code-writing LLMs, prompted with a few examples, write robot policy code that processes perception outputs and parameterises control primitives. The idea behind CaP-X (§4a) and most harnesses above.
- VoxPoser — Huang et al. (Stanford) — https://arxiv.org/abs/2307.05973 · paper, 2023-07-12 — LLMs compose 3D value maps that a motion planner turns into dense 6-DoF end-effector trajectories, for open-set instructions and objects without predefined motion primitives.
- ASPIRE — NVIDIA GEAR with UMich, UIUC, UC Berkeley, CMU — https://research.nvidia.com/labs/gear/aspire/ · paper https://arxiv.org/abs/2607.00272 · code https://github.com/NVlabs/ASPIRE · paper + code, 2026-06-30 — Agentic skill discovery: a coding agent writes and refines control programs from multimodal execution traces, saves verified fixes to a growing skill library, and uses evolutionary search over tasks and programs; ASPIRE "is a new type of continual learning" in which training means refining skills rather than gradient descent. "On 10 LIBERO-Pro tasks, success rises from 15% (GPT-6 Luna) to 99% (Claude Opus 5.5)"; it also reports robosuite and BEHAVIOR-1K results and early sim-to-real transfer. Robo Use (§1a) added a harness for its workflow in 0.6.0. Apache-2.0. 🆕
⚠️ The authors "have not verified that smaller or weaker LLMs can sustain the debugging loop". - ENPIRE — NVIDIA, CMU, UC Berkeley — https://research.nvidia.com/labs/gear/enpire/ · paper https://arxiv.org/abs/2606.19980 · code https://github.com/NVlabs/ENPIRE · paper + code, 2026-06-18 — "Agentic Robot Policy Self-Improvement in the Real World": a coding-agent harness that resets and verifies the environment, improves the policy, runs rollouts on one or several robots and evolves its approach, to "autonomously train a policy to achieve a 99% success rate on challenging, dexterous manipulation tasks" (CoRL 2026). Apache-2.0. 🆕
⚠️ Real-robot use needs a calibrated station. - SimEX — Amazon FAR, UT Austin, CMU — https://robo-simex.github.io/ · paper https://arxiv.org/abs/2609.38982 · paper, 2026-09-30 — "Simulation-Integrated Robotics AutoResearch": a coding agent builds a robot toolbox through probe-and-optimise cycles in simulation, then adapts toolbox and simulator together from a few real trials, "without any demonstration and with only 10 minutes of real-robot interaction". 🆕
⚠️ No code released. - SEAR — Luma AI — https://sear.bot/ · paper https://arxiv.org/abs/2610.18792 · paper + benchmark, 2026-10-13 — "Self-Evolving Agentic Robotics": agents act as the policy, write the code, or fine-tune a VLA, with controls that separate retained experience from retries, luck and privileged simulator state, "Across seven frontier language models, 330 simulated and five physical tasks". 🆕
⚠️ Code "coming soon": the repository holds only a README.
- SayCan — Ahn et al. (Google) — https://arxiv.org/abs/2204.01691 · paper, 2022-04-04 — "Do As I Can, Not As I Say: Grounding Language in Robotic Affordances": an LLM proposes steps and the value functions of pretrained skills ground them in what the robot can do in its current state.
- Gemini 4 Argon for Robotics — Google (Kanishka Rao) — https://developers.googleblog.com/gemini-4-argon-for-robotics/ · blog + API, 2026-10-13 — Google's frontier model as a robot agent. On ERQA, "Using pure reasoning, Argon scores 87%, outperforming every other frontier model"; it improves an SO-101 pick-and-place over three iterations (58.9 s, 22.9 s, 17.4 s), and from a single RGB photo builds a calibrated MuJoCo twin, defines a 77-task curriculum and exports 4,500+ episodes to fine-tune a VLA, "boosting real-world physical robot Success Rate from 56% to 75%". Paid Gemini API tier (
gemini-4-argon) and Antigravity. 🆕⚠️ Self-reported; the harness and tasks are not released. - Gemini Robotics-ER 2 — Google DeepMind — https://deepmind.google/models/gemini-robotics/embodied-reasoning/ · docs https://ai.google.dev/gemini-api/docs/robotics-overview · model + API, 2026-07-30 — An embodied-reasoning model for spatial reasoning, planning and multi-robot orchestration through function calling; "It then hands off motor execution to any given lower level vision-language-action (VLA) model." Public preview in the Gemini API (
gemini-robotics-er-2-preview). 🆕⚠️ Closed model; benchmark numbers on the page are self-reported. - Glimpse — Elorian — https://elorian.ai/blog/introducing-glimpse · model + API, 2026-10-16 — "a new category of visual thinking model: a system that reasons about the physical world from what it sees", trained to call multimodal tools and coordinate subagents; robotics is one target: one demo labels a robot episode and "splits the episode into six pick-and-place events", another plans and checks the work of a VLA. "Glimpse is available through API and our chat UI today": "Input tokens cost $0.60 per million and output tokens $3.00 per million". 🆕
⚠️ Benchmarks are self-reported; no weights (it was "post-trained on an open weights base model"). - Jev — TypeSafe — https://docs.typesafe.ai/introduction · model + docs, undated — "TypeSafe's flagship model and the first System One model": instead of generating text, it answers typed questions (Choice, Score, Noul) about a state with structured values that code can act on directly. Can Jev Nav? (§4a) tests it as a real-time navigation policy, and Robo Use (§1a) added a harness for it in 0.6.0. 🆕
⚠️ The docs give no date, access route or pricing. - UnifoLM-ER-1 — Unitree — https://huggingface.co/unitreerobotics/UnifoLM-ER-1 · open model, 2026-09-11 — An open 4B embodied-reasoning VLM built on Qwen3-VL-4B, released with Unitree's whole-body humanoid model UnifoLM-WLA-1.0, a "next-generation general-purpose humanoid robot foundation model with 6B parameters". Apache-2.0, ungated. 🆕
- FIBER — Amazon FAR with CMU and Stanford (Lin, Shi, Liu) — https://fiberaa.github.io/ · paper https://arxiv.org/abs/2610.24675 · code https://github.com/Yutang-Lin/FIBER · paper, 2026-10-17 — "Action-Aligned Latent Planning for Versatile Humanoid Loco-Manipulation": one perceptive planner, trained offline on the rollouts of separately trained skill experts, outputs latent commands for a frozen behaviour foundation model on a Unitree G1 with Dex3 hands; the planner replans at 5 Hz while the frozen model closes the loop at 50 Hz, for box pickup, door opening and object grasping from onboard depth. Isaac Sim (§5a) judges success by lift and hold time, or by passing the doorway: "Across 128 diverse variations per task in simulation, FIBER achieves 42.0% mean success, more than tripling projected latent replay (11.6%) and substantially outperforming kinematic planning (3.5%)." On the real robot it "succeeds in 6/10 box-pickup trials and 4/10 door-opening trials", and 2/10 grasps. A playground runs the planner and MuJoCo in the browser. Repository Apache-2.0. 🆕
⚠️ "Code coming soon.": the repository holds only the paper and figures; the playground's ONNX models are published with no licence, and it runs a distilled planner, "so success rates may be lower than reported". - RT-2 — Brohan et al. (Google DeepMind) — https://arxiv.org/abs/2307.15818 · paper, 2023-07-28 — "Vision-Language-Action Models Transfer Web Knowledge to Robotic Control": co-fine-tunes a web-scale vision-language model on robot trajectories and web tasks so one model maps robot observations to actions and carries over semantic reasoning from web data.
- openpi — Physical Intelligence — https://github.com/Physical-Intelligence/openpi · API beta https://www.pi.website/blog/api-beta · models + code, 2024-10-21; π0.5 2025 — Open weights for π0, π0-FAST and π0.5 with fine-tuned DROID, ALOHA and LIBERO checkpoints and a websocket policy server; "we provide base model checkpoints, pre-trained on 10k+ hours of robot data". A hosted fine-tuning and inference API entered beta on 2026-10-13. Apache-2.0. 🆕
⚠️ The API beta is by request, and the model behind it is not named. - GR00T N1.7 — NVIDIA — https://github.com/NVIDIA/Isaac-GR00T · weights https://huggingface.co/nvidia/GR00T-N1.7-3B · model, 2026-02-25 — A 3B foundation model for generalist robots: a Cosmos-Reason2-2B backbone with a flow-matching action head, pretrained with added human video. 🆕
⚠️ Code is Apache-2.0, but the model card says "This model is released under the NVIDIA Open Model License Agreement." while the README calls it Apache 2.0; the backbone is gated. - OpenVLA-OFT — Stanford — https://openvla-oft.github.io/ · paper https://arxiv.org/abs/2502.19645 · OpenVLA https://arxiv.org/abs/2406.09246 · paper + models, 2025-02-27 — A fine-tuning recipe (parallel decoding, action chunking, continuous actions, L1 loss) for the open 7B OpenVLA, "significantly boosting OpenVLA's average success rate across four task suites from 76.5% to 97.1%". MIT. 🆕
- SmolVLA — Hugging Face — https://huggingface.co/blog/smolvla · paper https://arxiv.org/abs/2506.01844 · model, 2025-06-02 — "a compact (450M), open-source Vision-Language-Action model for robotics that runs on consumer hardware", trained on community LeRobot data with asynchronous inference; the baseline LeRobot ships with its benchmark environments (§4f). Apache-2.0. 🆕
- MolmoAct 2 — Ai2 — https://allenai.org/blog/molmoact2 · paper https://arxiv.org/abs/2605.02881 · code https://github.com/allenai/molmoact2 · model, 2026-05-04 — An open action-reasoning VLA (a Molmo2-ER backbone with a flow-matching action expert) released with checkpoints, a bimanual dataset and training code; "Overall, MolmoAct 2 averages 87.1% success, compared with 48.4% for MolmoBot and 45.2% for π0.5." Code Apache-2.0. 🆕
⚠️ The weights state no licence; the comparison is Ai2's own. - LingBot-VLA 2.0 — Robbyant (Ant Group) — https://github.com/Robbyant/lingbot-vla-v2 · paper https://arxiv.org/abs/2607.06403 · model, 2026-07-07 — An open VLA trained across many robot configurations and egocentric video: "we revamp the data processing pipeline and curate around 60,000 hours of data for pretraining". Apache-2.0. 🆕
- Xiaomi-Robotics-1 — Xiaomi — https://github.com/XiaomiRobotics/Xiaomi-Robotics-1 · paper https://arxiv.org/abs/2607.15330 · model, 2026-07-16 — A Qwen3-VL plus diffusion-transformer VLA pretrained on robot-agnostic UMI data, "training on over 100k hours of real-world manipulation trajectories collected via UMI devices", then post-trained on data from several robots. Apache-2.0. 🆕
- Gemini Robotics 2 — Google DeepMind — https://deepmind.google/blog/gemini-robotics-2-brings-whole-body-intelligence-to-robots/ · SDK https://github.com/google-deepmind/gemini-robotics-sdk · model, 2026-07-30 — A whole-body, dexterous VLA shown on Apptronik's Apollo 2 humanoid and a Franka Duo, and On-Device 2, a VLA that runs locally and adapts to a new robot body with a few hours of data; the Safari SDK covers serving, evaluation and fine-tuning. 🆕
⚠️ "Our VLA and On-Device models are available to early-access partners"; most SDK functions need the Trusted Tester programme. - DemoMimic — Stanford (Sharma, Sahoo, Huang, Fei-Fei Li, Wu, Sadigh, Bohg) — https://demomimic.github.io/ · paper https://arxiv.org/abs/2609.01938 · paper, 2026-09-01 — "One Demonstration, Many Objects": a dexterous multi-fingered-hand policy learned from one human demonstration by attending to object geometry near the contact points, with contact-centric rewards for precise contact and sim-to-real consistency; on real hardware, "DemoMimic achieves 71% success across 16 objects, four tasks, and two robot-hand embodiments". 🆕
⚠️ Code "coming soon".
Primers: World Action Models: A Survey (Shen … Wang, 2026-06-18), which separates world-action models from world models, video generators and VLAs and says "This survey gives the field a common account", and From World Models to World Action Models: A Concise Tutorial for Robotics (Zhang … Zhang, 2026-07-01), which compares World Labs, JEPA and Cosmos. 🆕
- SuperMimic — UC Berkeley (Liao, Shao, Long, Zakka, Horowitz, Abbeel, Sreenath) — https://supermimic.github.io/ · paper https://arxiv.org/abs/2610.28480 · paper, 2026-10-20 — "Scaling Humanoid Whole-Body Control with One Differentiable World Model": a single Unitree G1 motion tracker for the public BONES-SEED motion set, trained by backpropagating through a world model fitted to logged MuJoCo Warp (§5a) transitions, so the simulator is never differentiated; the world model and the policy train together in one loop, and the policy runs on a real G1 (CoRL 2026 oral). "In 3 hours on a single GPU, SuperMimic surpasses the return of a state-of-the-art motion tracker baseline trained for 7 days on 128 GPUs" (SONIC), and it "fails on only 25 of 129,514 benchmark motions, versus 179 failures for the baseline". 🆕
⚠️ No code or weights; the in-browser demo is "All rights reserved" and licensed only for personal, non-commercial evaluation; failures are counted in simulation, one playback per motion, by SuperMimic's own termination rule, and "Benchmark motions are training motions for both methods, so the counts measure coverage, not generalization." - 3D Point World Models (3DPWM) — Oregon State University (Peri, Nguyen, Kim, Li, Lee) — https://pvskand.github.io/3dpwm/ · paper https://arxiv.org/abs/2607.00148 · code https://github.com/pvskand/3dpwm · paper, 2026-06-30 — "3D Point World Models: Point Completion Enables More Accurate Dynamics Learning": the object's partial point cloud is completed first, a Point Transformer V3 predicts how the points move under the robot's actions, and an MPPI planner plans with the rollouts, on ManiSkill (§4b), MimicGen and DexMimicGen tasks with a Franka gripper and a dexterous hand, and on a real Franka. "With GT completion, 3dPwm achieves 88% success on PickCube, decreasing to 78% on StackCube and 60% on PickYCB as task complexity increases." (full text) Repository MIT. 🆕
⚠️ The repository's code section reads "Coming soon.", with no weights; the quoted rates use ground-truth completion, which the paper says learned completion lowers on the harder tasks. - Dream-1 — Dream Labs — https://dreamlabs.ai/research/dream-1/ · API https://platform.dreamlabs.ai · client https://pypi.org/project/dreamlabs/ · model + API, 2026-10 — "a World Action Model that controls robots it has never seen, performs tasks it was never shown, and runs from $3/hour": it predicts video and actions together, takes camera images, robot state and a typed task, and returns joint or end-effector targets for bimanual parallel-gripper arms. Evaluated on 27 unseen simulated robots relative to the YAM it was trained on, and in real-robot clips; fine-tuning is offered as a service. Robo Use (§1a) added a harness for it in 0.5.0. Client Apache-2.0. 🆕
⚠️ Hosted only, no weights; a "research beta" that "has not been validated for safety-critical use". - DreamZero — NVIDIA GEAR — https://github.com/dreamzero0/dreamzero · paper https://arxiv.org/abs/2602.15922 · model, 2026-02-17 — "World Action Models are Zero-shot Policies": a world-action model on the Wan2.1-I2V-14B video model that predicts video and actions together; "This results in over 2x improvement in generalization to new tasks and environments compared to state-of-the-art VLAs". Code Apache-2.0. 🆕
⚠️ The DROID weights are CC BY-NC 4.0. Not related to Dream Labs. - Cosmos 3 — NVIDIA — https://nvidianews.nvidia.com/news/nvidia-launches-cosmos-3-the-open-frontier-foundation-model-for-physical-ai · policy https://huggingface.co/nvidia/Cosmos3-Nano-Policy-DROID · models, 2026-05-31 — An open model family for physical AI covering reasoning, world generation and actions; the Nano-Policy-DROID checkpoint is a world-action policy whose card reads "Given language instructions and visual observations from the DROID robot platform, generate robot action trajectories". OpenMDW-1.1, ungated. 🆕
- FLUX 3 Action — Black Forest Labs — https://bfl.ai/models/flux-3-action · weights https://huggingface.co/black-forest-labs/flux-3-action-so101 · code https://github.com/black-forest-labs/flux-action · model, 2026-09-22 — "FLUX 3 Action is an open weights 7B world action model." It generates action chunks together with the next video frames, and its SO-101 checkpoint runs through LeRobot. Code Apache-2.0. 🆕
⚠️ The weights are gated under the non-commercial FLUX Kommunity License. - V-JEPA 2 — Meta FAIR — https://github.com/facebookresearch/vjepa2 · paper https://arxiv.org/abs/2506.09985 · model, 2025-06-11 — A self-supervised video world model whose action-conditioned variant plans pick-and-place zero-shot on Franka arms after post-training "using less than 62 hours of unlabeled robot videos from the Droid dataset". MIT. 🆕
- RoboJEPA — FAIR at Meta — https://github.com/facebookresearch/robo_jepa · paper https://arxiv.org/abs/2610.10515 · model, 2026-10-07 — A latent world model trained across many robot embodiments, with compute scaling laws and zero-shot planning toward goal images on real robots; "RoboJEPA, at 8B parameters, is the largest JEPA predictor model trained to date". 🆕
⚠️ CC BY-NC-SA 4.0.
- Evaluating Gemini Robotics Policies in a Veo World Simulator — Google DeepMind — https://arxiv.org/abs/2512.10675 · site https://veo-robotics.github.io/ · paper, 2025-12-11 — An action-conditioned, multi-view Veo model used to evaluate robot policies and to edit scenes for generalisation and safety tests, checked against "1600+ real-world evaluations of eight Gemini Robotics policy checkpoints and five tasks for a bimanual manipulator". 🆕
⚠️ The simulator is not released. - RoboWorld — KAIST — https://arxiv.org/abs/2607.01060 · paper, 2026-07-01 — "Fast and Reliable Neural Simulators for Generalist Robot Policy Evaluation": a video world model rolls out policies and a VLM scores task progress, which is reported to "align strongly with real-world robot evaluation across tasks and environments, achieving Pearson's r = 0.989". 🆕
⚠️ Release of code and weights not verified. - ProWorld — Sun Yat-sen University, MBZUAI, Meituan and others — https://arxiv.org/abs/2610.15294 · paper, 2026-10-11 — A multi-view world model that feeds robot state back to the policy for closed-loop evaluation; "it attains a Pearson correlation of 0.95 and a Spearman correlation of 0.94 between generated and real success rates". 🆕
⚠️ Release of code and weights not verified. - GE-Sim 2.0 — AgiBot — https://github.com/AgibotTech/GE-Sim-V2 · paper https://arxiv.org/abs/2605.27491 · model, 2026-05-26 — A closed-loop video world simulator for manipulation with a robot-state module and a success judge; "GE-Sim 2.0 tops the public WorldArena leaderboard at only 2B parameters". 🆕
⚠️ Most code and data are CC BY-NC-SA 4.0. - Genie 3 — Google DeepMind — https://deepmind.google/blog/genie-3-a-new-frontier-for-world-models/ · model, 2025-08-05 — An interactive world model that turns a prompt into a world an agent can move through: "Genie 3 can generate dynamic worlds that you can navigate in real time at 24 frames per second"; DeepMind tested it with its SIMA agent. 🆕
⚠️ Research preview only (Project Genie); no robot interface.
Agents in control of a robot, simulated or real. For each, the entry says who judges success and how many trials ran when the source says so.
- Embodied Playground — HKU, Sharpa, Tsinghua, Stanford, UNC Chapel Hill, MIT (Chen, Zhou, He … Ma) — https://winniechen2002.github.io/Agent-bench-website/ · paper https://arxiv.org/abs/2610.27039 · code https://github.com/Winniechen2002/Embodied-Playground · benchmark, 2026-10-19 — "A Long-Horizon Agentic Robotics Benchmark for Physical, Strategic, and Spatial Reasoning": a foundation-model agent reads the goal, writes control code and acts through a describe, observe, step and finish interface on 410 environments (spring launchers, Tower of Hanoi, interlocking Kongming locks, cubes, household tasks), at access levels from RGB and proprioception only (L0) to read-only simulator state (L3), and the environment itself checks the goal, the task's rules and physical validity. EP-100 takes 100 tasks in five suites, four conditions each; the Launch suite runs on MuJoCo (§5a), the others on Isaac Lab or ManiSkill (§4b). "GPT-6-Astra achieves 13.80% Pass@1 and 19.94% Pass@2 on EP-100", and "Opus-5.5-extra achieves 29.22% Pass@1 and 35.85% Pass@2 using 683 valid attempts." (full text) Code Apache-2.0; original documentation and assets CC BY 4.0. 🆕
⚠️ The paper runs GPT-6-Astra "with high reasoning effort and L3 privileged observations" and does not state the Claude run's level; most suites need Isaac Sim and an NVIDIA GPU; bundled third-party assets keep their own terms. - Can Jev Nav? — Dimensional (Pomichter, Dasyam, Ventura) — https://research.dimensional.org/system-one-navigation/ · runs https://research.dimensional.org/system-one-navigation/tasks/ · code https://github.com/dimensionalOS/dimos · benchmark, 2026 — "Have System One “LLMs” finally broken the real-time robotics frontier?": 327 start-to-object navigation tasks for a quadruped in Habitat-Sim homes (HSSD), 1,962 recorded runs scored with SPL and SoftSPL, comparing coding agents, TypeSafe's Jev and Dimensional's own skill-level agent, with cost per run. Mean SPL: Dimcode with navigation skills 0.743, Astra as a coding agent 0.522 ($1.610 per run), Jev 0.263 ($0.081). 🆕
⚠️ No licence stated for the tasks; HSSD scenes are CC BY-NC 4.0 (§5c); the top row had navigation skills the other agents lacked. - CaP-X — NVIDIA, UC Berkeley, Stanford, CMU — https://capgym.github.io/ · paper https://arxiv.org/abs/2603.22435 · code https://github.com/capgym/cap-x · benchmark, 2026-03-23 — "A Framework for Benchmarking and Improving Coding Agents for Robot Manipulation": models write Python over perception and control primitives on LIBERO-PRO, robosuite and BEHAVIOR tasks, in simulation and on real robots, at levels of abstraction from privileged state to raw perception, across 12 models; "today's best frontier models can directly generate executable robot control code and achieve over 30% average success". It adds a training-free agent (CaP-Agent0) and RL with verifiable rewards (CaP-RL). MIT. 🆕
⚠️ BEHAVIOR and Isaac Sim terms apply to some tasks; SAM 3 weights are gated. - RLE-Bench — Harvard, Georgia Tech — https://rle-bench.github.io/ · paper https://arxiv.org/abs/2609.34210 · code https://github.com/RLE-Bench/RLE-Bench · benchmark + leaderboard, 2026-09-14 — "A Qualifying Exam for Coding Agents as Robot Learning Engineers": robot-learning tasks across four workflows (interactive control, policy learning, perception and estimation, mechanical design), run by Claude Code, Codex, Antigravity and Grok Build and scored by task metrics combined into an RLE Index; "GPT-6 Astra and Claude Fable 5.1 lead the leaderboard, with visual grounding emerging as a key differentiator." (full text) MIT. 🆕
- EmbodiedSWE-Bench — Yale, ByteDance Seed, Princeton, Stanford, CMU, UW, UCLA — https://embodiedswe.github.io/ · paper https://arxiv.org/abs/2609.27308 · code https://github.com/EmbodiedSWE/EmbodiedSWE · benchmark, 2026-09-23 — "Coding Agents for Long Horizon Dexterous Robotics": Codex and Claude Code write solution programs in Isaac Sim and Newton, "6 suites · 28 tasks · 17 embodiments", and hidden graders score physical task progress offline; "GPT-6 Astra passes 0.9 inside two hours; every other curve flattens well below ceiling." Apache-2.0. 🆕
⚠️ The site also reports reward-hacking rates by model. - CodeActionBench — Nanyang Technological University — https://codeactionbench.org/ · paper https://arxiv.org/abs/2609.33807 · code https://github.com/lyhkk/CodeActionBench · benchmark, 2026-09-27 — "Evaluating Agentic Code-as-Policy for Embodied Manipulation": Codex CLI and Claude Code drive RoboTwin 2.0's dual-arm robot over MCP on 25 SAPIEN tasks, judged by a hidden physical verifier; "The strongest configuration, GPT-6 Astra with Codex CLI, solves 22 of 25 tasks at least once in three attempts". MIT. 🆕
⚠️ "Full data (coming soon)". - An Unexpected Robot Policy — RoboProbe and RoboDojo teams (HKU, Tsinghua, UC Berkeley and others) — https://arxiv.org/abs/2609.24170 · code https://github.com/RoboProbe/RoboProbe · paper, 2026-09-21 — "Early Evaluations of GPT-6 Astra on RoboDojo and Beyond": the LLM picks every end-effector target on all 42 RoboDojo simulation tasks; "GPT-6 Astra achieves 22.48% average success rate and 28.97 Score over 2,100 trials, ranking above every public entry." Apache-2.0. 🆕
⚠️ On the real robot, "Testing was halted after the model repeatedly issued physically unreasonable or unsafe actions" (full text). - RobotWorld — Yang et al. — https://robotworldai.github.io/ · paper https://arxiv.org/abs/2610.10409 · code https://github.com/robotworldai/robotworld · benchmark, 2026-10-07 — "Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments": agents drive simulated robots through tools on 84 tasks (manipulation, mobile manipulation, locomotion, driving, aerial) with success checks the agent cannot see; "Astra completes 16 of 84 tasks (19.0%), followed by Opus 5.5 with 13 (15.5%)" (full text). 🆕
⚠️ One episode per model and task; no licence for the code; affiliations not stated. - LIBERO-Agent — Fudan University, Shanghai Innovation Institute — https://arxiv.org/abs/2609.39507 · code https://github.com/dzj441/Libero-Agent · benchmark, 2026-09-30 — General-purpose agents (Codex, Claude Code and others) send native 7-D end-effector actions to a simulated Panda over MCP instead of calling high-level skills, and the simulator checks success. "Results reveal a pronounced reliability gap" on hard and long-horizon tasks; "Among these agents, GPT-6 Astra achieves the strongest overall performance." MIT. 🆕
⚠️ Only the 30-task primary suite is released. - RoboQuest — DeCLaRe Lab, NTU — https://declare-lab.github.io/RoboQuest/ · paper https://arxiv.org/abs/2610.10388 · code https://github.com/declare-lab/RoboQuest · benchmark, 2026-10-07 — Ten RoboCasa365 mobile-manipulation tasks that need searching, inspecting or testing before committing, with models acting as the policy; "Of six frontier models, Gemini 4 Argon and GPT-6 Astra do best and are tied, solving 23.8% and 23.2% of episodes." MIT; demonstrations in §7. 🆕
- BiGym 2.0 — Imperial College London — https://bigym2.github.io/ · paper https://arxiv.org/abs/2610.07594 · code https://github.com/swirl-uk/BiGym2 · benchmark, 2026-10-06 — Unitree G1 household tasks where VLA, imitation and RL baselines meet coding agents that write
policy.py, scored on hidden seeds; "GPT-6 Astra and Claude Opus 5.5 write controllers that beat every demo-driven RL baseline on the nine-task mean". Apache-2.0; agent rollouts in §7. 🆕⚠️ π0.5 still has the best nine-task mean. - AgenticGenPlan — Princeton, FBK, CMU, Cambridge — https://agenticgentamp.github.io/ · paper https://arxiv.org/abs/2609.30233 · code https://github.com/tomsilver/robocode · KinDER https://prpl-group.com/kinder-site/ · paper, 2026-09-24 — "Coding Agents for Generalized Task and Motion Planning Problems": Claude Code and Codex write one program per KinDER physical-reasoning environment (RSS 2026) that must generalise to held-out instances, "(56% to 95% versus 47% for the planners, on the 16 environments where a planner is available)". MIT. 🆕
- Embodied Agent Arena — HKUST (Guangzhou), CUHK, Knowin AI — https://embodied-agent-arena.github.io/embodied-agent-arena/ · paper https://arxiv.org/abs/2610.00854 · benchmark, 2026-10-01 — "Are Frontier VLM Agents Ready to Be Robot Generalists?": cases drawn from many sources across geometry, spatial reasoning, affordance, planning and manipulation, scored on the final state with a minimal harness; "Different models lead; even the strongest manipulation result leaves over half the tasks incomplete." MIT. 🆕
⚠️ The dataset states no licence. - Embodied First Exam — MLL Lab (Kangrui Wang … Manling Li) — https://www.embodied-first-exam.ai/ · code https://github.com/mll-lab-nu/embodied-first-exam · blog https://www.embodied-first-exam.ai/blogs/first-exam/ · benchmark, 2026-10-19 — "Code agents have passed many exams in the digital world. What is their first exam to enter the physical world?": frozen tests from many robot suites (a core set of 449 tasks), answered with a program and marked by replay in a separate sandbox; "GPT-6 Astra, the strongest agent, solves only about a third of the standard and standard-realtime tests." MIT. 🆕
⚠️ One attempt per test. - AA-Robotics v0.1 — Artificial Analysis — https://artificialanalysis.ai/articles/aa-robotics · benchmark, 2026-10-14 — An independent evaluator's first robotics index: models in Codex, Claude Code or Antigravity drive a real Hiwonder arm over HTTP on eight tabletop tasks with partial credit, with time and cost; "GPT-6 Astra places first in AA-Robotics v0.1 with a score of 69". 🆕
⚠️ "A human judge scores the end state"; no code or data; the number of repeats is not stated. - EmbodiedBench — UIUC, Northwestern, Toronto, TTIC — https://embodiedbench.github.io/ · paper https://arxiv.org/abs/2502.09560 · benchmark, 2025-02-13 — Multimodal LLM agents with high- and low-level actions on "a diverse set of 1,128 testing tasks across four environments", from household planning to navigation and manipulation (ICML 2025). 🆕
⚠️ The code repository has no licence file. - Embodied Agent Interface — Stanford, Northwestern, Amazon, MIT — https://embodied-agent-interface.github.io/ · paper https://arxiv.org/abs/2410.07166 · benchmark, 2024-10-09 — Splits embodied decision-making into goal interpretation, subgoal decomposition, action sequencing and transition modelling, with fine-grained error metrics in BEHAVIOR and VirtualHome (NeurIPS 2024 D&B); "o1-preview has the highest task (81.0%) and execution success rates (91.0%) in BEHAVIOR." MIT.
⚠️ Symbolic actions only, no continuous control. - Claude plays robotics — Anthropic Frontier Red Team — https://www.anthropic.com/research/claude-plays-robotics · study, 2026-07-09 — Twelve models at four levels of control (direct actions, programs, steering a pretrained policy, supervising RL) on simulated classic control, Unitree G1 and Go2 and LIBERO tasks, plus a real Go2; under direct control on LIBERO, models "only complete the full task a small percentage of the time (from 0 to 5.5%)". 🆕
⚠️ The code is promised at github.com/safety-research/embody, which returns 404. - Project Fetch: Phase two — Anthropic — https://www.anthropic.com/research/project-fetch-phase-two · study, 2026-06-18 — Claude Code with Opus 4.7 re-runs 2025's human-team experiment on a quadruped "robodog" (connect to its camera and lidar, find and fetch a beach ball), three trials per task: "Opus 4.7 was, on average, more than 37 times faster than Team Claude-less". 🆕
⚠️ A researcher approved commands and task transitions; closed-loop ball handling stayed hard ("Claude struggled to capture this subtlety"). - Drone-Bench — Andon Labs, with Anthropic's Project Pilot — https://andonlabs.com/evals/drone-bench · benchmark, 2026-07 — Models write code for a DJI Tello EDU to find and follow a person across five sub-tasks, each against a human baseline, on a benchmark held out from model developers: "since no run has ever beaten the Reconstruct baseline, end-to-end success is currently 0%". Project Pilot (2026-07-24) adds that "even Fable 5, the current frontier model, reaches the human baseline on average for only three of the five tasks". 🆕
⚠️ Cheating is common and judged by an LLM ("fable-5-1: 66.0% of 162 reviewed runs with any cheating"); no public code or data. - Butter-Bench — Andon Labs — https://andonlabs.com/evals/butter-bench · paper https://arxiv.org/abs/2510.21860 · benchmark, 2025-10-23 — "Can LLMs control robots? We answer this by testing how good models are at passing the butter – or more generally, do delivery tasks in a household setting." "The best LLMs score 40% on Butter-Bench, while the mean human score is 95%." 🆕
⚠️ No public code. - Agent as Policy for Robotic Manipulation — Notre Dame, UC San Diego, San Diego State — https://agent-as-policy-2026.github.io/ · paper https://arxiv.org/abs/2609.12541 · code https://github.com/agent-as-policy-2026/agent-as-policy · paper, 2026-09-11 — A general-purpose agent is the policy on real bimanual YAM arms: it reads the cameras, writes programs, commands motion and checks the result, with GPT-6 Astra in Codex CLI by default; "Taken together, the eight rows hold 57 successes in 62 trials." Trials and traces are public (§7). Code Apache-2.0. 🆕
⚠️ The 57/62 figure is on the project page, not in the paper. - Can a Frontier VLM Serve as a Robot Policy for Memory-Dependent Manipulation? — Harvard (Jedlicki, Sethi, Yang) — https://embodied-astra.github.io/ · paper https://arxiv.org/abs/2610.17350 · code https://github.com/jackbjed/embodied-astra-robomme · paper, 2026-10-12 — A frozen GPT-6 Astra outputs the waypoints for all 16 RoboMME memory tasks with the whole episode in context: "On the 800 test episodes it succeeds in 92.1%, above the previously reported human score of 90.5%". Code, prompts and per-episode logs are released. MIT. 🆕
⚠️ Simulation, a single run, an undated closed model; the human score used an oracle skill interface. - GPT-6 Astra on robotic manipulation — Robocurve — https://openai.robocurve.org/gpt-6-astra/ · report, 2026-09-04 — A real-robot report per model release: models drive bimanual YAM arms through the Inspect Robots agent policy, 20 trials per task, graded on a 0–4 stage rubric; "On the bowl task Astra placed the block in 19 of 20 trials, against Fable 5.1's 8 of 20 and Fable 5 in 1 of 20". 🆕
⚠️ By its own limitations section, trials were "not interleaved" and grading was "operator-judged with the model known". - HumanCLAW — Meta, NTU, UW, Brown, Northwestern — https://human-claw.github.io/ · paper https://arxiv.org/abs/2607.27180 · code https://github.com/Human-CLAW/HumanCLAW · benchmark, 2026-07-29 — "Can Vision-Language Models Act Through a Body?": a VLM picks an atomic skill every 0.5 s for a simulated human body to find and sit on a target across 1,218 episodes in HSSD houses; "No current VLM solves the benchmark. The best model finds and sits on the target in only 16.8% of episodes". Apache-2.0. 🆕
⚠️ A human avatar, not a robot; the data is gated and the HSSD scenes are non-commercial. - Embodied Agents Take Control (MIP) — Zhou … Wu — https://jianzhou0420.github.io/mip/ · paper https://arxiv.org/abs/2607.26148 · code https://github.com/jianzhou0420/MIP · paper, 2026-07-28 — Minimal-interface agents (Claude Agent SDK, Codex CLI, mini-SWE-agent) navigate with only
observe()andstep()across Habitat benchmarks such as R2R-CE and RxR-CE, plus Go2 trials: "On 100-episode subsets, the best agents exceed published scores and approach or surpass our human tester." MIT. 🆕⚠️ 100-episode subsets; the scene datasets have their own non-commercial terms. - DrivingBench — Ramabadran, Mahns, Gessler — https://drivingbench.com/ · paper https://arxiv.org/abs/2609.38948 · harness https://github.com/aditya-ramabadran/drivingbench_harness_v1 · benchmark, 2026-09-30 — "Can Vision-Language Models Drive a Toyota Corolla?": agents in Codex, Claude Code or Cursor drive a real car with modified openpilot around a cone course through three MCP tools; "Astra is the only model to finish the course, on its second attempt, with no other attempt passing 50% of the course." MIT. 🆕
⚠️ One course and a handful of attempts, with a safety driver on board.
- LIBERO — UT Austin — https://libero-project.github.io · paper https://arxiv.org/abs/2306.03310 · benchmark, 2023-06-05 — Language-specified MuJoCo manipulation with teleoperated demonstrations: "For benchmarking purpose, we create four task suites (130 tasks in total)". Widely used to compare VLAs and close to saturated (§4g). MIT.
⚠️ Dormant repository; results move with the MuJoCo version (§4g). - LIBERO-PRO — Zhou … Sun — https://github.com/Zxy-MLlab/LIBERO-PRO · paper https://arxiv.org/abs/2510.03827 · benchmark, 2025-10-04 — Perturbs LIBERO tasks to separate generalisation from memorisation: "although existing models achieve over 90% accuracy under the standard LIBERO evaluation, their performance collapses to 0.0%". MIT. 🆕
- LIBERO-Plus — Fei … Qiu (CVPR 2026) — https://github.com/sylvestf/LIBERO-plus · paper https://arxiv.org/abs/2510.13626 · benchmark, 2025-10-15 — "a comprehensive benchmark with 10,030 tasks" across seven perturbation dimensions, a drop-in replacement for
libero, "with performance dropping from 95% to below 30% under modest perturbations". 🆕⚠️ No licence file for the code (assets MIT); one trial per variant. - robosuite — ARISE Initiative — https://robosuite.ai · paper https://arxiv.org/abs/2009.12293 · code https://github.com/ARISE-Initiative/robosuite · framework, v1.5.2 2025-12-24 — A modular MuJoCo framework with benchmark environments for many arms and, since v1.5, humanoids: "This paper discusses the key system modules and the benchmark environments of our new release robosuite v1.5." LIBERO and RoboCasa build on it. MIT. 🆕
- RoboCasa — UT Austin and others — https://robocasa.ai · RoboCasa365 paper https://arxiv.org/abs/2603.04356 · leaderboard https://robocasa.ai/leaderboard.html · benchmark, 2024-06-04; RoboCasa365 2026-02-18 — Large-scale kitchen simulation for mobile manipulation: "RoboCasa365 introduces 365 everyday tasks across 2,500 diverse kitchen environments", with a leaderboard whose submissions the team verifies. Code MIT; assets and datasets CC BY 4.0. 🆕
⚠️ v1.0.1 lengthened task horizons, so compare results on the same version. - Meta-World — Yu … Levine; maintained by Farama — https://github.com/Farama-Foundation/Metaworld · paper https://arxiv.org/abs/1910.10897 · benchmark, 2019-10-24; v3.1.1 2026-06-28 — Sawyer-arm MuJoCo tasks with multi-task and meta-RL splits, scored by success rate: "MT1 benchmarks can be created with any of the 50 tasks available in Meta-World." MIT.
⚠️ Tasks changed between versions; cite the version. - ManiSkill3 — Hao Su lab (UC San Diego) — https://github.com/mani-skill/ManiSkill · paper https://arxiv.org/abs/2410.00425 · simulator + benchmark, 2024-10-01; v3.0.1 2026-04-21 — GPU-parallel simulation and rendering on SAPIEN with tasks across many domains: "Simulation with rendering on ManiSkill3 can run 10-1000x faster with 2-3x less GPU memory usage than other platforms". Apache-2.0. 🆕
⚠️ "The assets are licensed under CC BY-NC 4.0"; needs a GPU with Vulkan. - RLBench — Imperial College London — https://github.com/stepjam/RLBench · paper https://arxiv.org/abs/1909.12271 · benchmark, 2019-09-26 — "The benchmark features 100 completely unique, hand-designed tasks ranging in difficulty" on CoppeliaSim and PyRep.
⚠️ A custom licence "solely for non-commercial, internal or academic research purposes"; last release 2021. - CALVIN — University of Freiburg — https://github.com/mees/calvin · paper https://arxiv.org/abs/2112.03227 · benchmark, 2021-12-06 — Long-horizon, language-conditioned manipulation for a Franka across four PyBullet environments, with splits that test on an unseen environment. Code MIT.
⚠️ Dataset terms not stated; the datasets are hundreds of gigabytes. - SimplerEnv — Li … Xiao — https://simpler-env.github.io · paper https://arxiv.org/abs/2405.05941 · benchmark, 2024-05-09 — A collection of simulated environments for Google Robot and WidowX setups that "shows strong correlation with real-world performance", for evaluating real-robot policies without the robot. MIT.
⚠️ Two robot setups; the correlation was validated on policies of its time. - RoboTwin 2.0 — Chen … Mu — https://github.com/RoboTwin-Platform/RoboTwin · paper https://arxiv.org/abs/2506.18088 · leaderboard https://robotwin-platform.github.io/leaderboard · benchmark, 2025-06-22 — A bimanual data generator and benchmark with strong domain randomisation: "The framework is instantiated across 50 dual-arm tasks and five robot embodiments." MIT. 🆕
⚠️ Leaderboard entries are submitted by email; the page describes no independent re-run. - VLABench — OpenMOSS — https://vlabench.github.io · paper https://arxiv.org/abs/2412.18194 · benchmark, 2024-12-24 — Manipulation that needs world knowledge and long-horizon reasoning, for VLAs and LLM or VLM workflows: "VLABench provides 100 carefully designed categories of tasks, with strong randomization in each category of task". MIT.
⚠️ Still a preview: "The complete leaderboard will be released soon." - RoboLab — NVIDIA — https://research.nvidia.com/labs/srl/projects/robolab/ · paper https://arxiv.org/abs/2604.09860 · leaderboard https://research.nvidia.com/labs/srl/projects/robolab/leaderboard.html · code https://github.com/NVlabs/RoboLab · benchmark, 2026-04-10 — "We introduce an accompanying RoboLab-120 benchmark, consisting of 120 tasks categorized into three competency axes", built on Isaac Lab with predicate-based success (RSS 2026); its leaderboard lists an agent entry, a VLM orchestrating a VLA, next to VLAs. Apache-2.0. 🆕
⚠️ Needs an RTX GPU (48 GB+ VRAM recommended); results shift between Isaac Sim versions. - BEHAVIOR-1K — Stanford Vision and Learning Lab — https://behavior.stanford.edu · paper https://arxiv.org/abs/2403.09227 · challenge https://behavior.stanford.edu/challenge/index.html · benchmark, 2024-03-14; v3.9.3 2026-10-07 — 1,000 everyday household activities in OmniGibson on Isaac Sim, scored by goal predicates with partial credit; the BEHAVIOR Challenge 2026 uses 100 tasks with 20,000 demonstrations. Code MIT. 🆕
⚠️ The data bundle's terms say "Data may only be used for non-commercial academic research."; needs an RTX GPU and the Isaac Sim EULA. - RoboDojo — MMLab@HKU, UC Berkeley, Tsinghua, PKU and others — https://robodojo-benchmark.com/ · paper https://arxiv.org/abs/2607.04434 · code https://github.com/RoboDojo-Benchmark/RoboDojo · benchmark, 2026-07-05 — 42 Isaac Sim tasks plus real tasks on standardised remote hardware, scored by success rate and a composite score on a public leaderboard that now includes an LLM agent (§4a). MIT. 🆕
⚠️ Running it needs Isaac Sim and cuRobo, which has a non-commercial licence. - X2Real — X Square Robot — https://github.com/X-Square-Robot/x2real · paper https://arxiv.org/abs/2609.27449 · benchmark, 2026-09-23 — 44 long-horizon Isaac Lab-Arena tasks calibrated to real hardware and scored by success and progress, with an agent track (paper v2) where GPT-6 Astra acts as a per-step end-effector policy, "exceeding wall-oss-preview by 1.89 percentage points, but obtains a lower progress score" (full text). Apache-2.0. 🆕
⚠️ The comparison VLA, wall-oss-preview, is the authors' own. - MIKASA-Robo — Cherepanov … Panov — https://github.com/CognitiveAISystems/MIKASA-Robo · paper https://arxiv.org/abs/2502.10550 · VLA edition https://arxiv.org/abs/2610.00604 · benchmark, 2025-02-14 — Memory-dependent manipulation on ManiSkill (ICLR 2026): "We introduce MIKASA-Robo-VLA, a benchmark of 90 language-conditioned manipulation tasks." MIT. 🆕
- RoboMME — Dai … Chai (ICML 2026) — https://robomme.github.io · paper https://arxiv.org/abs/2603.04639 · benchmark, 2026-03-04 — Memory for generalist policies on ManiSkill: "Our benchmark comprises 16 manipulation tasks constructed under a carefully designed taxonomy". Apache-2.0. 🆕
⚠️ Leaderboard entries are self-run. - RoboCerebra — Han … Liu (NeurIPS 2025) — https://robocerebra.github.io · paper https://arxiv.org/abs/2506.06677 · benchmark, 2025-06-07 — Long-horizon household manipulation for a VLM planner over a VLA: "RoboCerebra contains 1,000 human-annotated trajectories across 100 task variants, each spanning up to 3,000 simulation steps." Apache-2.0. 🆕
- Gymnasium-Robotics — Farama Foundation — https://robotics.farama.org · code https://github.com/Farama-Foundation/Gymnasium-Robotics · environments, v1.4.2 2026-01-02 — Goal-conditioned MuJoCo environments (Fetch, Shadow Hand, Adroit, Franka Kitchen, mazes, multi-agent MuJoCo) used across RL and robot learning. MIT.
⚠️ Its D4RL ports differ from the originals. - MolmoSpaces — Ai2 — https://molmospaces.allen.ai · paper https://arxiv.org/abs/2602.11337 · code https://github.com/allenai/molmospaces · scenes + benchmark, 2026-02-11 — "MolmoSpaces consists of over 230k diverse indoor environments" with objects for MuJoCo, Isaac and ManiSkill, plus a navigation and manipulation benchmark (MuJoCo) and a leaderboard. Code Apache-2.0; data CC BY 4.0. 🆕
⚠️ Non-commercial Objaverse objects are included by default; a filter removes them. - StationeryBench — Robocurve — https://github.com/robocurve/stationerybench · benchmark, 2026-09-10 — "Five bimanual desk-stationery manipulation tasks for VLA models and LLM agents, built on Inspect Robots". MIT. 🆕
⚠️ New, few users yet.
- DeepMind Control Suite — Google DeepMind — https://github.com/google-deepmind/dm_control · paper https://arxiv.org/abs/1801.00690 · benchmark, 2018-01-02; 1.0.49 2026-10-14 — "A set of Python Reinforcement Learning environments powered by the MuJoCo physics engine.": the standard continuous-control suite, plus locomotion, soccer and MJCF tooling. Apache-2.0.
- HumanoidBench — UC Berkeley — https://humanoid-bench.github.io · paper https://arxiv.org/abs/2403.10506 · benchmark, 2024-03-15 — A Unitree H1 with two dexterous hands in MuJoCo: "HumanoidBench is the first-of-its-kind simulated humanoid robot benchmark, including 27 distinct whole-body control tasks". MIT.
⚠️ Quiet since 2025-09. - MyoSuite — MyoHub — https://github.com/MyoHub/myosuite · paper https://arxiv.org/abs/2205.13600 · benchmark, v3.0.0 2026-10-11 — Contact-rich musculoskeletal control in MuJoCo; v3 adds GPU training through mjlab: "Train with thousands of environments in parallel on a single GPU, then replay the policy on the CPU." Apache-2.0. 🆕
⚠️ v3 changed observations for many environments, so v2 results do not compare. - MuJoCo Playground — Google DeepMind and collaborators — https://playground.mujoco.org · paper https://arxiv.org/abs/2502.08844 · environments, 2025-02-12 — GPU locomotion, dexterous-manipulation and vision environments on MJX and MuJoCo Warp, with sim-to-real transfer; "researchers can train policies in minutes on a single GPU". Apache-2.0. 🆕
⚠️ Needs a CUDA GPU. - LocoMuJoCo — Al-Hafez … Tateo — https://github.com/robfiras/loco-mujoco · paper https://arxiv.org/abs/2311.02496 · benchmark, 2023-11-04; v1.1.0 2026-03-10 — Imitation learning for locomotion in MuJoCo and MJX from retargeted motion capture: "Includes 12 humanoid and 4 quadruped environments, featuring 4 biomechanical human models". Code MIT. 🆕
⚠️ The datasets are CC BY-NC-ND 4.0. - DexJoCo — CASIA with SJTU, MBZUAI, PKU and CUHK — https://dexjoco.github.io · paper https://arxiv.org/abs/2605.16257 · code https://github.com/brave-eai/dexjoco · benchmark, 2026-05-15 — Task-oriented dexterous manipulation with human demonstrations and π0.5 baselines: "DexJoCo addresses these gaps with 11 functionally grounded tasks built around the Franka Panda and Allegro Hand." MIT. 🆕
- Microduck Arena — Pollen Robotics (Hugging Face) — https://huggingface.co/spaces/pollen-robotics/microduck-arena · code https://github.com/pollen-robotics/microduck-arena · leaderboard, 2026-09-10 — A hosted locomotion leaderboard for a small biped where "Submitted policies are evaluated server-side on a fixed course over five perturbed seeds", so the host measures every time instead of trusting submitters. 🆕
⚠️ No licence declared; times reproduce only on the Space's hardware. - Lucen Worlds — Lucen Robotics — https://lucenworlds.com/ · game + leaderboard, undated — A browser game of daily challenges (a penalty shootout, a balance beam, a slalom and more) for simulated builds of Lucen's modular robot kit, such as the Laika v2 biped, with MuJoCo running in the page and separate leaderboards for people, AI and policies. Agents connect through a link: "Claude Code or any MCP client, or anything that can make HTTP calls (Gemini, a script)" gets the tools the game gives any agent (look, walk to a point, kick, move a hand, grip), and "It plans; the game's autopilot does the steering." Policies trained on Lucen's MouseMouse hub, which has an Apache-2.0 MCP server (
lucen-mcp), load into the game. Unlike Microduck Arena above, scores are not measured on a server: "Results come from players' own browsers." 🆕⚠️ The game has no public source or licence; the Laika v2 model files are CC BY-NC 4.0; new, with few players yet.
- Habitat 3.0 — Meta — https://github.com/facebookresearch/habitat-lab · paper https://arxiv.org/abs/2310.13724 · library, 2023-10-19; v0.3.4 2026-05-07 — Embodied-AI tasks on Habitat-Sim (navigation, rearrangement, humanoid avatars and human-in-the-loop collaboration), the base of many navigation benchmarks in §4a. MIT.
⚠️ "Beyond v0.3.4 this project is no longer receiving official active development or maintenance by Meta internal teams."; the scene datasets are non-commercial (§5c). - VLN-CE — Krantz … Lee — https://github.com/jacobkrantz/VLN-CE · paper https://arxiv.org/abs/2004.02857 · benchmark, 2020-04-06 — Vision-and-language navigation (R2R, RxR) in continuous Matterport3D scenes with low-level actions and an EvalAI leaderboard. MIT.
⚠️ A frozen stack ("VLN-CE uses Habitat-Sim 0.1.7"); Matterport3D has its own terms. - MultINav-Bench — Beijing Institute of Technology, Zhongguancun Academy, PKU — https://arxiv.org/abs/2610.17089 · code https://github.com/ZGCA-HMI-Lab/MultINav · benchmark, 2026-10-12 — "We introduce MultINav-Bench, a 3K-episode ObjectGoal benchmark in MolmoSpaces spanning channel, container, and mixed tasks", where the robot must open doors and containers, with a VLM-agent baseline. Code Apache-2.0; data CC BY 4.0. 🆕
⚠️ A week old; inherits MolmoSpaces' non-commercial objects. - MetaDrive — Li … Zhou — https://github.com/metadriverse/metadrive · paper https://arxiv.org/abs/2109.12674 · simulator, 2021-09-26; 0.4.3 2024-12-06 — Procedural and real-log driving scenarios for single-agent, multi-agent and safe RL: "It can run up to +1000 FPS on a standard PC." Apache-2.0.
⚠️ No release since 2024-12. - CARLA — CARLA team — https://carla.org · code https://github.com/carla-simulator/carla · paper https://arxiv.org/abs/1711.03938 · simulator, 2017-11-10; 0.9.16 2025-09-16 — An open urban driving simulator on Unreal Engine (0.10 moved to Unreal Engine 5) and the base of closed-loop benchmarks such as Bench2Drive. Code MIT; assets CC BY.
- Bench2Drive — Thinklab-SJTU — https://github.com/Thinklab-SJTU/Bench2Drive · paper https://arxiv.org/abs/2406.03877 · benchmark, 2024-06-06 — Closed-loop end-to-end driving on CARLA that "requires E2E-AD models to pass 44 interactive scenarios under different locations and weathers which sums up to 220 routes" (NeurIPS 2024 D&B).
⚠️ CC BY-NC-ND 4.0 for code and assets (the dataset card says Apache-2.0). - NAVSIM — Dauner … Chitta — https://github.com/autonomousvision/navsim · paper https://arxiv.org/abs/2406.15349 · benchmark, 2024-06-21; v2 2025-06-04 — Non-reactive pseudo-simulation on nuPlan and OpenScene logs with the EPDMS metric and public leaderboards; "NAVSIM enabled a new competition held at CVPR 2024, where 143 teams submitted 463 entries". Code Apache-2.0. 🆕
⚠️ Not fully closed-loop; nuPlan data terms apply. - AlpaSim — NVIDIA — https://github.com/NVlabs/alpasim · challenge https://nvidia-alpasime2eclosedloopchallenge2026.hf.space/ · simulator, v2026.5 2026-06-04 — A modular, data-driven closed-loop driving simulator with neural-reconstruction rendering; its 2026 end-to-end challenge scores submissions on private held-out scenes for capability and safety. Code Apache-2.0. 🆕
⚠️ Scene data is gated under an NVIDIA AV dataset licence.
- RoboHarm — Robocurve, with UCLA — https://robocurve.org/roboharm/ · paper https://arxiv.org/abs/2610.14043 · code https://github.com/robocurve/roboharm · benchmark + paper, 2026-09-18 — "Do Frontier Robot Policies Refuse Unsafe Instructions?": harmful-instruction scenes on real bimanual YAM arms, run with Inspect Robots. The follow-up paper, "Refuses in Chat, Complies with Arms", takes frontier LLMs from text to robot control and finds "refusal falls from 73% of text requests to 18-25% once the model controls the robot". 🆕
⚠️ CC BY-NC 4.0. - ASIMOV — Google DeepMind with Princeton — https://asimov-benchmark.github.io/ · paper https://arxiv.org/abs/2503.08663 · v2 https://arxiv.org/abs/2509.21651 · benchmark, 2025-03-11 — Semantic-safety datasets for foundation models acting as robot brains, built from real images and hospital injury reports, plus generated robot constitutions: "We measure a top alignment rate of 84.3% on the ASIMOV Benchmark using generated constitutions". 🆕
⚠️ The code repository has no licence; the data licence is not stated. - ASIMOV-Agentic — Google DeepMind — https://huggingface.co/datasets/google/asimov_agentic · benchmark, 2026-07-21 — Agentic robot-safety evaluation with an Inspect harness, introduced with Gemini Robotics 2: refusal, feasibility and uncertainty tasks, gauge reading, a closed-loop VLA emulator and human-proximity monitoring. 🆕
⚠️ Gated (automatic approval); the card's CC BY 4.0 tag conflicts with its README's MIT licence; no results on the card. - SafeVLA-Bench — Notre Dame and UPenn — https://safevla.org/ · paper https://arxiv.org/abs/2606.00773 · leaderboard https://safevla.org/leaderboard.html · benchmark, 2026-05-30 — A post-hoc signal-temporal-logic safety layer over LIBERO and RoboCasa-365 rollouts that ranks policies by safety, not success: "the fifteen tabletop policies above 90% mean success still have 18-28% unsafe-episode rates". 🆕
⚠️ Code and data not released yet; the release repository returns 404. - HazardArena — Fudan University and others — https://github.com/HazardArena-Team/HazardArena · paper https://arxiv.org/abs/2604.12447 · benchmark, 2026-04-14 — Matched safe and unsafe twin scenes that need the same motion, showing that "safe-only fine-tuning improves benign task success while also increasing hazardous execution on matched unsafe scenarios". Code and assets MIT. 🆕
- VLA-Arena — Peking University — https://vla-arena.github.io/ · paper https://arxiv.org/abs/2512.22539 · code https://github.com/PKU-Alignment/VLA-Arena · benchmark, 2025-12-27 — "VLA-Arena comprises 11 task suites organized into four dimensions: Safety, Distractor, Extrapolation, and Long Horizon, totaling 170 tasks" (ICML 2026), and finds "memorization over generalization, superficial visual perception, and a neglect of safety constraints". Apache-2.0. 🆕
- LIBERO-Safety — Tsinghua AIR, BAAI and others — https://libero-safety.github.io/ · paper https://arxiv.org/abs/2606.23686 · code https://github.com/LIBERO-SAFETY/LIBERO-Safety · benchmark, 2026-06-22 — Physical and semantic safety as a LIBERO fork (ECCV 2026), with "a large-scale dataset of 19,664 strictly collision-free demonstrations with extensive domain randomization". Code Apache-2.0. 🆕
⚠️ The data states no licence. - MANIGUARD — Northwestern, with Stanford and William & Mary — https://nu-ideas-lab.github.io/ManiGuard/ · paper https://arxiv.org/abs/2608.17386 · code https://github.com/NU-IDEAS-Lab/ManiGuard · benchmark, 2026-08-18 — Safety specified apart from success and checked at runtime by temporal-logic monitors on BEHAVIOR-1K household tasks, with safety-annotated demonstrations and rollouts on a physical Franka; "6-21% of successful rollouts violate the specification". Code Apache-2.0. 🆕
⚠️ Data CC BY-NC 4.0; BEHAVIOR assets are licensed separately.
- vla-eval — MAUM.AI, Ai2, Seoul National University — https://github.com/allenai/vla-evaluation-harness · paper https://arxiv.org/abs/2603.13966 · leaderboard https://allenai.github.io/vla-evaluation-harness/leaderboard/ · harness, v0.8.0 2026-09-30 — Separates model servers from Docker-isolated simulation benchmarks so any VLA runs on any integrated benchmark, and "achieves up to 47x wall-clock speedup, completing 2,000 LIBERO episodes in ~18 minutes". Apache-2.0. 🆕
⚠️ The leaderboard is in beta and aggregates numbers reported in papers rather than re-running them. - XPolicyLab — MMLab@HKU, Tsinghua and the XPolicyLab community — https://github.com/XPolicyLab/XPolicyLab · paper https://arxiv.org/abs/2608.09892 · standard + harness, 2026-08-10 — Shared observation, action and trajectory schemas with a client-server adapter, so N policies and M environments need N+M integrations; the same adapters serve RoboTwin, RoboDojo and real robots. "The ecosystem integrates 42 robot policies and standardizes their installation, debugging, serving, and evaluation workflows." Apache-2.0. 🆕
- LeRobot — Hugging Face — https://github.com/huggingface/lerobot · v0.6.0 post https://huggingface.co/blog/lerobot-release-v060 · EnvHub https://huggingface.co/docs/lerobot/envhub · library, v0.6.0 2026-07-06 — The open robot-learning library (policies, datasets, recording and rollout on real robots) whose
lerobot-evalgained six simulation benchmarks in v0.6.0, each with a Docker image and a SmolVLA baseline: "Together with LIBERO, Meta-World, and NVIDIA IsaacLab-Arena, that makes nine benchmark families under one roof". Apache-2.0. 🆕⚠️ EnvHub runs third-party code: "Loading environments from the Hub executes Python code from third-party repositories." - Isaac Lab-Arena — NVIDIA, "with the evaluation and task layers designed in close collaboration with Lightwheel" — https://github.com/isaac-sim/IsaacLab-Arena · framework, 2025-08; v0.3 2026 — Builds simulated evaluation environments from a scene, an embodiment and a task, runs policies in parallel on GPU and reports metrics and sensitivity analyses, with RoboCasa- and LIBERO-style task packs and LeRobot EnvHub integration. Apache-2.0. 🆕
⚠️ Alpha ("Do not use this in production."); Linux and Isaac Sim only. - RoboArena — Atreya, Pertsch … Finn, Levine — https://robo-arena.github.io/ · paper https://arxiv.org/abs/2506.18123 · code https://github.com/robo-arena/roboarena · leaderboard, 2025-06-22 — Distributed, double-blind pairwise A/B evaluation of generalist policies on real DROID robots, ranked from preferences; the evaluation data is public (§7). MIT. 🆕
⚠️ In June 2026 the team reported "some evidence of benchmark manipulation on RoboArena" (post) and now lets only organisations without an active policy on the leaderboard submit evaluations. - RoboChallenge — Dexmal with Hugging Face — https://robochallenge.ai · paper https://arxiv.org/abs/2510.17950 · real-robot benchmark, 2025-10-20 — "an online evaluation system to test robotic control algorithms" on real robots, with a public leaderboard, competitions and "our initial benchmark Table30". 🆕
⚠️ Data licence not stated. - PhAIL — Positronic Robotics — https://phail.ai · paper https://arxiv.org/abs/2605.29710 · code https://github.com/Positronic-Robotics/positronic · leaderboard, 2026-05-28 — A real-robot VLA leaderboard on a Franka FR3 that compares time-to-success distributions with bootstrap intervals and significance tests instead of one success rate; "the closest pair (OpenPI vs. GR00T) remains unresolved within our budget". Code Apache-2.0; data CC BY 4.0. 🆕
⚠️ Also sold as a commercial evaluation service. - Reality Check — Poke & Wiggle — https://pokeandwiggle.com/leaderboard · leaderboard, 2026-09-29 — PAW-GEN-10: ten real bimanual FR3 Duo environments at three training-data tiers, 3,600 rollouts per model, with per-tier results and 95% intervals for MolmoAct 2, π0.5, DiT-Flow and GR00T N1.7; "Each model gets 30 rollouts for every combination of the evaluation axes below." 🆕
⚠️ The team fine-tunes every model itself; no licence or submission process stated. - ArmnetBench v0.1 — Armnet — https://armnet-dev.github.io/armnetbench-v0.1/ · paper https://arxiv.org/abs/2607.24481 · real-robot benchmark, 2026-07-27 — Parallel real-world evaluation of manipulation policies on a farm of low-cost SO-101 arms, every rollout labelled by a person; "Cable clipping remained unsolved. Every evaluated policy scored 0%." Data Apache-2.0. 🆕
⚠️ Manual resets and human scoring.
- Faster and Better? — Shanghai Jiao Tong University — https://arxiv.org/abs/2609.37771 · paper, 2026-09-29 — "Benchmark Bugs and Design Limitations Distort the Evaluation of Vision-Language-Action Acceleration": an audit of suspicious gains across seven simulation benchmarks (RoboTwin, LIBERO-Plus, VLABench and others) in which "we identify 22 bugs of these types and 4 design limitations"; "bug fixes can reverse method rankings". 🆕
- MINERVA — University of Tokyo (Matsuo-Iwasawa Lab) — https://arxiv.org/abs/2609.03715 · code https://github.com/k1000dai/MINERVA · paper, 2026-09-03 — "How Small Can a Manipulation Policy Be and Still Solve LIBERO?": "A 0.54M-parameter policy achieves 95.1% average success over 2,000 rollouts on the four standard LIBERO suites", and permuting task IDs shows it memorises tasks, so a LIBERO score alone does not show generalisation. Apache-2.0. 🆕
- FailBench — Metric AI Lab — https://metric-ai-lab.github.io/failbench/ · paper https://arxiv.org/abs/2609.03611 · paper, 2026-09-03 — "How Reliable are VLMs at Judging Robot Task Success?": "2,197 manipulation attempts across 14 public sources (12 real-world, 2 simulated)"; "Evaluating 13 VLM-based detectors, we find the best model achieves only 0.77 mean balanced accuracy.", with "a systematic bias toward predicting success under ambiguous evidence". 🆕
⚠️ Code, data and leaderboard are marked "soon". - A Statistical Audit of Physical AI Benchmark Redundancy — Metric AI Lab — https://arxiv.org/abs/2608.25940 · paper, 2026-08-26 — 51 models across physical-AI benchmarks show heavy redundancy; "collapsing the two substitute pairs into single columns moves 22 of 51 models by three or more places". 🆕
⚠️ Mixes model-card numbers with the authors' own runs; the code link returns 404. - GPUSimBench — Shanghai AI Lab, NTU, Nanjing University and others — https://arxiv.org/abs/2607.13059 · paper, 2026-07-06 — Tests GPU simulators (Isaac Lab, Genesis) for scale, physical fidelity and determinism: "we unveil and quantify the inherent non-determinism introduced by GPU-batched execution" (IROS 2026). 🆕
⚠️ No code link. - MuJoCo version silently changes a LIBERO task (LeRobot issue 4390) — LeRobot issue tracker — huggingface/lerobot#4390 · issue, 2026-08-09 — A collision fix in MuJoCo 3.4.0 leaves one LIBERO task's bowl tilted at reset and moves VLA scores sharply: "Whole-suite scores shift by up to ~9 pp from this single task." Pin and report the simulator version. 🆕
⚠️ One reporter; the fix is an open pull request.
- MuJoCo — Google DeepMind — https://mujoco.org · code https://github.com/google-deepmind/mujoco · physics engine, 3.16.0 2026-10-14 — "MuJoCo is a free and open source physics engine that aims to facilitate research and development in robotics"; LIBERO, robosuite, RoboCasa, Meta-World and many suites above run on it. Apache-2.0.
⚠️ Frequent releases change results; pin the version (§4g). - MuJoCo Warp — Google DeepMind and NVIDIA — https://github.com/google-deepmind/mujoco_warp · GPU physics, first release 2026-01-08 — GPU MuJoCo on NVIDIA Warp for batched simulation, with a batch ray-traced renderer; "MJWarp is maintained by Google DeepMind and NVIDIA as part of the Newton project." Apache-2.0. 🆕
⚠️ Needs an NVIDIA GPU. - Newton — Linux Foundation project started by Disney Research, Google DeepMind and NVIDIA — https://github.com/newton-physics/newton · announcement https://www.linuxfoundation.org/press/linux-foundation-announces-contribution-of-newton-by-disney-research-google-deepmind-and-nvidia-to-accelerate-open-robot-learning · physics engine, 1.0.0 2026-03-10; 1.7.0 2026-10-14 — A GPU physics engine on Warp and OpenUSD with several solvers (MuJoCo Warp, VBD, MPM and others), and Isaac Lab 3.0's default backend; "Newton is a Linux Foundation project that is community-built and maintained." Apache-2.0. 🆕
⚠️ The CUDA path needs a Turing-or-newer GPU and an R580+ driver. - Isaac Lab — NVIDIA — https://github.com/isaac-sim/IsaacLab · install https://isaac-sim.github.io/IsaacLab/v3.0.0/source/isaaclab/installation.html · framework, v3.0.0 2026-10-20 — NVIDIA's robot-learning framework; 3.0 runs one task API on Newton, on standalone PhysX and RTX, or on full Isaac Sim: "Kit-less Newton workflows do not require Isaac Sim." BSD-3-Clause. 🆕
⚠️ The Isaac Sim paths and NVIDIA assets are proprietary. - Isaac Sim — NVIDIA — https://github.com/isaac-sim/IsaacSim · requirements https://docs.isaacsim.omniverse.nvidia.com/latest/installation/requirements.html · simulator, v6.2.0 2026-10-14 — A robot simulator on Omniverse and OpenUSD with PhysX, RTX sensors, URDF and MJCF import, synthetic data and ROS 2; RoboLab, RoboDojo, BEHAVIOR and EmbodiedSWE-Bench run on it. 🆕
⚠️ Repo code is Apache-2.0, but Kit and the assets fall under a proprietary licence that bars redistribution, and "GPUs without RT Cores (A100, H100) are not supported." - Genesis World — Genesis AI — https://github.com/Genesis-Embodied-AI/genesis-world · docs https://genesis-world.readthedocs.io/en/latest/ · simulator, v1.0.0 2026-05-27; v1.4.4 2026-10-11 — "Genesis World is a simulation platform for physical AI developments", with rigid, MPM, SPH, FEM, PBD and IPC solvers and a photoreal renderer. Apache-2.0. 🆕
- SAPIEN — Hao Su lab (UC San Diego) and collaborators — https://github.com/haosulab/SAPIEN · docs https://sapien-sim.github.io/docs/ · simulator, 3.0.3 2026-03-10 — A PhysX 5 simulator for articulated objects with Vulkan and ray-traced rendering; the backend of ManiSkill and RoboTwin. Apache-2.0.
- Drake — Toyota Research Institute and MIT — https://drake.mit.edu · code https://github.com/RobotLocomotion/drake · toolbox, v1.58.0 2026-10-15 — Model-based design and simulation for robots: multibody dynamics, mathematical programming and control, with "core development led by the Toyota Research Institute". BSD-3-Clause.
- Gazebo — Gazebo project (Open Robotics community) — https://gazebosim.org · simulator, Jetty LTS 2025-09 — The ROS-aligned open robot simulator. Apache-2.0.
⚠️ No 2026 release: "The next release, Gazebo M (skipping K and L), is planned for March 2027" (announcement). - Habitat-Sim — Meta FAIR — https://github.com/facebookresearch/habitat-sim · simulator, final tag v0.3.4 2026-05-07 — A fast 3D simulator for embodied AI in scanned and synthetic homes (HM3D, HSSD, MP3D); Can Jev Nav? and the VLN benchmarks above run on it. MIT.
⚠️ No longer maintained by Meta after v0.3.4. - Project SuperDex — Meta — https://github.com/facebookresearch/project_superdex · site https://projectsuperdex.com/ · simulator, v1.0.0 2026-08-24 — A dexterous-manipulation platform: contact-first physics, a robotics SDK, a Studio GUI and an RL and MPC harness. Code Apache-2.0; assets CC BY 4.0. 🆕
⚠️ "Certain third party dependencies and third party assets in this repo are licensed for non-commercial/academic uses only." - mjlab — Zakka, Liao, Yi, Le Lay, Sreenath, Abbeel — https://github.com/mujocolab/mjlab · paper https://arxiv.org/abs/2601.22074 · framework, v1.6.0 2026-08-09 — Isaac Lab's manager-based API on MuJoCo Warp, a lightweight GPU RL framework. Apache-2.0. 🆕
⚠️ "mjlab requires an NVIDIA GPU for training. macOS is supported for evaluation only." - SPEAR — Roberts et al. — https://github.com/spear-sim/spear · paper https://arxiv.org/abs/2607.06701 · simulator, v1.1.0 2026-09-05 — Python control of any Unreal Engine application with fast photoreal rendering; v1.1.0 adds an MCP server: "We provide an MCP server that interoperates with Claude Code". Code MIT; assets CC0. 🆕
⚠️ Needs the Unreal Engine toolchain. - Project AirSim — IAMAI Consulting (former Microsoft AirSim engineers) — https://github.com/iamaisim/ProjectAirSim · docs https://iamaisim.github.io/ProjectAirSim/ · simulator, v1.0.1 2026-09-07 — A simulator for drones, fixed-wing aircraft and ground vehicles, headless or on Unreal Engine 5, with ROS 2 support, continued "After Microsoft discontinued the Project AirSim initiative". MIT. 🆕
Coding agents increasingly build the simulation as well as act in it; Gemini 4 Argon (§2a) builds a MuJoCo twin from one photo.
- Plumb — Westlake University and HKU (Liang, Chen, Huang, Chen) — https://yubel426.github.io/Plumb/ · code https://github.com/Inception3D/Plumb · harness + benchmark, code 2026-10-21 — "An Agent Harness for Inverse Physics of Dynamic Scenes": a coding agent (Codex, or Claude Code in a container) gets a video, a MuJoCo sandbox, and SAM 3 (§6) tracking and Pi3X depth and camera poses as tools, and writes the MJCF scene, its initial state, the physical parameters and the fitting code. A separate evaluator replays only the delivered scene and initial state, never the agent's code, against the datasets' annotations, including from a camera the agent never saw. "We evaluate on 50 videos, 25 from PhysInOne and 25 from Physion v1.5, selected by category." With the same GPT-6.1 Sol model, Plumb beats Codex without the 3D tools on every accuracy metric, yet "even with Plumb, only about a third of the physical quantities, such as restitution, deceleration and mass ratios, fall within a factor of 1.25 of the ground truth." Code MIT. 🆕
⚠️ "arXiv coming soon"; the Pi3X weights are CC BY-NC 4.0, the SAM 3 weights are gated, and PhysInOne is CC BY-NC-SA 4.0; needs an NVIDIA GPU; few outside users yet. - GATOR — NVIDIA and Simon Fraser University (Wu, Birchfield, Rabeti, Chang, Wen) — https://research.nvidia.com/labs/lpr/gator/ · paper https://arxiv.org/abs/2610.11215 · paper, 2026-10-08 — "Generative and Agentic 3D Object Reconstruction From Casual Images": "a generative and agentic framework that recovers textured object assets and their scene-relative pose from one or more images", in which GPT-6-Astra, run through the Codex command-line agent with Blender access, repairs each generated mesh; reconstructed rooms settle under gravity in Isaac Sim (§5a). "On ten sampled Toys4K objects, GATOR without refinement takes about 30 seconds and already outperforms the from-scratch agent with a twenty-minute budget on chamfer distance and LPIPS." Paper CC BY 4.0. 🆕
⚠️ Code "Coming Soon"; "Agentic refinement relies on a proprietary model and adds up to ten minutes of agent time per object."; the output is GLB meshes, not robot-ready MJCF or URDF. - Video2World — Aether AI and UC San Diego — https://aetherlabsai.github.io/Video2World/ · paper https://arxiv.org/abs/2610.04432 · code https://github.com/AetherLabsAI/Video2World · benchmark, 2026-10-03 — Coding agents rebuild a robot or human video as interactive simulator code (MuJoCo, SAPIEN, Isaac Sim), scored on geometry, dynamics and task success; "the best of nine frontier agents finishes 25.5% of tasks, while a human-assisted reference reconstruction reaches 58.8%". Apache-2.0. 🆕
- RoboSimStudio — UC Berkeley and NVIDIA (Coumans, Macklin, Goldberg and others) — https://robosimstudio.github.io/ · paper https://arxiv.org/abs/2610.18835 · code https://github.com/robosimstudio/robosimstudio · toolkit, 2026-10-13 — A real-to-sim toolkit on Newton where Codex or Claude Code builds a simulation from a task description and a video, "allowing both humans and coding agents to quickly create and iteratively refine simulations". Apache-2.0. 🆕
⚠️ A week old; needs an NVIDIA GPU. - EmbodiedGen — Horizon Robotics — https://github.com/HorizonRobotics/EmbodiedGen · paper https://arxiv.org/abs/2607.07459 · generator, v2.1.0 2026-08-19 — Generates simulation-ready, physics-checked 3D assets and scenes for several simulators; "EmbodiedGen ships a Claude Code plugin whose slash commands wrap the generation and spatial-computing skills". Apache-2.0. 🆕
⚠️ The third-party models it calls keep their own terms; needs a CUDA GPU.
- HX Friends — ROBOTIS — https://docs.robotis.com/docs/systems/hx_friends/ · HX5-D20 code and MuJoCo models https://github.com/ROBOTIS-GIT/robotis_hand · announcement https://x.com/passionvirus/status/2112665580235091979 · open hardware, 2026-10-20 — "HX Friends is a collection of robot hand designs built on the DYNAMIXEL ecosystem.": five reference designs, the HX5-D20 ("A 20 DOF anthropomorphic reference with a thumb and four fingers"), HX4-D12 and HX3-D11 hands, the HX2-D01 gripper and the HX1-D03 finger, each with CAD and URDF downloads except the gripper. ROBOTIS's ROS 2 repository adds MuJoCo models of the HX5-D20 with left, right and two-hand scenes (open-source resources), next to the hands in MuJoCo Menagerie (§5c). Repository Apache-2.0. 🆕
⚠️ The HX4-D12, HX3-D11 and HX1-D03 CAD and URDF are Dropbox downloads with no licence stated; the HX2-D01 has no files yet ("Interactive 3D model coming soon."). - MuJoCo Menagerie — Google DeepMind — https://github.com/google-deepmind/mujoco_menagerie · PyPI https://pypi.org/project/mujoco-menagerie/ · robot models, PyPI 2026-09-04 — Curated MJCF models of arms, hands, humanoids, quadrupeds and drones; the pip package fetches each model on first load. 🆕
⚠️ "XML and asset files in each individual model directory of this repository are subject to different license terms." - robot_descriptions — Stéphane Caron — https://github.com/robot-descriptions/robot_descriptions.py · robot models, v3.3.0 2026-10-12 — "Access 200+ robot descriptions from the main Python robotics frameworks": loaders that fetch and cache URDF and MJCF for MuJoCo, Pinocchio and others. Apache-2.0; each description keeps its own licence.
- SO-ARM100 and SO-101 — The Robot Studio with Hugging Face — https://github.com/TheRobotStudio/SO-ARM100 · docs https://huggingface.co/docs/lerobot/so101 · open hardware, 2024-05 — A low-cost 3D-printed arm with leader-follower teleoperation and simulation models, supported by LeRobot and Inspect Robots and used in the Gemini 4 Argon demo. Apache-2.0. 🆕
- ChemLab-Assets — MBZUAI NLP — https://github.com/mbzuai-nlp/ChemLab-Assets · assets, v1.0 2026-10-14 — "Agents that work in a chemistry lab need more than meshes.": "203 simulation-ready chemistry laboratory instruments" from 119 manufacturers with part-level geometry, joints, executable behaviour models and machine-readable contracts, as GLB, URDF, per-part OBJ and Blender sources, all converted to USD and stepped in Isaac Lab. Code MIT. 🆕
⚠️ "Models, contracts and data are CC BY-NC 4.0"; the README says to treat them as behaviour models for research, "not as certified digital twins". - HSSD — Khanna … Savva — https://huggingface.co/datasets/hssd/hssd-hab · paper https://arxiv.org/abs/2306.11290 · scenes, 2023-06-20 — Human-authored synthetic homes for Habitat; it "contains a diverse set of 211 scenes and more than 18000 models of real-world objects", and Can Jev Nav? and HumanCLAW use it.
⚠️ CC BY-NC 4.0. - HM3D — Meta AI and Matterport — https://aihabitat.org/datasets/hm3d/ · paper https://arxiv.org/abs/2109.08238 · scans, 2021-09-16 — Building-scale 3D scans of real spaces for embodied AI.
⚠️ "HM3D is free and available here for academic, non-commercial research."; access needs a Matterport account. - PartNet-Mobility — SAPIEN team — https://huggingface.co/datasets/sapien-sim/PartNetMobility · articulated objects, 2020 — Articulated object models used by SAPIEN, ManiSkill and many manipulation benchmarks.
⚠️ "Researcher shall use the Database only for non-commercial research and educational purposes."; access is approved by hand. - Objaverse-XL — Ai2 — https://github.com/allenai/objaverse-xl · site https://objaverse.allenai.org · 3D objects, 2023-07-11 — A very large collection of 3D objects with download scripts, a common source for generated scenes and benchmark objects; "The use of the dataset as a whole is licensed under the ODC-By v1.0 license."
⚠️ Each object keeps its own licence, some non-commercial; filter before commercial use. - NVIDIA SimReady — NVIDIA — https://docs.omniverse.nvidia.com/simready/latest/index.html · validators https://github.com/NVIDIA/simready-foundation · asset specification, docs updated 2026-10-14 — An OpenUSD specification with validators for simulation-ready assets (semantics, physics, sensors) that NVIDIA's asset packs follow; "NVIDIA initiated SimReady, which will evolve under AOUSD governance." (FAQ) Validators Apache-2.0. 🆕
⚠️ Terms differ per asset pack: a warehouse pack on Hugging Face is CC BY 4.0, while assets bundled with Isaac Sim are proprietary.
Models that agents call as tools to see: segmentation, pose, depth, grasps and tracking, plus tools to inspect robot data.
- Flora — NVIDIA GEAR — https://research.nvidia.com/labs/gear/flora/ · paper https://research.nvidia.com/labs/gear/flora/assets/paper/flora.pdf · models https://huggingface.co/collections/nvidia/flora · models + paper, 2026-10-15 — Three video models fine-tuned from Qwen3.5 that label robot and human episodes: Flora-Rewarder gives "An integer episode score from 1 (complete failure) to 5 (complete success)" or a score for "each 0.5-second step (1 = failure, 2 = suboptimal, 3 = normal)", and Flora-TempSeg and Flora-Captioner write timestamped captions. NVIDIA presents it as "an industrial-strength data engine used for training GR00T models", "roughly ~125× cheaper than GPT-6 Astra" and "labeling 200,000 hours of data per day across 512 GB300s"; on RoboReward its MAE is 40.0 against GPT-6 Astra's 55.4, while Astra leads on the Robot200 captioning subset and on RubricBench. 🆕
⚠️ Weights and inference scripts are under the NVIDIA OneWay Noncommercial License, and "Flora is not intended or supported for commercial or production use". - S4VY — Jingdong Zhang, Xin Li, Jan Kautz, Wenping Wang, Chris Choy — https://arxiv.org/abs/2609.36875 · code https://github.com/Evergreen0929/S4VY · paper, 2026-09-29 — "We introduce S4VY, a Segment Anything model built on feed-forward 4D visual geometry": class-agnostic 4D instance masks that track each object across all frames from RGB, prompt-free or point- and box-prompted, plus an agentic system for natural-language grounding in 4D scenes. 🆕
⚠️ Code "Coming soon": "The code is pending release approval and will be published in this repository." - SAM 3 — Meta — https://github.com/facebookresearch/sam3 · paper https://arxiv.org/abs/2511.16719 · model, 2025-11-20 — Segments and tracks every instance of a concept given as text or an example, in images and video; "It achieves 75-80% of human performance on our new SA-CO benchmark which contains 270K unique concepts". CaP-X uses it as a perception primitive. 🆕
⚠️ A custom SAM licence with trade-control and ITAR restrictions; the weights are gated. - Grounded SAM 2 — IDEA Research — https://github.com/IDEA-Research/Grounded-SAM-2 · pipeline, v1.0 2024-12-11 — "Grounded SAM 2 is a foundation model pipeline towards grounding and track anything in Videos": open-vocabulary detection (Grounding DINO, Florence-2 or DINO-X) plus SAM 2 for text-prompted segmentation and tracking. Apache-2.0.
⚠️ The newer DINO-X detectors are available only through a cloud API. - FoundationPose — NVIDIA — https://github.com/NVlabs/FoundationPose · model, 2023-12-13 — 6D pose estimation and tracking of novel objects, with or without a CAD model (CVPR 2024).
⚠️ Non-commercial licence: "The Work and any derivative works thereof only may be used or intended for use non-commercially." - GraspGen — NVIDIA — https://github.com/NVlabs/GraspGen · paper https://arxiv.org/abs/2507.13097 · model, 2025-07-17 — A diffusion model that generates 6-DoF grasps for parallel-jaw and suction grippers, served over ZMQ and callable by agents: "Added MCP for calling GraspGen as a tool by an LLM." Dataset CC BY 4.0. 🆕
⚠️ The code is under a non-commercial NVIDIA licence. - VGGT — Meta (Wang, Karaev, Vedaldi, Rupprecht, Novotny et al.) — https://github.com/facebookresearch/vggt · paper https://arxiv.org/abs/2503.11651 · model, 2025-03-14 — Feed-forward prediction of cameras, depth, point maps and tracks from one to hundreds of views (CVPR 2025 Best Paper). 🆕
⚠️ "We've updated the license for VGGT to permit commercial use (excluding military applications)."; the original VGGT-1B weights stay CC BY-NC 4.0, and the commercial checkpoint is gated. - Depth Anything 3 — ByteDance Seed — https://github.com/ByteDance-Seed/Depth-Anything-3 · paper https://arxiv.org/abs/2511.10647 · model, 2025-11-13 — Depth and camera-ray estimation from any number of views, plus metric and monocular depth and a streaming mode for long video. Code Apache-2.0. 🆕
⚠️ The largest weights are CC BY-NC 4.0; smaller ones are Apache-2.0. - CoTracker3 — Meta — https://github.com/facebookresearch/co-tracker · paper https://arxiv.org/abs/2410.11831 · model, 2024-10-15 — Tracks any pixel through a video, offline or online, trained by pseudo-labelling real videos.
⚠️ "The majority of CoTracker is licensed under CC-BY-NC". - Molmo 2 — Ai2 — https://allenai.org/blog/molmo2 · code https://github.com/allenai/molmo2 · ER model https://huggingface.co/allenai/Molmo2-ER · models, 2025-12-11 — Open vision-language models for video QA, pointing with coordinates and timestamps, and tracking; Molmo2-ER is the embodied-reasoning backbone of MolmoAct 2 (§2b). Apache-2.0. 🆕
⚠️ Trained partly on third-party data "subject to academic and non-commercial research use only". - Rerun — Rerun — https://github.com/rerun-io/rerun · tool, 0.39.0 2026-10-12 — "Log, query, visualize, and stream to training on shared columnar storage built for multimodal data": the open viewer for robot logs, reading MCAP and LeRobot data. MIT and Apache-2.0. 🆕
- Foxglove — Foxglove — https://foxglove.dev/blog/foxglove-goes-agentic · tool, 2026-08-18 — Robot-data visualisation with an in-app agent and a local MCP server over recordings (layouts, scripts, semantic search). 🆕
⚠️ Proprietary; "Agentic workflows are available in beta to all customers."
- EgoXperience — Ropedia, Nanyang Technological University, Tsinghua University, Peking University (Ji, Pang, Li … Liu) — https://ropedia.github.io/EgoXperience/ · paper https://arxiv.org/abs/2610.26625 · code https://github.com/Ropedia/EgoXperience · paper + dataset, 2026-10-19 — "Learning World-Grounded Egocentric Representations from Human Experience": human activity captured in eight synchronized modalities (RGB, depth, camera poses, MANO hands, object masks, object 6DoF trajectories, metric object meshes and task captions), a benchmark (EgoX-Bench) and EgoX, a 1.2B-parameter model that "extends the VGGT feed-forward reconstruction paradigm from scenes to egocentric video" (§6). "EgoX-Data-6M contains 50 hours and 5.4 million multimodal frames across 270 tasks and 101 objects, spanning workspace, cross-room, and outdoor activities." 🆕
⚠️ Code, data, benchmark and weights are all marked coming soon (the Hugging Face repositories hold only cards); no licence is stated. - Open X-Embodiment — Open X-Embodiment Collaboration (Google DeepMind and partner institutions) — https://robotics-transformer-x.github.io/ · paper https://arxiv.org/abs/2310.08864 · dataset, 2023-10-13 — Robot datasets from many labs pooled in one format, with the RT-1-X and RT-2-X models: "It contains 1M+ real robot trajectories spanning 22 robot embodiments". CC BY 4.0; code Apache-2.0.
⚠️ Constituent datasets carry their own citations. - DROID — Khazatsky, Pertsch et al. — https://droid-dataset.github.io/ · paper https://arxiv.org/abs/2403.12945 · dataset, 2024-03-19 — In-the-wild Franka teleoperation: "76k demonstration trajectories or 350 hours of interaction data, collected across 564 scenes and 84 tasks"; RoboArena (§4f) evaluates on the same platform. CC BY 4.0 per the paper.
- LeRobot Community Dataset v3 — Hugging Face — https://huggingface.co/datasets/lerobot/community_dataset_v3 · dataset, 2026-07-16 — Crowdsourced cross-embodiment data, mostly SO-100 and SO-101 arms, "featuring 791 datasets across 46 robot types, enabling cross-embodiment pretraining for generalist robot policies". Apache-2.0. 🆕
- AgiBot World 2026 — AgiBot — https://huggingface.co/datasets/agibot-world/AgiBotWorld2026 · dataset, 2026-03-11 — Real-world data from AgiBot's dual-arm robot in LeRobot format, "collected from 100% real-world environments, covering commercial spaces, home, and other general-purpose scenarios". 🆕
⚠️ CC BY-NC-SA 4.0; the card gives no size. - FineART — Scale AI and Hugging Face LeRobot — https://huggingface.co/datasets/lerobot/FineART · paper https://arxiv.org/abs/2609.36416 · dataset, 2026-09-29 — Bimanual trajectories with dense human-written subtask labels, "comprising 40,543 episodes (1,718 hours) and 533,913 subtasks across 151 tasks". 🆕
⚠️ The licence is "other" with no terms stated. - RoboArena data dump — RoboArena — https://huggingface.co/datasets/RoboArena/DataDump_07-17-2026 · dataset, 2026-07-17 — Real evaluation sessions on the DROID platform with "autonomous policy rollouts, task-success scores, policy-preference annotations, and related metadata", useful for checking simulated rankings against real ones. MIT. 🆕
⚠️ A dated snapshot. - Agent as Policy trials — Notre Dame, UC San Diego, San Diego State — https://huggingface.co/datasets/Agent-as-Policy/agent-as-policy · dataset, 2026-09-14 — "162 real-robot trials in which an LLM agent acts directly as the policy on a bimanual YAM arm setup", with the full agent traces (tool calls, images, commands); a RoboDojo simulation set sits next to it. CC BY 4.0. 🆕
- BiGym 2.0 coding-agent rollouts — Imperial College London — https://huggingface.co/datasets/SWIRL-Lab/bigym2-agent-rollouts · dataset, 2026-09-29 — Programs written by Claude Code with Opus 5.5 and Codex with GPT-6 Astra, replayed on hidden seeds with traces: "nine tasks, three sessions each: 54 programs, 5,400 episodes." CC BY 4.0. 🆕
- Kinex benchmark release — Kinex contributors — https://huggingface.co/datasets/stacx-lab/kinex-benchmark-release · viewer https://huggingface.co/spaces/stacx-lab/kinex-benchmark-release · dataset, 2026-10-15 — Episodes, demos and agent traces from Kinex's LIBERO, RoboTwin and RoboDojo evaluation (§1a). 🆕
⚠️ The card states no licence. - RoboQuest demonstrations — DeCLaRe Lab, NTU — https://huggingface.co/datasets/declare-lab/RoboQuest · dataset, 2026-10-05 — "5,000 verified demonstrations of the ten RoboQuest tasks (500 per task, 366 hours at 20 Hz)" in RoboCasa365 kitchens, with two-level language labels. MIT. 🆕
- Agentic Robotics — hu-po — https://www.youtube.com/watch?v=Hco-tpjsEuc · livestream, 2026-10-09 — A 1.5-hour paper-reading stream on coding agents that build and improve robot skills, covering NVIDIA's ASPIRE and ENPIRE and Ken Goldberg's case for agentic coding. 🆕
- Robotics' End Game — Jim Fan (NVIDIA), Sequoia AI Ascent 2026 — https://www.youtube.com/watch?v=3Y8aq_ofEVs · talk, 2026-04-30 — Argues that world models, egocentric video and world-action models will replace teleoperation-heavy VLA pipelines, and "why he thinks we'll pass the physical Turing test within 2–3 years". 🆕
- The Physical Turing Test — Jim Fan (NVIDIA), Sequoia AI Ascent 2025 — https://www.youtube.com/watch?v=_2NijXqBESI · talk, 2025-05-07 — Simulation at scale, digital twins and "digital cousins" as the path to general-purpose robots. 🆕
- Fully autonomous robots are much closer than you think — Sergey Levine on the Dwarkesh Podcast — https://www.youtube.com/watch?v=48pxVdmkMIE · transcript https://www.dwarkesh.com/p/sergey-levine · podcast, 2025-09-12 — Robot foundation models, the data and deployment flywheel, and timelines: "His median estimate for when robots will be able to run households entirely autonomously? 2030." 🆕
- This is the State of the Art in Robotics — Chelsea Finn (Physical Intelligence), Y Combinator — https://www.youtube.com/watch?v=cRZNwgvcWUg · talk, 2026-08-12 — Physical Intelligence's reliability work: "She shares how reinforcement learning pushed robot throughput up 2x, how their systems can run autonomously for hours". 🆕
- Robots That Predict the Future: Dyna-2 & Physical AI — Jason Ma (Dyna Robotics) on Latent Space — https://www.latent.space/p/dyna · video https://www.youtube.com/watch?v=BHiYomkfoGQ · podcast, 2026-10-13 — "Our first robotics pod with Dyna’s Chief Scientist on their big bet on World Action Models", and what hour-long autonomy takes. 🆕
- Karol Hausman on The Generalist — Karol Hausman (Physical Intelligence) — https://www.generalist.com/p/karol-hausman-physical-intelligence · video https://www.youtube.com/watch?v=Gtiv9j2fRDU · podcast, 2026-03-17 — Physical Intelligence's generalist-model thesis: "The core thesis: the same scaling dynamics that transformed language models may also unlock robotic intelligence." 🆕
- A Tale of Two Cultures: Can Agentic Coding Close the Gap? — Ken Goldberg (UC Berkeley), ICRA 2026 plenary — https://2026.ieee-icra.org/attend/plenary-sessions/ · talk, 2026-06-02 — Code as policy and agentic coding as a way through robotics' data gap: "Recent advances in agentic coding (e.g., Claude Code and OpenClaw) suggest a new path for this fourth approach." 🆕
⚠️ No recording found; the link is the abstract.
- Robot-Use Agents — Phillip Isola (MIT) — https://web.mit.edu/phillipi/www/writing/robot-use-agents.html · essay, 2026-09-07 — The argument for "robot-use agents": LLM agents can use robots the way computer-use agents use computers, so capability can spread through cloud control rather than one model per robot; "Every robot with an internet connection becomes a potential tool for Fable, Astra, and other AIs." 🆕
- Towards Agentic Robots: Model, Data, Environment, and Harness — Qin, Wu, Feng … Lin, Jiang, Yue — https://arxiv.org/abs/2610.20442 · survey, 2026-10-14 — A survey that sorts the field into "seventeen categories across four functional domains: Model, Data, Environment, and Harness". 🆕
- Survey on Multimodal Embodied Agents: From Computer-Use to Robot-Use — Show Lab, NUS — https://figshare.com/articles/preprint/Survey_on_Multimodal_Embodied_Agents_A_Unified_Capability-centric_Perspective_from_Computer-Use_to_Robot-Use/33529048 · paper list https://github.com/showlab/Awesome-Multimodal-Embodied-Agent · survey, 2026-09-11 — Asks "what changes when a multimodal agent moves from computer-use to robot-use?" and organises both around perceive, anticipate, plan, act and verify. CC BY 4.0. 🆕
- What work can robots do? — Anthropic Economics — https://www.anthropic.com/research/what-work-can-robots-do · report, 2026-09-30 — A robot exposure index over US job tasks: robots "can perform three-quarters of physical tasks in the US, making up 34% of working hours, but mostly in limited settings", yet "Robots are cost-competitive for just 0.3% of job tasks." 🆕
⚠️ The task ratings come from Claude applying a rubric.
- Robot-use agents and harnesses: Dimensional (DimOS, Can Jev Nav?) · Robocurve (Inspect Robots, RoboHarm; "Robocurve is a Public Benefit Corporation helping society understand the frontier of physical AI.") · General Robotics (GRID Auto-Engineering, 2026-09-09: "Robotics needs its own harnesses, and over the last few months, we have built them into GRID.") · Waddle Labs (agents that write robot programs and call VLAs; early access) · TypeSafe (Jev) · BenchFlow (Robo Use).
- Robot foundation models and world models: Physical Intelligence · Google DeepMind (Gemini Robotics) · NVIDIA GEAR · Dream Labs · Skild AI · Generalist · Dyna Robotics · Figure · Black Forest Labs · Elorian · Hugging Face LeRobot and Pollen Robotics.
- Robot evaluation and verification: Andon Labs (Butter-Bench, Drone-Bench) · Positronic Robotics (PhAIL; "runs independent evaluation for physical AI") · Poke & Wiggle (Reality Check) · RoboChallenge · Artificial Analysis (AA-Robotics) · Instance (a success-detector API for robot rollouts) · Antioch ("High-fidelity simulation for Physical AI, at cloud scale.").
- Awesome Agent Evals — BenchFlow's sister list on evaluating AI agents, with the same rules as this one.
- Awesome Robot Use Agent — Papers, projects, demos and benchmarks for robot-use agents (MIT).
- Awesome Robot Use Agent — A second list on the same topic.
- Awesome Astra Embodied AI — Uses of GPT-6 Astra in embodied AI and robotics.
- Awesome Multimodal Embodied Agent — The paper list behind Show Lab's survey (CC0).
A daily scan finds new work from robotics lab blogs, arXiv cs.RO, Hugging Face, GitHub and talks, vets it against CONTRIBUTING.md, verifies every number against the live page, and opens one pull request that a maintainer must approve; on a day with nothing new it changes nothing. SCAN.md describes the sources, the bar, the state kept in .scanner/, and what it needs to run: one repository secret, CLAUDE_CODE_OAUTH_TOKEN, two optional Slack secrets (SLACK_BOT_TOKEN, SLACK_CHANNEL_ID), Actions allowed to open pull requests, and a ruleset on main that requires an approving review.
PRs welcome. Keep the bar high: something to run or read, a one-line why, who judged success and over how many trials, the licence, and a verified URL. See CONTRIBUTING.md. A great list is as much about what it leaves out.
To the extent possible under law, BenchFlow and contributors have waived all copyright and related rights to this list and its annotations (CC0 1.0). The linked resources remain under their own licences.
