|
| 1 | +<div align="center"> |
| 2 | + |
| 3 | +# EmbodiedSkillBench |
| 4 | + |
| 5 | +### Benchmarking How Well Agents with Embodied Skills Perform on Physical Tasks |
| 6 | + |
| 7 | +**Reusable embodied capabilities. Closed-loop physical interaction. Measurable skill utility.** |
| 8 | + |
| 9 | +<p> |
| 10 | + <a href="#overview">Overview</a> · |
| 11 | + <a href="#key-results">Results</a> · |
| 12 | + <a href="#getting-started">Get started</a> · |
| 13 | + <a href="RUN_BENCH.md">Run the benchmark</a> · |
| 14 | + <a href="evaluation/README.md">Evaluation</a> |
| 15 | +</p> |
| 16 | + |
| 17 | +<p> |
| 18 | + <img src="https://img.shields.io/badge/Python-3.10–3.12-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python 3.10 to 3.12"> |
| 19 | + <img src="https://img.shields.io/badge/Benchmark-LIBERO--Pro-0A9A85?style=flat-square" alt="LIBERO-Pro benchmark"> |
| 20 | + <img src="https://img.shields.io/badge/Tasks-Clean100-6654C0?style=flat-square" alt="Clean100 tasks"> |
| 21 | + <a href="https://github.com/HanZhi306/EmbodiedSkill-Bench/actions/workflows/checks.yml"><img src="https://github.com/HanZhi306/EmbodiedSkill-Bench/actions/workflows/checks.yml/badge.svg" alt="Code checks"></a> |
| 22 | +</p> |
| 23 | + |
| 24 | +<img src="docs/images/benchmark-overview.png" alt="EmbodiedSkillBench overview: module collection and skill assembly; an agent observing and acting in the simulator; trajectory logging and evaluation of success, actions, tokens, and time." width="100%"> |
| 25 | + |
| 26 | +<p><sub>Figure 2 from the manuscript. From heterogeneous embodied modules to reusable skills and trace-based evaluation.</sub></p> |
| 27 | + |
| 28 | +</div> |
| 29 | + |
| 30 | +> **Public code release.** The benchmark runtime, construction tools, Clean100 roster, and offline evaluator are available. The paper's 250 skill packages, skill indexes, and source-resource snapshots are **not included** in this release. Native control can run without them; skill-based experiments require separately supplied local packages. See [release scope](docs/release-scope.md). |
| 31 | +
|
| 32 | +## Overview |
| 33 | + |
| 34 | +EmbodiedSkillBench measures **when reusable embodied skills help an agent solve physical tasks, and at what execution cost**. It packages heterogeneous action models, perception interfaces, and procedural knowledge as agent-readable skills, then compares agents under a shared observation-action protocol. |
| 35 | + |
| 36 | +The paper studies three modes: |
| 37 | + |
| 38 | +| Mode | Agent behavior | |
| 39 | +| :--- | :--- | |
| 40 | +| **Native control** | Plans and acts through typed robot controls without a skill library. | |
| 41 | +| **Skill-only execution** | Solves the task with the supplied action, perception, and experience skills. | |
| 42 | +| **Agent-Directed Reuse (ADR)** | Retains native control and decides whether to invoke, modify, combine, or ignore skills. | |
| 43 | + |
| 44 | +The current public launcher supports native control and skill use with local packages. The standalone ADR and Jev experiment integrations are not part of this code release. |
| 45 | + |
| 46 | +<div align="center"> |
| 47 | + |
| 48 | +| Resource modules¹ | Embodied skills¹ | Perturbed task cases | Evaluation metrics | |
| 49 | +| :---: | :---: | :---: | :---: | |
| 50 | +| **1,052** | **250** | **100** | **4** | |
| 51 | +| Models, code, text & metadata | Action · Perception · Experience | 25 base tasks × 4 perturbations | Success · Actions · Tokens · Time | |
| 52 | + |
| 53 | +</div> |
| 54 | + |
| 55 | +<sub>¹ Corpus statistics describe the manuscript's repository; the corpus and skill assets are not distributed here.</sub> |
| 56 | + |
| 57 | +## Key results |
| 58 | + |
| 59 | +**Skills supply missing capability, but their value depends on what the agent can already do.** Under common selected skill provision, the first three agents improve in success and use fewer tokens. GPT-6-Astra succeeds more often with native control, exposing a tradeoff between skill restrictions and execution cost. |
| 60 | + |
| 61 | +| Agent | Native SR | With-skill SR | Change | Token reduction | |
| 62 | +| :--- | ---: | ---: | ---: | ---: | |
| 63 | +| GPT-5.6-Sol | 11% | **63%** | +52 pp | 85.0% | |
| 64 | +| Kimi-K3 | 43% | **62%** | +19 pp | 58.0% | |
| 65 | +| Claude-Opus-5 | 44% | **64%** | +20 pp | 76.7% | |
| 66 | +| GPT-6-Astra | **92%** | 61% | −31 pp | 75.5% | |
| 67 | + |
| 68 | +**Selective reuse recovers much of the stronger agent's success while reducing cost.** With GPT-6-Astra, ADR reaches 85% success using 14.1% fewer tokens and 11.9% less time than native control. Jev delegates local motion decisions and further reduces execution cost, with a larger success tradeoff. |
| 69 | + |
| 70 | +| GPT-6-Astra configuration | SR | Actions / task | Tokens / task (k) | Time / task (min) | |
| 71 | +| :--- | ---: | ---: | ---: | ---: | |
| 72 | +| Native control | **92%** | 16.59 | 2,300.6 | 20.52 | |
| 73 | +| Skill-only | 61% | **2.24** | **563.5** | **3.40** | |
| 74 | +| Agent-Directed Reuse | 85% | 12.87 | 1,976.3 | 18.09 | |
| 75 | +| With Jev skill | 81% | 20.90 | 1,142.0 | 8.74 | |
| 76 | + |
| 77 | +<sub>Reported manuscript results on Clean100, seed 26. SR is official task success; effort metrics are per-task means including failures. Tokens count recorded input and output, including cached input once. These values have not been regenerated from public rollout archives in this release.</sub> |
| 78 | + |
| 79 | +→ [Full result tables and experimental settings](docs/results.md) |
| 80 | + |
| 81 | +## Getting started |
| 82 | + |
| 83 | +Use Python 3.10–3.12; Python 3.12 is recommended. The source check and offline evaluator do not need a GPU or model API. |
| 84 | + |
| 85 | +```bash |
| 86 | +git clone https://github.com/HanZhi306/EmbodiedSkill-Bench.git |
| 87 | +cd EmbodiedSkill-Bench |
| 88 | + |
| 89 | +python3.12 -m venv ../esb-venv |
| 90 | +source ../esb-venv/bin/activate |
| 91 | +python -m pip install -e . |
| 92 | +esbench doctor |
| 93 | +``` |
| 94 | + |
| 95 | +`doctor` validates the frozen 100-task roster and reports `skill_assets: "not_included"` for this public checkout. If a local skill pool and index are supplied, it also checks their file inventory and hashes. It does not validate simulator installation or API access. |
| 96 | + |
| 97 | +Real simulator runs require **Linux, an NVIDIA GPU with EGL support, bubblewrap, LIBERO-Pro data/assets, and an authenticated agent CLI**. The paper's native interface uses **P3 joint-position control with an audited PyRoKi IK service**. The setup guide distinguishes it from the P2.5 OSC development interface. |
| 98 | + |
| 99 | +| I want to… | Start here | |
| 100 | +| :--- | :--- | |
| 101 | +| Run native control on LIBERO-Pro | [Installation and execution](RUN_BENCH.md) | |
| 102 | +| Compute metrics from archived episodes | [Offline evaluation](evaluation/README.md) | |
| 103 | +| Build skills from my own local resources | [Skill construction](docs/skill-construction.md) | |
| 104 | +| Understand what is available now | [Release scope](docs/release-scope.md) | |
| 105 | + |
| 106 | +## Code map |
| 107 | + |
| 108 | +```text |
| 109 | +EmbodiedSkill-Bench/ |
| 110 | +├── src/embodied_skill_bench/ |
| 111 | +│ ├── agents/ # CLI harnesses and isolated agent execution |
| 112 | +│ ├── experiments/ # Skill-use, native-control, and direct-VLA runners |
| 113 | +│ └── runtime/ # Typed controls, simulator adapters, and episode traces |
| 114 | +├── configs/ # Clean100 roster and dependency revisions |
| 115 | +├── evaluation/ # Per-task metrics, aggregation, and archived replay |
| 116 | +├── scripts/ # Resource collection, structuring, and materialization |
| 117 | +├── inputs/ # Editable construction examples |
| 118 | +├── prompts/ # Skill-authoring instructions |
| 119 | +├── protocol.json # Module-access and assembly-plan schemas |
| 120 | +└── docs/ # Results, release scope, and construction guide |
| 121 | +``` |
| 122 | + |
| 123 | +## Evaluation protocol |
| 124 | + |
| 125 | +Clean100 crosses **25 base tasks** with **language, object, task, and swap perturbations**. It contains 24 LIBERO-10, 28 Goal, 24 Object, and 24 Spatial cases, all at environment seed 26. The roster records task-file hashes and VLA seeds. |
| 126 | + |
| 127 | +| Metric | What is measured | |
| 128 | +| :--- | :--- | |
| 129 | +| **Success** | Official final simulator task status, reconciled with the episode summary. | |
| 130 | +| **Actions** | Accepted state-changing agent calls, deduplicated and reconciled with the runtime counter. | |
| 131 | +| **Tokens** | Recorded model input + output usage, with cached input counted once. | |
| 132 | +| **Time** | Agent launch to exit, including model calls, tools, and waiting. | |
| 133 | + |
| 134 | +Read-only queries and low-level simulator steps do not count as agent actions. Aggregation gives each task equal weight and retains failed tasks. Missing token usage stays unavailable; see the [evaluator guide](evaluation/README.md) for coverage and audit details. |
| 135 | + |
| 136 | +## Acknowledgments |
| 137 | + |
| 138 | +The benchmark builds on [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO), [LIBERO-Pro](https://huggingface.co/datasets/zhouxueyang/LIBERO-Pro), [RPent](https://github.com/RLinf/RPent), [PyRoKi](https://github.com/chungmin99/pyroki), and [CaP-X](https://github.com/capgym/cap-x), alongside the action, perception, and experience resources described in the manuscript. External code, data, and model checkpoints retain their upstream licenses and access requirements. |
| 139 | + |
| 140 | +The paper's full title is **Embodied Skill Bench: Benchmarking How Well Agents with Embodied Skills Perform on Physical Tasks**. A paper link and final citation metadata will be added when available. |
0 commit comments