Skip to content

Commit 33ed952

Browse files
HanZhiHanZhi
authored andcommitted
Initial public code release for EmbodiedSkillBench
0 parents  commit 33ed952

82 files changed

Lines changed: 25302 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.githooks/pre-commit‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
#!/bin/sh
2+
set -eu
3+
exec python3 scripts/check_public_release.py

‎.github/workflows/checks.yml‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
name: Code checks
2+
3+
on:
4+
push:
5+
pull_request:
6+
7+
permissions:
8+
contents: read
9+
10+
jobs:
11+
public-code:
12+
runs-on: ubuntu-latest
13+
strategy:
14+
fail-fast: false
15+
matrix:
16+
python: ['3.10', '3.12']
17+
steps:
18+
- uses: actions/checkout@v4
19+
- uses: actions/setup-python@v5
20+
with:
21+
python-version: ${{ matrix.python }}
22+
- name: Check public release contents
23+
run: python scripts/check_public_release.py
24+
- name: Install editable source package
25+
run: python -m pip install -e .
26+
- name: Check public task roster
27+
run: esbench doctor
28+
- name: Test native release and evaluator contracts
29+
run: python -m unittest discover -s tests -v

‎.gitignore‎

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# Embodied skill packages and their duplicated source snapshots stay local.
2+
**/skill-pool/
3+
**/skills-index.json
4+
**/SKILLS.md
5+
**/data/
6+
**/datasets/
7+
**/01_cards.json
8+
**/01_inventory.json
9+
**/02_groups.json
10+
**/02_contracts.json
11+
**/03_handoff.json
12+
13+
# Environments, provider assets, and experiment output.
14+
.venv/
15+
venv/
16+
.deps/
17+
.runtime/
18+
runtime-local/
19+
output/
20+
outputs/
21+
runs/
22+
episodes/
23+
checkpoints/
24+
models/
25+
*.safetensors
26+
*.pt
27+
*.pth
28+
*.ckpt
29+
*.npz
30+
*.npy
31+
*.pkl
32+
*.pickle
33+
*.zip
34+
*.tar
35+
*.tar.gz
36+
*.log
37+
38+
# Credentials and private configuration.
39+
.env
40+
.env.*
41+
.hf_token
42+
*.pem
43+
*.key
44+
credentials*.json
45+
config.local.*
46+
configs/local/
47+
.codex/
48+
.claude/
49+
.kimi/
50+
51+
# Build and editor output.
52+
__pycache__/
53+
*.py[cod]
54+
*.egg-info/
55+
build/
56+
dist/
57+
.pytest_cache/
58+
.ruff_cache/
59+
.DS_Store
60+
.idea/
61+
.vscode/

‎README.md‎

Lines changed: 140 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
<div align="center">
2+
3+
# EmbodiedSkillBench
4+
5+
### Benchmarking How Well Agents with Embodied Skills Perform on Physical Tasks
6+
7+
**Reusable embodied capabilities. Closed-loop physical interaction. Measurable skill utility.**
8+
9+
<p>
10+
<a href="#overview">Overview</a> &nbsp;·&nbsp;
11+
<a href="#key-results">Results</a> &nbsp;·&nbsp;
12+
<a href="#getting-started">Get started</a> &nbsp;·&nbsp;
13+
<a href="RUN_BENCH.md">Run the benchmark</a> &nbsp;·&nbsp;
14+
<a href="evaluation/README.md">Evaluation</a>
15+
</p>
16+
17+
<p>
18+
<img src="https://img.shields.io/badge/Python-3.10–3.12-3776AB?style=flat-square&logo=python&logoColor=white" alt="Python 3.10 to 3.12">
19+
<img src="https://img.shields.io/badge/Benchmark-LIBERO--Pro-0A9A85?style=flat-square" alt="LIBERO-Pro benchmark">
20+
<img src="https://img.shields.io/badge/Tasks-Clean100-6654C0?style=flat-square" alt="Clean100 tasks">
21+
<a href="https://github.com/HanZhi306/EmbodiedSkill-Bench/actions/workflows/checks.yml"><img src="https://github.com/HanZhi306/EmbodiedSkill-Bench/actions/workflows/checks.yml/badge.svg" alt="Code checks"></a>
22+
</p>
23+
24+
<img src="docs/images/benchmark-overview.png" alt="EmbodiedSkillBench overview: module collection and skill assembly; an agent observing and acting in the simulator; trajectory logging and evaluation of success, actions, tokens, and time." width="100%">
25+
26+
<p><sub>Figure 2 from the manuscript. From heterogeneous embodied modules to reusable skills and trace-based evaluation.</sub></p>
27+
28+
</div>
29+
30+
> **Public code release.** The benchmark runtime, construction tools, Clean100 roster, and offline evaluator are available. The paper's 250 skill packages, skill indexes, and source-resource snapshots are **not included** in this release. Native control can run without them; skill-based experiments require separately supplied local packages. See [release scope](docs/release-scope.md).
31+
32+
## Overview
33+
34+
EmbodiedSkillBench measures **when reusable embodied skills help an agent solve physical tasks, and at what execution cost**. It packages heterogeneous action models, perception interfaces, and procedural knowledge as agent-readable skills, then compares agents under a shared observation-action protocol.
35+
36+
The paper studies three modes:
37+
38+
| Mode | Agent behavior |
39+
| :--- | :--- |
40+
| **Native control** | Plans and acts through typed robot controls without a skill library. |
41+
| **Skill-only execution** | Solves the task with the supplied action, perception, and experience skills. |
42+
| **Agent-Directed Reuse (ADR)** | Retains native control and decides whether to invoke, modify, combine, or ignore skills. |
43+
44+
The current public launcher supports native control and skill use with local packages. The standalone ADR and Jev experiment integrations are not part of this code release.
45+
46+
<div align="center">
47+
48+
| Resource modules¹ | Embodied skills¹ | Perturbed task cases | Evaluation metrics |
49+
| :---: | :---: | :---: | :---: |
50+
| **1,052** | **250** | **100** | **4** |
51+
| Models, code, text & metadata | Action · Perception · Experience | 25 base tasks × 4 perturbations | Success · Actions · Tokens · Time |
52+
53+
</div>
54+
55+
<sub>¹ Corpus statistics describe the manuscript's repository; the corpus and skill assets are not distributed here.</sub>
56+
57+
## Key results
58+
59+
**Skills supply missing capability, but their value depends on what the agent can already do.** Under common selected skill provision, the first three agents improve in success and use fewer tokens. GPT-6-Astra succeeds more often with native control, exposing a tradeoff between skill restrictions and execution cost.
60+
61+
| Agent | Native SR | With-skill SR | Change | Token reduction |
62+
| :--- | ---: | ---: | ---: | ---: |
63+
| GPT-5.6-Sol | 11% | **63%** | +52 pp | 85.0% |
64+
| Kimi-K3 | 43% | **62%** | +19 pp | 58.0% |
65+
| Claude-Opus-5 | 44% | **64%** | +20 pp | 76.7% |
66+
| GPT-6-Astra | **92%** | 61% | −31 pp | 75.5% |
67+
68+
**Selective reuse recovers much of the stronger agent's success while reducing cost.** With GPT-6-Astra, ADR reaches 85% success using 14.1% fewer tokens and 11.9% less time than native control. Jev delegates local motion decisions and further reduces execution cost, with a larger success tradeoff.
69+
70+
| GPT-6-Astra configuration | SR | Actions / task | Tokens / task (k) | Time / task (min) |
71+
| :--- | ---: | ---: | ---: | ---: |
72+
| Native control | **92%** | 16.59 | 2,300.6 | 20.52 |
73+
| Skill-only | 61% | **2.24** | **563.5** | **3.40** |
74+
| Agent-Directed Reuse | 85% | 12.87 | 1,976.3 | 18.09 |
75+
| With Jev skill | 81% | 20.90 | 1,142.0 | 8.74 |
76+
77+
<sub>Reported manuscript results on Clean100, seed 26. SR is official task success; effort metrics are per-task means including failures. Tokens count recorded input and output, including cached input once. These values have not been regenerated from public rollout archives in this release.</sub>
78+
79+
→ [Full result tables and experimental settings](docs/results.md)
80+
81+
## Getting started
82+
83+
Use Python 3.10–3.12; Python 3.12 is recommended. The source check and offline evaluator do not need a GPU or model API.
84+
85+
```bash
86+
git clone https://github.com/HanZhi306/EmbodiedSkill-Bench.git
87+
cd EmbodiedSkill-Bench
88+
89+
python3.12 -m venv ../esb-venv
90+
source ../esb-venv/bin/activate
91+
python -m pip install -e .
92+
esbench doctor
93+
```
94+
95+
`doctor` validates the frozen 100-task roster and reports `skill_assets: "not_included"` for this public checkout. If a local skill pool and index are supplied, it also checks their file inventory and hashes. It does not validate simulator installation or API access.
96+
97+
Real simulator runs require **Linux, an NVIDIA GPU with EGL support, bubblewrap, LIBERO-Pro data/assets, and an authenticated agent CLI**. The paper's native interface uses **P3 joint-position control with an audited PyRoKi IK service**. The setup guide distinguishes it from the P2.5 OSC development interface.
98+
99+
| I want to… | Start here |
100+
| :--- | :--- |
101+
| Run native control on LIBERO-Pro | [Installation and execution](RUN_BENCH.md) |
102+
| Compute metrics from archived episodes | [Offline evaluation](evaluation/README.md) |
103+
| Build skills from my own local resources | [Skill construction](docs/skill-construction.md) |
104+
| Understand what is available now | [Release scope](docs/release-scope.md) |
105+
106+
## Code map
107+
108+
```text
109+
EmbodiedSkill-Bench/
110+
├── src/embodied_skill_bench/
111+
│ ├── agents/ # CLI harnesses and isolated agent execution
112+
│ ├── experiments/ # Skill-use, native-control, and direct-VLA runners
113+
│ └── runtime/ # Typed controls, simulator adapters, and episode traces
114+
├── configs/ # Clean100 roster and dependency revisions
115+
├── evaluation/ # Per-task metrics, aggregation, and archived replay
116+
├── scripts/ # Resource collection, structuring, and materialization
117+
├── inputs/ # Editable construction examples
118+
├── prompts/ # Skill-authoring instructions
119+
├── protocol.json # Module-access and assembly-plan schemas
120+
└── docs/ # Results, release scope, and construction guide
121+
```
122+
123+
## Evaluation protocol
124+
125+
Clean100 crosses **25 base tasks** with **language, object, task, and swap perturbations**. It contains 24 LIBERO-10, 28 Goal, 24 Object, and 24 Spatial cases, all at environment seed 26. The roster records task-file hashes and VLA seeds.
126+
127+
| Metric | What is measured |
128+
| :--- | :--- |
129+
| **Success** | Official final simulator task status, reconciled with the episode summary. |
130+
| **Actions** | Accepted state-changing agent calls, deduplicated and reconciled with the runtime counter. |
131+
| **Tokens** | Recorded model input + output usage, with cached input counted once. |
132+
| **Time** | Agent launch to exit, including model calls, tools, and waiting. |
133+
134+
Read-only queries and low-level simulator steps do not count as agent actions. Aggregation gives each task equal weight and retains failed tasks. Missing token usage stays unavailable; see the [evaluator guide](evaluation/README.md) for coverage and audit details.
135+
136+
## Acknowledgments
137+
138+
The benchmark builds on [LIBERO](https://github.com/Lifelong-Robot-Learning/LIBERO), [LIBERO-Pro](https://huggingface.co/datasets/zhouxueyang/LIBERO-Pro), [RPent](https://github.com/RLinf/RPent), [PyRoKi](https://github.com/chungmin99/pyroki), and [CaP-X](https://github.com/capgym/cap-x), alongside the action, perception, and experience resources described in the manuscript. External code, data, and model checkpoints retain their upstream licenses and access requirements.
139+
140+
The paper's full title is **Embodied Skill Bench: Benchmarking How Well Agents with Embodied Skills Perform on Physical Tasks**. A paper link and final citation metadata will be added when available.

0 commit comments

Comments
 (0)