Skip to content

Latest commit

 

History

History
50 lines (38 loc) · 2.61 KB

README.md

File metadata and controls

50 lines (38 loc) · 2.61 KB

Human-in-the-loop with Portal and MakerMods Metal arms

Run Dream-1, take over with the leader arms when it needs help, and record the corrected attempts. Train ACT on those recordings, then repeat the correction loop.

The tutorial

Follow these chapters in order:

  1. Set up the rig and check human control.
  2. Run Dream-1 and record corrections.
  3. Train ACT on the corrected takes.

To collect human-only demonstrations instead, see the alternative recording path.

You need two Metal leaders, two Metal followers, three USB cameras, Linux hardware hosts, Python 3.12, uv, and a LiveKit server. ACT training also needs a CUDA machine. Dream-1 needs a Dream Labs API key.

Start with Dream-1 zero-shot and correct it, or collect teleop demos. Convert the archives to a LeRobot dataset and train ACT on them, distilling Dream-1 into a small local policy. Run ACT while you correct it; corrections feed the next training run.
Chapter Purpose
00. Overview Roles and vocabulary
01. Setup Connect the rig and check human control
02. Dream-1 Run, correct, record, and convert
03. ACT Train, run, correct, and retrain

Read these after completing the walkthrough, or when you need to change the code:

Chapter Purpose
04. Shared schema Cameras, state, and actions
05. Robot loop Publish observations and apply actions
06. Operator loop Send human actions and mark takes
07. Recording Align actions with observations
08. Handoff Switch between human and policy
09. Reference Settings, alternatives, and troubleshooting

Metal arms have no brakes. Start the robot with the followers resting: that is where they park. p parks them (walk back, switch off) or powers them up, and the robot parks before every exit. A fault never switches the motors off.

The runnable peers are in src/; policies are in src/policies/. All peers share portal.yaml. For checks without hardware, see test/README.md.