Skip to content

Real-corpus test run: three LeRobot v3 datasets, two breakages and some rough edges #742

Closed

Description

@hoangchunghien

Following up on your note on #741, here is a real-corpus run: three public LeRobot Dataset v3 repositories from different robots, run through the steps of examples/lerobot/workflow.py (import, doctor, process with the default checks, curation, snapshot, v3 export, verify). Then I imported every episode of two of them one at a time.

hflow 0.2.21 at 8dd2a41, Python 3.13.9, FFmpeg 9.0.2, Ubuntu 24.04.

Dataset (pinned commit) Robot, cameras, rate Workflow result
lerobot/droid_100 (87301a2) Franka, 3 cameras, 15 fps, state/action 7/7 Episodes 0-4: import, doctor, process, curation and snapshot pass; v3 export fails
lerobot/aloha_static_coffee (b144896) ALOHA bimanual, 4 cameras, 50 fps, 14/14 Episodes 0-2: every step passes, verify included
lerobot/unitreeh1_fold_clothes (a0926b4) Unitree H1, 2 cameras, 50 fps, 19/40 Episodes 0-2: everything passes except v3 export

All three use AV1 sources. Doctor reported every converted episode as conforming, and curation kept all of them (5/5, 3/3, 3/3).

What broke

1. The PTS window admits the previous episode's last frame, so 15 of 100 DROID episodes are refused. Importing every episode of droid_100 one at a time, 85 import and 15 fail (47, 48, 52-55, 57, 67, 74-77, 79, 81, 83), all on observation.images.exterior_image_1_left:

timestamp disagreement at frame 0: parquet=0.000000s video_pts=-0.066666s

_relative_video_pts_times keeps packets with time >= start_seconds - frame_period. The previous episode's last frame sits exactly one period before the window start, so whether it is admitted depends on float rounding. Packet times relative to from_timestamp for that camera:

Episode packets just before the window result
46 -0.133333, -0.066667, 0.0 -0.066667 excluded, imports
47 -0.133333, -0.066666, -0.0 -0.066666 admitted as frame 0, refused
48 -0.133333, -0.066666, -0.0 refused
50 -0.133333, -0.066667, 0.0 imports

With the bound changed to start_seconds - frame_period / 2 (locally, as a probe), all 15 refused episodes and both controls import. I have not run the test suite against that change, so treat it as a pointer, not a patch.

2. One broken source episode blocks every v3 export. In unitreeh1_fold_clothes, observation.images.cam_left/chunk-000/file-000.mp4 is 370 s long with 18,500 packets, while the episode metadata and cam_right cover 380 s and 19,000. Episode 37 has no left-camera video. That is a defect in the published dataset, and refusing episode 37 is right. But the v3 export materializes the whole source archive through the importer (_materialize_source_archive), so exporting a selection of episodes 0-2 fails on episode 37. The error also does not name the episode:

video has 1 packets at or after the window start but the episode has 500 data rows: .../cam_left/chunk-000/file-000.mp4

Something like "episode 37: cam_left video ends at 370.0 s, the episode window is 370.0-380.0 s" would have saved me the ffprobe. DROID's export fails the same way, on the episodes from item 1, though its selection was 0-4.

What felt awkward

  • --episode-index cannot be repeated. --camera says "repeat for multiple", so I passed --episode-index five times. The import converted only the last one, with no warning. Either accepting several values or refusing the repeat would help.
  • Image-only datasets get a misleading error. hflow import lerobot --repo lerobot/libero_10_image (no video features) says camera key 'observation.image' not found in dataset. Available: []. I never named that camera; the default did. "This dataset has no video features (image-only datasets are not supported)" would say what is going on.
  • The workflow example is tied to its own manifest. workflow.py always reads source-manifest.json next to itself, so running it on another corpus means editing the repository. A --source-manifest option would make it a reusable harness. I called its step functions from a small driver instead.
  • Export time follows the archive, not the selection. Exporting three ALOHA episodes took 433 s, because all 50 episodes × 4 cameras are re-imported first. Same cause as item 2.
  • One-episode imports download whole shared files. Importing one DROID episode fetched 443 MB, because Dataset v3 packs many episodes into each video file. That is how the format works, but a line in import-lerobot-v3.md would set expectations.

What worked well

  • Import errors for a wrong camera or an out-of-range episode come back in about 5 s with a clear message, before any download.
  • The revision resolves to an immutable commit and lands in prepared-manifest.json with per-episode content ids.
  • AV1 to H.264 conversion handled 3 and 4 camera episodes at 15 and 50 fps without trouble, and process plus curation took seconds.
  • ALOHA's export round-tripped and verify passed.

I can turn item 1 into a PR with a regression test if you want it, once there is a slot.

Activity

  1. kstonekuan commented on Oct 29, 2026

    @kstonekuan
    Contributor

    Thanks for the run. #756 fixes repeated episode selection, #765 fixes the image-only error, and #745 fixes the PTS boundary and selected-episode export. This stays open for the remaining workflow and documentation rough edges.

  2. ayam04 commented on Oct 30, 2026

    @ayam04
    Contributor

    Taking the image-only dataset item: hflow import lerobot --repo <image-only> currently fails on camera key 'observation.image' not found in dataset. Available: [], naming a key the default supplied rather than the dataset's missing video features. I'll refuse it up front with that reason. The other rough edges in this report (workflow --source-manifest, the docs line) stay open.

  3. ayam04 commented on Nov 4, 2026

    @ayam04
    Contributor

    Thanks for the corpus run, the packet table made this one obvious. Your probe was right: #771 moves the window bound to half a frame period, which is the widest tolerance that still excludes the previous episode's last packet and still admits a first packet that quantization moved early. It also puts the episode index in the refusal message, since that is what made the second breakage expensive to read. Your 15 droid episodes are the regression test. I left the v3 export materialisation alone in #771, that one needs a call on whether the archive imports the selection or validates it first, and I did not want to guess at your intent.

  4. kstonekuan commented on Nov 11, 2026

    @kstonekuan
    Contributor

    Resolved in #745, #756, #765, and #766. Thanks for the corpus run!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions