Skip to content

[Bug]: LeRobot v3 export silently widens float32 columns and drops Parquet schema metadata #776

Closed

Description

@Leo-Y-Zhang

Version or commit

hflow 0.2.21, 3a99508

Environment

Linux x86-64, Python 3.11, FFmpeg 7.0

Minimal reproduction

Found on a real-corpus run of the documented examples/lerobot/workflow.py, like #742 but on three different public LeRobot v3 datasets. It used episodes 0–2 of each, with all declared cameras:

Dataset (pinned revision) Setup
lerobot/columbia_cairlab_pusht_real real PushT, wrist + external 240×320 cameras, 10 Hz
lerobot/ucsd_pick_and_place_dataset pick-and-place, one 224×224 camera, 5 Hz
lerobot/xarm_push_medium simulated xArm push, one 84×84 camera, 15 Hz
# manifest.json: {"repository": ..., "revision": ..., "cameras": [...], "episodes": [0, 1, 2]}
uv run python examples/lerobot/workflow.py --data-dir <dir> --source-manifest manifest.json

Then compare a source data file with its export:

import pyarrow.parquet as pq
src = pq.read_schema("<dir>/_lerobot_cache/<repo>/data/chunk-000/file-000.parquet")
out = pq.read_schema("<dir>/v3/data/chunk-000/file-000.parquet")
print(src.equals(out, check_metadata=True))  # False on all 9 exported files

The same thing reproduces on synthetic data with the existing fake corpus in tests/test_lerobot_export.py: give the source float32 columns and fixed-size arrays, export, and compare the schemas.

Expected behavior

examples/README.md says the exporter slices data rows from the source parquets "so cameras, feature schema, dtypes, shapes, and frame timing match the source". The exported Parquet should keep the source column types, array shapes and metadata.

Actual behavior

Every stage returns success and the verifier reports OK, but on all nine exported files the same columns change type:

observation.state: list<float> -> list<double>
action:            list<float> -> list<double>
timestamp:         float -> double
next.reward:       float -> double

Field and schema metadata are dropped, and fixed-size lists become variable-size lists. For example, the xArm export's meta/info.json declares action as float32 with shape [3] but writes float64 arrays.

The cause is that _write_v3_repository rebuilds each data file with pa.Table.from_pylist(rows) and no schema, so Arrow infers float64 for Python floats and variable-size lists for arrays. I have a fix and will open a PR that passes the source schema through.

Additional context

A second, separate finding from the same run: episode_index, frame_index, index, task_index and next.done stay in the exported Parquet, but on all three datasets they disappear from the features registry in the exported meta/info.json. No values change, so this is different from the existing task-index and task-registry issues. I haven't checked whether the official LeRobot loader minds, and I'm leaving it for a separate change.

Activity

  1. github-actions commented on Nov 10, 2026

    @github-actions

    👋 Hi @Leo-Y-Zhang — thank you for opening your first issue on HFlow!

    We appreciate you taking the time to report this. A maintainer will take a look as soon as possible.

    In the meantime, feel free to:

    We are glad to have you here! 🙌

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions