Skip to content

fix(lerobot): preserve source Parquet schemas during export - #777

Merged
kstonekuan merged 2 commits into
Hebbian-Robotics:mainfrom
Leo-Y-Zhang:lerobot-export-schema
Nov 12, 2026
Merged

kstonekuan merged 2 commits into
Hebbian-Robotics:mainfrom
Leo-Y-Zhang:lerobot-export-schema

Conversation

@Leo-Y-Zhang

@Leo-Y-Zhang Leo-Y-Zhang commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Part of #776 (the first finding). Exporting episodes from a LeRobot v3 dataset no longer changes its column types. The exported Parquet now keeps the source float32 widths, fixed-size array shapes, and field and schema metadata, which is what examples/README.md promises. Before this, exports of three public datasets widened state, action, timestamp and reward columns to float64 while the workflow and verifier reported success.

Why

_write_v3_repository rebuilt each data file with pa.Table.from_pylist(rows) and no schema, so Arrow inferred float64 for Python floats and variable-size lists for arrays. The fix reads each source data file's Arrow schema during the existing staging step and passes it to from_pylist. Selection, index renumbering and video copying are unchanged.

EXPORT_VERSION moves from lerobot-export-v1 to lerobot-export-v2 because the exported bytes change. examples/README.md now says to regenerate v3 exports made with v1. hflow's canonical episode format and transform behaviour version are untouched. The second finding in #776 (integer/boolean feature declarations missing from the exported info.json) is left for a separate change.

Validation

Checklist

  • I added or updated outcome-focused tests for changed business logic.
  • I updated documentation for changed behavior, flags, formats, or requirements.
  • I ran uv run ruff check --fix, uv run ruff format, and uv run ty check.
  • I ran the relevant pytest suite.
  • I did not add recordings, generated media, credentials, private URLs, or runtime artifacts.
  • I preserved stored-data compatibility or documented an explicit version change.

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.

@greptile-apps

greptile-apps Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; the previous test coverage gap is fixed.

Summary

The exporter keeps each source data file's Arrow schema when writing selected episodes.

  • LeRobot exports keep the source data types and array shapes.

Reviews (2) · Last reviewed commit: "test(lerobot): cover field metadata in t..." · Reviewed by Greptile

Comment thread tests/test_lerobot_export.py Resolved
Give observation.state field-level metadata so the existing
check_metadata=True assertion also fails if the exporter drops
field metadata, not only schema metadata.

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you remove the closing keyword for #776? The missing feature declarations reported there remain unresolved.

@Leo-Y-Zhang

Copy link
Copy Markdown
Contributor Author

Thanks, done: it now says Part of #776. The missing feature declarations are fixed on a follow-up branch, together with a check that each exported file's schema matches its source. I'll open that once this one is in, since the repo allows one open PR at a time.

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Ruff, formatting, ty, and the full suite pass locally (2,629 passed, 47 skipped). LGTM, merging.

@kstonekuan
kstonekuan merged commit 93ad1b5 into Hebbian-Robotics:main Nov 12, 2026
3 checks passed
@kstonekuan kstonekuan mentioned this pull request Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants