Repository navigation
fix(lerobot): preserve source Parquet schemas during export - #777
Conversation
[Medium impact] The PR appears safe to merge; the previous test coverage gap is fixed.
|
Give observation.state field-level metadata so the existing check_metadata=True assertion also fails if the exporter drops field metadata, not only schema metadata.
kstonekuan
left a comment
There was a problem hiding this comment.
Could you remove the closing keyword for #776? The missing feature declarations reported there remain unresolved.
|
Thanks, done: it now says Part of #776. The missing feature declarations are fixed on a follow-up branch, together with a check that each exported file's schema matches its source. I'll open that once this one is in, since the repo allows one open PR at a time. |
kstonekuan
left a comment
There was a problem hiding this comment.
Thanks! Ruff, formatting, ty, and the full suite pass locally (2,629 passed, 47 skipped). LGTM, merging.
Summary
Part of #776 (the first finding). Exporting episodes from a LeRobot v3 dataset no longer changes its column types. The exported Parquet now keeps the source float32 widths, fixed-size array shapes, and field and schema metadata, which is what
examples/README.mdpromises. Before this, exports of three public datasets widened state, action, timestamp and reward columns to float64 while the workflow and verifier reported success.Why
_write_v3_repositoryrebuilt each data file withpa.Table.from_pylist(rows)and no schema, so Arrow inferred float64 for Python floats and variable-size lists for arrays. The fix reads each source data file's Arrow schema during the existing staging step and passes it tofrom_pylist. Selection, index renumbering and video copying are unchanged.EXPORT_VERSIONmoves fromlerobot-export-v1tolerobot-export-v2because the exported bytes change.examples/README.mdnow says to regenerate v3 exports made with v1. hflow's canonical episode format and transform behaviour version are untouched. The second finding in #776 (integer/boolean feature declarations missing from the exportedinfo.json) is left for a separate change.Validation
uv run pytest -q tests/test_lerobot_export.py::test_export_preserves_source_data_schema: fails onmain(schema equality: float32 to double, fixed-size to variable-size lists, metadata dropped), passes with this change.uv run pytest -q tests/test_lerobot_export.py tests/test_lerobot_workflow.py: 36 passed.uv run ruff check .,uv run ruff format --check .,uv run ty check: all pass.uv run pytest -q -n 2in a Debian container with its own FFmpeg 5.1: the same 39 tests fail onmainand on this branch (FFmpeg, packaging and converter tests). None fails only with this change, and the branch has one more passing test (the new one).main's own CI is green on3a99508, so I take those 39 to be environmental.lerobot/xarm_push_mediumepisodes 0–2 (revision in [Bug]: LeRobot v3 export silently widens float32 columns and drops Parquet schema metadata #776): 0/3 exported data files match the source schema onmain(the four columns listed in [Bug]: LeRobot v3 export silently widens float32 columns and drops Parquet schema metadata #776), 3/3 with this change. The original run across all three datasets in [Bug]: LeRobot v3 export silently widens float32 columns and drops Parquet schema metadata #776 went from 0/9 to 9/9.Checklist
uv run ruff check --fix,uv run ruff format, anduv run ty check.Open in [code]smith · Need help on this PR? Tag
@codesmith-botwith what you need.