Skip to content

fix(lerobot): open dataset videos only as local MP4 files - #805

Closed
cursor[bot] wants to merge 2 commits into
mainfrom
cursor/lerobot-ffmpeg-input-allowlist-1c8d
Closed

cursor[bot] wants to merge 2 commits into
mainfrom
cursor/lerobot-ffmpeg-input-allowlist-1c8d

Conversation

@cursor

@cursor cursor Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Summary

The LeRobot importer now opens each downloaded camera file with -protocol_whitelist file and -format_whitelist mov,mp4,m4a,3gp,3g2,mj2, placed before -i (which now names file:<resolved path>). This applies to both the transcode (_transcode_mp4_to_h264) and the packet-time probe (_get_video_pts_times, which is also used by _relative_video_pts_times). A dataset whose camera file is an HLS playlist, a concat script, or anything else that is not ISO BMFF is refused. ffmpeg refuses it before that demuxer reads its header, so no segment it names is opened or requested. The import fails with the existing RuntimeError from the ffmpeg step, and nothing is published.

Legitimate LeRobot MP4 videos decode exactly as before, so the converter version is unchanged.

Why

The dataset publisher controls both video_path in meta/info.json and the downloaded bytes. ffmpeg and ffprobe used to run with their default protocols and demuxers. A playlist under that name made the import host open other local files or request URLs listed inside it, and another local video could be transcoded into the published episode.

A protocol whitelist alone, like the file whitelist that video import uses, does not stop this. Playlist segments can themselves be file: URLs or relative paths. The demuxer whitelist is what binds the check to the downloaded object: ffmpeg probes that file's bytes and refuses any other demuxer in the same process that would act on them. Nothing runs between a separate check and the open. The existing relative-path check on the Hub filename in _download_file is unchanged.

Scope: I searched every other ffmpeg/ffprobe call (importers/video.py, media.py, source_sampling.py, blur.py, episode.py, window_measurements.py, _video_measurements/*, ffmpeg/_contact_sheet.py, video.py). They open files the operator supplies locally, or MP4s and H.264 that HFlow writes itself, not objects a remote publisher controls. The LeRobot importer is the only path where remote content reaches ffmpeg. The MP4 demuxer's external data references stay off (its enable_drefs default), and only the file protocol is allowed in any case.

Regression test

tests/test_lerobot_converter.py::test_import_refuses_a_non_mp4_camera_file_without_publishing runs import_lerobot_dataset end to end against a benign, decodable Matroska camera video. It serves the video under an .mkv video_path and under the default .mp4 template, and covers both the whole-file and windowed transcode paths. It asserts the import raises and writes neither landing/ nor prepared-manifest.json.

Validation

The playlist tests that prove the fix are in #802. They are not included in this PR.

  • uv run pytest -q tests/test_lerobot_playlist_segments.py (from test: refuse LeRobot playlist segments that point outside the cache #802)
    • Without the fix: 4 failed. inotify saw the outside video opened for both file-url and relative segments, with and without a video window.
    • With the fix applied: 4 passed.
  • uv run pytest -q tests/test_lerobot_converter.py::test_import_refuses_a_non_mp4_camera_file_without_publishing
    • With the fix: 4 passed.
    • With src/hflow/importers/lerobot.py reverted to main: 4 failed (DID NOT RAISE). The Matroska file was converted and published.
  • An ad hoc variant check, not committed, also passed (25 cases). It covered HLS with file: URL and absolute-path segments, and an ffconcat script, each under .mp4, .m3u8, .txt and .ffconcat names, with and without a window: no outside-file open, import refused, no landing/. An HLS playlist with an http://127.0.0.1 segment got no request to the listener. Run directly, both the managed ffmpeg 8.1.2 and system ffmpeg 6.1 report Format not on whitelist for the hls and concat demuxers and still open a real MP4.
  • uv run pytest -q -n 4 tests/test_lerobot*.py tests/test_ffmpeg_standard_input.py tests/test_cli.py on this branch: 181 passed.
  • uv run pytest -q -n 4 (full suite) on the fix commit: 2602 passed, 8 skipped, 36 failed. All 36 failures are in tests/test_packaging.py and tests/test_packaging_manifest_parsing.py, where the native-overlay C compile fails in this VM. The identical 36 fail on unmodified main.
  • uv run ruff check, uv run ruff format --check, uv run ty check: all pass.

Checklist

  • I added or updated outcome-focused tests for changed business logic.
  • I updated documentation for changed behavior, flags, formats, or requirements. (None needed: LeRobot v3 stores camera video as MP4.)
  • I ran uv run ruff check --fix, uv run ruff format, and uv run ty check.
  • I ran the relevant pytest suite.
  • I did not add recordings, generated media, credentials, private URLs, or runtime artifacts.
  • I preserved stored-data compatibility or documented an explicit version change.
Open in Web Open in Cursor 

Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Note

High Risk
Hardens the only path where remote Hugging Face dataset bytes reach FFmpeg, closing a vector where malicious camera media could open arbitrary local files or trigger network fetches during import.

Reviewed by Cursor Bugbot for commit 728a1a5. Bugbot is set up for automated code reviews on this repo. Configure here.

The LeRobot importer passed each downloaded camera file to ffmpeg and
ffprobe with FFmpeg's default protocols and demuxers. A dataset publisher
controls the file name and bytes, so a playlist or concat script made
the import open other local files (and reach URLs) named inside it, and
could transcode another local video into the published episode.

Open those files with a file-only protocol whitelist and a format
whitelist limited to the ISO BMFF (MP4) demuxer, so FFmpeg refuses a
probed playlist or script before it reads its header.
@greptile-apps

greptile-apps Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking issues were found.

Summary

The LeRobot importer now limits FFmpeg and ffprobe to local files and MP4-family formats.

  • LeRobot imports only read camera videos as local MP4-family files.

Reviews (2) · Last reviewed commit: "test(lerobot): refuse a non-MP4 camera f..." · Reviewed by Greptile

greptile-apps[bot]
greptile-apps Bot previously approved these changes Nov 20, 2026

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you include a regression test in this PR that uses a benign non-MP4 camera fixture and checks that import rejects it without publishing an episode?

@kstonekuan

Copy link
Copy Markdown
Contributor

@cursor Could you update this PR with a regression test using a benign non-MP4 camera fixture? Check that import rejects it without publishing an episode.

@cursor

cursor Bot commented Nov 20, 2026 •

Copy link
Copy Markdown
Author

I added the regression test you asked for and pushed it to the PR. With the fix it passes; with the importer reverted to main it fails, so it catches the bug coming back.

The test is test_import_refuses_a_non_mp4_camera_file_without_publishing in tests/test_lerobot_converter.py:

  • Fixture: a harmless, playable Matroska (MKV) camera video. The test runs a full import_lerobot_dataset call against it.
  • File name: the video is served under an .mkv path and also under the default .mp4 path. The second case shows the importer rejects the file based on its contents, not its name.
  • Both ffmpeg paths: each name is tried with the whole-file transcode and with the windowed (seek-first) transcode, so 4 cases in total.
  • What it checks: the import fails with ffmpeg's transcode error, and neither landing/ nor prepared-manifest.json is written.

Results:

  • With the fix: all 4 cases pass.
  • With src/hflow/importers/lerobot.py reverted to main: all 4 fail. The importer converted the MKV and published an episode.
  • LeRobot, ffmpeg standard-input and CLI tests: 181 passed.
  • Lint and type checks (ruff check, ruff format, ty check): pass.

I also updated the PR description to describe the new test and its results. The bot footers that had been added below my section are unchanged.

#805

Open in Web Open in Cursor 

A decodable Matroska camera video, served under an .mkv name and under
the default .mp4 template, must make the import fail without writing
landing/ or prepared-manifest.json, on both the whole-file and windowed
transcode paths.
@greptile-apps
greptile-apps Bot dismissed their stale review November 20, 2026 13:20

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new regression passes, and Ruff, ty, and the full suite passed locally.

@kstonekuan

Copy link
Copy Markdown
Contributor

Landed on main in b9f3e35.

@kstonekuan kstonekuan closed this Nov 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants