Repository navigation
Description
VERSION
8dd2a41 (latest main)
Environment
OS: macOS / Linux (Ubuntu 22.04 / 24.04)
Python: 3.11+ (tested on Python 3.14.7)
FFmpeg: System FFmpeg with libx264 support
Minimal production
uv run python -c "
import tempfile, subprocess
from pathlib import Path
from hflow.importers.lerobot import _transcode_mp4_to_h264
from hflow.video import split_annex_b_stream
from hflow.ffmpeg import ffmpeg_path
with tempfile.TemporaryDirectory() as tmp:
mp4_path = Path(tmp) / 'test.mp4'
subprocess.run([
str(ffmpeg_path()), '-y', '-f', 'lavfi', '-i', 'testsrc=duration=0.5:size=64x64:rate=10',
'-c:v', 'libx264', str(mp4_path)
], check=True, capture_output=True)
# Transcode MP4 to H.264 access units using LeRobot importer
units = _transcode_mp4_to_h264(mp4_path, gop_seconds=1.0, frames_per_second=10.0)
print(f'Total units: {len(units)}')
for i, u in enumerate(units):
print(f'Unit {i} start hex: {u[:12].hex()}')
try:
split_annex_b_stream(u)
except Exception as e:
print(f'Unit {i} validation failed: {type(e).__name__}: {e}')
"Expected behavior
According to
src/hflow/format.py
and docs/ARCHITECTURE.md:
"Camera streams are in-band H.264 in foxglove.CompressedVideo messages (protobuf-encoded): Annex B byte stream, SPS/PPS on every keyframe, no B-frames, one decodable access unit per message."
Every access unit extracted from FFmpeg by _transcode_mp4_to_h264 should be a conforming Annex B byte sequence starting with a 3-byte or 4-byte NAL start code prefix (00 00 00 01 09... for the AUD NAL unit). Passing each unit to
hflow.video.split_annex_b_stream
should succeed with exactly one access unit.
Actual behavior
Total units: 5
Unit 0 start hex: 100000000167f4000a919641
Unit 0 validation failed: ValueError: no access-unit delimiter (AUD, NAL type 9) found; splitting requires a stream encoded with aud=1
Unit 1 start hex: 30000001419a3b119f0a9f2a
Unit 1 validation failed: ValueError: no access-unit delimiter (AUD, NAL type 9) found; splitting requires a stream encoded with aud=1
Unit 2 start hex: 30000001419a4f0864ca6119
Unit 2 validation failed: ValueError: no access-unit delimiter (AUD, NAL type 9) found; splitting requires a stream encoded with aud=1
Unit 3 start hex: 30000001419a727843c994c0
Unit 3 validation failed: ValueError: no access-unit delimiter (AUD, NAL type 9) found; splitting requires a stream encoded with aud=1
Unit 4 start hex: 30000001419a927843c994c0
Unit 4 validation failed: ValueError: no access-unit delimiter (AUD, NAL type 9) found; splitting requires a stream encoded with aud=1
Additional context and proposed fix
def _split_h264_by_aud(h264_data: bytes) -> list[bytes]:
"""Split H.264 stream on AUD (0x00000109) boundaries."""
units: list[bytes] = []
pattern = b"\x00\x00\x00\x01\x09"
offset = 0
while True:
boundary_index = h264_data.find(pattern, offset)
if boundary_index == -1:
break
if boundary_index > offset:
units.append(h264_data[offset:boundary_index])
offset = boundary_index + 5
if offset < len(h264_data):
units.append(h264_data[offset:])
return unitsWhen a boundary is found, offset = boundary_index + 5 skips past the start code and AUD NAL type byte, cutting them off from units.append(h264_data[offset:boundary_index]).
Existing converter unit tests missed this because tests/test_lerobot_converter.py stubs _convert_single_episode with _install_publish_through_convert (fake_convert writing dummy bytes episode-{episode_index}), so _transcode_mp4_to_h264 was never exercised on real video frames in the converter test suite.
Proposed Fix
hflow already has a robust, spec-compliant Annex B splitter in
hflow.video.split_annex_b_stream
. In
src/hflow/importers/lerobot.py
:
diff
- return _split_h264_by_aud(completed_process.stdout)
- from hflow.video import split_annex_b_stream
- return [unit.data for unit in split_annex_b_stream(completed_process.stdout)]
Activity
fix(lerobot): preserve AUD start codes when transcoding video (Hebbia…
fix(lerobot): preserve AUD start codes when transcoding video (#743) (#…