Skip to content

[Bug]: LeRobot converter _split_h264_by_aud strips AUD start codes and corrupts access unit boundaries #743

Closed

Description

@shobhitagnihotri69

VERSION

8dd2a41 (latest main)

Environment

OS: macOS / Linux (Ubuntu 22.04 / 24.04)
Python: 3.11+ (tested on Python 3.14.7)
FFmpeg: System FFmpeg with libx264 support

Minimal production

uv run python -c "
import tempfile, subprocess
from pathlib import Path
from hflow.importers.lerobot import _transcode_mp4_to_h264
from hflow.video import split_annex_b_stream
from hflow.ffmpeg import ffmpeg_path

with tempfile.TemporaryDirectory() as tmp:
    mp4_path = Path(tmp) / 'test.mp4'
    subprocess.run([
        str(ffmpeg_path()), '-y', '-f', 'lavfi', '-i', 'testsrc=duration=0.5:size=64x64:rate=10',
        '-c:v', 'libx264', str(mp4_path)
    ], check=True, capture_output=True)

    # Transcode MP4 to H.264 access units using LeRobot importer
    units = _transcode_mp4_to_h264(mp4_path, gop_seconds=1.0, frames_per_second=10.0)
    print(f'Total units: {len(units)}')
    for i, u in enumerate(units):
        print(f'Unit {i} start hex: {u[:12].hex()}')
        try:
            split_annex_b_stream(u)
        except Exception as e:
            print(f'Unit {i} validation failed: {type(e).__name__}: {e}')
"

Expected behavior

According to

src/hflow/format.py
and docs/ARCHITECTURE.md:

"Camera streams are in-band H.264 in foxglove.CompressedVideo messages (protobuf-encoded): Annex B byte stream, SPS/PPS on every keyframe, no B-frames, one decodable access unit per message."

Every access unit extracted from FFmpeg by _transcode_mp4_to_h264 should be a conforming Annex B byte sequence starting with a 3-byte or 4-byte NAL start code prefix (00 00 00 01 09... for the AUD NAL unit). Passing each unit to

hflow.video.split_annex_b_stream
should succeed with exactly one access unit.

Actual behavior

Total units: 5
Unit 0 start hex: 100000000167f4000a919641
Unit 0 validation failed: ValueError: no access-unit delimiter (AUD, NAL type 9) found; splitting requires a stream encoded with aud=1
Unit 1 start hex: 30000001419a3b119f0a9f2a
Unit 1 validation failed: ValueError: no access-unit delimiter (AUD, NAL type 9) found; splitting requires a stream encoded with aud=1
Unit 2 start hex: 30000001419a4f0864ca6119
Unit 2 validation failed: ValueError: no access-unit delimiter (AUD, NAL type 9) found; splitting requires a stream encoded with aud=1
Unit 3 start hex: 30000001419a727843c994c0
Unit 3 validation failed: ValueError: no access-unit delimiter (AUD, NAL type 9) found; splitting requires a stream encoded with aud=1
Unit 4 start hex: 30000001419a927843c994c0
Unit 4 validation failed: ValueError: no access-unit delimiter (AUD, NAL type 9) found; splitting requires a stream encoded with aud=1

Additional context and proposed fix

def _split_h264_by_aud(h264_data: bytes) -> list[bytes]:
    """Split H.264 stream on AUD (0x00000109) boundaries."""
    units: list[bytes] = []
    pattern = b"\x00\x00\x00\x01\x09"
    offset = 0
    while True:
        boundary_index = h264_data.find(pattern, offset)
        if boundary_index == -1:
            break
        if boundary_index > offset:
            units.append(h264_data[offset:boundary_index])
        offset = boundary_index + 5
    if offset < len(h264_data):
        units.append(h264_data[offset:])
    return units

When a boundary is found, offset = boundary_index + 5 skips past the start code and AUD NAL type byte, cutting them off from units.append(h264_data[offset:boundary_index]).

Existing converter unit tests missed this because tests/test_lerobot_converter.py stubs _convert_single_episode with _install_publish_through_convert (fake_convert writing dummy bytes episode-{episode_index}), so _transcode_mp4_to_h264 was never exercised on real video frames in the converter test suite.

Proposed Fix

hflow already has a robust, spec-compliant Annex B splitter in

hflow.video.split_annex_b_stream
. In

src/hflow/importers/lerobot.py
:

diff

  • return _split_h264_by_aud(completed_process.stdout)
  • from hflow.video import split_annex_b_stream
  • return [unit.data for unit in split_annex_b_stream(completed_process.stdout)]

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions