Skip to content

Commit fc2411b

Browse files
authored
refactor(mcap): encode and rewrite video messages through hebbian-mcap 0.1.1 (#785)
The video importer, LeRobot importer and transform build CompressedVideo messages and the schema record with the shared Rust encoder. Transform and doctor read pass-through video through CompressedVideoCodec, which walks ROS 2 CDR by the channel's schema and replaces protobuf payloads at the wire level, so the Python re-encoding workaround is removed. Bump TRANSFORM_BEHAVIOR_VERSION to 13: the schema record and library identifier change, and repaired protobuf messages keep their field order. foxglove-schemas-protobuf moves to the dev group because only tests import it.
1 parent b2b3559 commit fc2411b

10 files changed

Lines changed: 105 additions & 164 deletions

File tree

‎docs/ARCHITECTURE.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,9 @@ On-disk identifiers (metadata record names, channel naming) are neutral and form
121121

122122
The `hebbian-mcap` dependency owns generic grouping and MCAP serialization. HFlow
123123
owns group selection, derived targets, episode schemas, transformations and provenance.
124-
The shared package also owns H.264 inspection and optional camera-to-MP4 export;
124+
The shared package also owns `CompressedVideo` encoding, reading and payload
125+
replacement for protobuf and ROS 2 CDR channels, H.264 inspection and optional
126+
camera-to-MP4 export;
125127
`hflow.mcap_video` adapts HFlow media limits to that exporter. Codec dependencies
126128
remain behind the `video` extra.
127129
We maintain both repositories; coordinated changes to `mcap-tools` are welcome.

‎docs/FORMAT.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,7 +19,7 @@ The single overriding rule: **a canonical episode is spec-conforming MCAP.** Eve
1919

2020
- MCAP magic, `Header`, data section, `DataEnd`, summary section, `Footer`, closing magic, per the [MCAP spec](https://mcap.dev/spec).
2121
- `Header.profile` is the empty string `""` (the file mixes protobuf video channels with pass-through channels of arbitrary encoding, so no single profile applies).
22-
- `Header.library` is informational only (currently `hebbian-mcap/0.1.0`). Its exact value is not part of the format contract, and no reader may key behavior off it (see [Identifier rules](#identifier-rules)).
22+
- `Header.library` is informational only (currently `hebbian-mcap/0.1.1`). Its exact value is not part of the format contract, and no reader may key behavior off it (see [Identifier rules](#identifier-rules)).
2323
- Chunks are compressed with **zstd** by default (`"none"` is permitted). Each `Chunk` record carries `uncompressed_crc`; the `Footer` carries a summary CRC.
2424
- The summary section repeats all `Schema` and `Channel` records and contains `Statistics`, all `ChunkIndex` records, `AttachmentIndex`/`MetadataIndex` records, and `SummaryOffset` records. A canonical episode always has a complete summary; unindexed files are not canonical.
2525
- Each topic names exactly one channel. MCAP allows several channels to share a topic; a canonical episode does not, because topic-keyed reads would be ambiguous.

‎pyproject.toml‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -22,12 +22,11 @@ keywords = [
2222
requires-python = ">=3.11"
2323
dependencies = [
2424
"duckdb>=2.0.0",
25-
"foxglove-schemas-protobuf>=0.4.0",
2625
"httpx>=0.28.1,<1",
2726
"httpx2>=2.12.0",
2827
"huggingface-hub>=1.31.0,<2",
2928
"mcap>=1.4.0",
30-
"hebbian-mcap==0.1.0",
29+
"hebbian-mcap==0.1.1",
3130
"mcap-protobuf-support>=0.5.4",
3231
"mcap-ros2-support>=0.5.7",
3332
"numpy>=2.3.5",
@@ -57,7 +56,7 @@ mediapipe = ["mediapipe>=1.0.1"]
5756
# dependency is real, and confined to the one check that needs it. Headless
5857
# because a data pipeline runs where there is no display.
5958
motion = ["opencv-python-headless>=5.0.0.93"]
60-
video = ["av>=18.1.0", "hebbian-mcap[video]==0.1.0"]
59+
video = ["av>=18.1.0", "hebbian-mcap[video]==0.1.1"]
6160
native-build = ["Cython>=3.3.0", "setuptools>=84.0.0"]
6261
openai = ["openai>=3.3.1"]
6362

@@ -72,6 +71,7 @@ hflow = "hflow.cli:main"
7271
[dependency-groups]
7372
dev = [
7473
"av>=18.1.0", # the video extra, exercised by timestamp-preserving camera export tests
74+
"foxglove-schemas-protobuf>=0.4.0", # tests build foxglove messages to feed readers
7575
"Cython>=3.3.0", # the native-build extra, present so its real build path is tested
7676
"hflow-server", # the workspace API package, present in dev so the suite tests it
7777
"obstore>=0.11.1", # the bucket extra, present in dev so the suite tests it

‎src/hflow/behavior.py‎

Lines changed: 7 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -108,4 +108,10 @@
108108
# hebbian-mcap. Message semantics and group isolation are preserved, but the
109109
# library identifier, schema/channel placement, and compressed bytes can change.
110110
# Regenerate derived outputs rather than mixing both writers under one identity.
111-
TRANSFORM_BEHAVIOR_VERSION: str = "12"
111+
#
112+
# "13": video messages are encoded and rewritten by hebbian-mcap 0.1.1. The
113+
# CompressedVideo schema record now carries protobuf ``json_name`` entries, the
114+
# library identifier names the new release, and a pass-through protobuf message
115+
# that gains an access unit delimiter keeps its original field order. Decoded
116+
# message content is unchanged.
117+
TRANSFORM_BEHAVIOR_VERSION: str = "13"

‎src/hflow/doctor.py‎

Lines changed: 20 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -17,12 +17,13 @@
1717
"""
1818

1919
import math
20-
from collections.abc import Callable, Iterator
20+
from collections.abc import Iterator
2121
from dataclasses import dataclass, field
2222
from enum import StrEnum
2323
from pathlib import Path
2424
from typing import Any
2525

26+
from hebbian_mcap import CompressedVideoCodec
2627
from mcap.reader import make_reader
2728
from mcap.records import Channel, Schema
2829

@@ -190,7 +191,7 @@ def _check_video_payload(
190191

191192

192193
class VideoEncodingUnsupported(ValueError):
193-
"""No decoder factory in the supported set handles this message encoding.
194+
"""The channel's message encoding or schema is outside the supported set.
194195
195196
A subclass of ``ValueError`` because every existing caller treats
196197
resolution failure as a ``ValueError``; the distinct type lets the video
@@ -200,19 +201,17 @@ class VideoEncodingUnsupported(ValueError):
200201
"""
201202

202203

203-
def _resolve_video_decoder(topic: str, channel: Channel, schema: Schema) -> Callable[[bytes], Any]:
204+
def _resolve_video_codec(topic: str, channel: Channel, schema: Schema) -> CompressedVideoCodec:
204205
"""Resolve either supported encoded-video payload representation."""
205-
from mcap_protobuf.decoder import DecoderFactory as ProtobufDecoderFactory
206-
from mcap_ros2.decoder import DecoderFactory as Ros2DecoderFactory
207-
208-
for factory in (Ros2DecoderFactory(), ProtobufDecoderFactory()):
209-
decoder = factory.decoder_for(channel.message_encoding, schema)
210-
if decoder is not None:
211-
return decoder
212-
raise VideoEncodingUnsupported(
213-
f"video topic {topic!r} has message encoding {channel.message_encoding!r} "
214-
"that no available decoder handles"
215-
)
206+
try:
207+
return CompressedVideoCodec(
208+
channel.message_encoding, schema.encoding, schema.name, schema.data
209+
)
210+
except ValueError as error:
211+
raise VideoEncodingUnsupported(
212+
f"video topic {topic!r} has message encoding {channel.message_encoding!r} "
213+
f"that no available decoder handles ({error})"
214+
) from error
216215

217216

218217
def _unreadable_message(stream: Any, error: Exception) -> str:
@@ -503,7 +502,7 @@ def iter_all_messages() -> Iterator[tuple[int, int, bytes]]:
503502
video_log_times: dict[int, list[int]] = {}
504503
video_keyframes: dict[int, list[bool | None]] = {}
505504
try:
506-
video_decoders: dict[int, Callable[[bytes], Any]] = {}
505+
video_codecs: dict[int, CompressedVideoCodec] = {}
507506

508507
for channel_id, log_time, payload in iter_all_messages():
509508
previous = last_log_time_by_channel.get(channel_id)
@@ -523,12 +522,12 @@ def iter_all_messages() -> Iterator[tuple[int, int, bytes]]:
523522
continue
524523
message_index = video_message_counts.get(channel_id, 0)
525524
video_message_counts[channel_id] = message_index + 1
526-
decoder = video_decoders.get(channel_id)
527-
if decoder is None:
525+
codec = video_codecs.get(channel_id)
526+
if codec is None:
528527
channel = summary.channels[channel_id]
529528
schema = summary.schemas[channel.schema_id]
530529
try:
531-
decoder = _resolve_video_decoder(channel.topic, channel, schema)
530+
codec = _resolve_video_codec(channel.topic, channel, schema)
532531
except VideoEncodingUnsupported as error:
533532
video_check_skipped_topics.add(topics_by_channel_id[channel_id])
534533
collector.add(
@@ -538,14 +537,14 @@ def iter_all_messages() -> Iterator[tuple[int, int, bytes]]:
538537
"cannot run on this topic (the file bytes are not implicated)",
539538
)
540539
continue
541-
video_decoders[channel_id] = decoder
542-
decoded = decoder(payload)
540+
video_codecs[channel_id] = codec
541+
decoded = codec.decode(payload)
543542
is_keyframe = _check_video_payload(
544543
collector,
545544
topics_by_channel_id[channel_id],
546545
message_index,
547546
decoded.format,
548-
bytes(decoded.data),
547+
decoded.data,
549548
is_first_message=message_index == 0,
550549
)
551550
video_log_times.setdefault(channel_id, []).append(log_time)

‎src/hflow/importers/lerobot.py‎

Lines changed: 5 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1276,10 +1276,8 @@ def _feature_rows(feature_name: str) -> list | None:
12761276
landing_relative_key = _landing_relative_key(episode_index)
12771277
output_file_name = Path(landing_relative_key).name
12781278

1279-
from foxglove_schemas_protobuf.CompressedVideo_pb2 import CompressedVideo
1280-
from mcap_protobuf.schema import build_file_descriptor_set
1279+
from hebbian_mcap import COMPRESSED_VIDEO_SCHEMA, encode_compressed_video
12811280

1282-
video_schema_data = build_file_descriptor_set(CompressedVideo).SerializeToString()
12831281
state_schema_name = "lerobot_msgs/msg/State"
12841282
action_schema_name = "lerobot_msgs/msg/Action"
12851283
state_schema_text = f"float32[{numeric_schemas['observation.state'].dim}] position"
@@ -1296,7 +1294,7 @@ def _feature_rows(feature_name: str) -> list | None:
12961294
mcap_writer.start(profile="", library="hflow LeRobot source adapter")
12971295

12981296
video_schema_id = mcap_writer.register_schema(
1299-
"foxglove.CompressedVideo", "protobuf", video_schema_data
1297+
"foxglove.CompressedVideo", "protobuf", COMPRESSED_VIDEO_SCHEMA
13001298
)
13011299
state_schema_id = mcap_writer.register_schema(
13021300
state_schema_name, "ros2msg", state_schema_data
@@ -1328,16 +1326,12 @@ def _feature_rows(feature_name: str) -> list | None:
13281326
)
13291327
for camera_key in camera_keys:
13301328
access_units, _presentation_timestamps = video_data_by_camera[camera_key]
1331-
video_message = CompressedVideo()
1332-
video_message.timestamp.seconds = log_time_ns // NANOSECONDS_PER_SECOND
1333-
video_message.timestamp.nanos = log_time_ns % NANOSECONDS_PER_SECOND
1334-
video_message.frame_id = camera_key
1335-
video_message.data = access_units[frame_index]
1336-
video_message.format = "h264"
13371329
mcap_writer.add_message(
13381330
channel_id=video_channel_ids[camera_key],
13391331
log_time=log_time_ns,
1340-
data=video_message.SerializeToString(),
1332+
data=encode_compressed_video(
1333+
log_time_ns, camera_key, access_units[frame_index], "h264"
1334+
),
13411335
publish_time=log_time_ns,
13421336
sequence=frame_index,
13431337
)

‎src/hflow/importers/video.py‎

Lines changed: 5 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -8,9 +8,8 @@
88
from fractions import Fraction
99
from pathlib import Path
1010

11-
from foxglove_schemas_protobuf.CompressedVideo_pb2 import CompressedVideo
11+
from hebbian_mcap import COMPRESSED_VIDEO_SCHEMA, encode_compressed_video
1212
from mcap.writer import Writer
13-
from mcap_protobuf.schema import build_file_descriptor_set
1413

1514
from hflow._field_guards import (
1615
require_finite_float,
@@ -374,7 +373,7 @@ def _import_inspected_video(
374373
schema_id = writer.register_schema(
375374
name=CANONICAL_VIDEO_SCHEMA_NAME,
376375
encoding="protobuf",
377-
data=build_file_descriptor_set(CompressedVideo).SerializeToString(),
376+
data=COMPRESSED_VIDEO_SCHEMA,
378377
)
379378
channel_id = writer.register_channel(
380379
topic=f"/{config.camera_name}/compressed",
@@ -408,19 +407,14 @@ def _import_inspected_video(
408407
)
409408
for frame_index, access_unit in enumerate(access_units):
410409
timestamp_ns = _sample_timestamp_ns(config, frame_index)
411-
message = CompressedVideo()
412-
message.timestamp.seconds, message.timestamp.nanos = divmod(
413-
timestamp_ns, NANOSECONDS_PER_SECOND
414-
)
415-
message.frame_id = config.camera_name
416-
message.format = "h264"
417-
message.data = access_unit.data
418410
writer.add_message(
419411
channel_id=channel_id,
420412
log_time=timestamp_ns,
421413
publish_time=timestamp_ns,
422414
sequence=frame_index,
423-
data=message.SerializeToString(),
415+
data=encode_compressed_video(
416+
timestamp_ns, config.camera_name, access_unit.data, "h264"
417+
),
424418
)
425419
writer.finish()
426420
# A hard link is an atomic create-if-absent on the same filesystem;

0 commit comments

Comments
 (0)