Skip to content

fix: align sampled RGB frames across decode gaps - #764

Merged
kstonekuan merged 13 commits into
Hebbian-Robotics:mainfrom
harsh21234i:fix/733-rgb-frame-gap-sampling
Nov 13, 2026
Merged

kstonekuan merged 13 commits into
Hebbian-Robotics:mainfrom
harsh21234i:fix/733-rgb-frame-gap-sampling

Conversation

@harsh21234i

@harsh21234i harsh21234i commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

  • Keep each sampled RGB frame's sample tick and source frame index, including when decoding resumes after a damaged opening.
  • Omit samples where FFmpeg repeats an earlier picture or conceals a picture across a lost keyframe.
  • Preserve original sample timestamps in MediaPipe and align detection intervals to source messages.
  • Split no-hand intervals when sampling ticks are skipped.
  • Bound the FFmpeg frame-hash pass and keep its process helper inside the standalone video-measurements package.

Fixes #733

Validation

  • GitHub Actions passed on Python 3.14 and Python 3.11 for the latest commit.
  • Python 3.14 lint, format, typecheck, and minimum-NumPy compatibility checks passed.
  • The module-boundary, stalled-decoder, and media-process tests passed locally.

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

@greptile-apps

greptile-apps Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

RetriggerView in GreptileConfidence Score: 5/5

[Medium impact] The PR appears safe to merge; no blocking issues remain.

Summary

The PR keeps sampled RGB pictures aligned with their original sample ticks and source messages after decode gaps.

  • RGB sampling skips bad pictures without shifting later frame times.
  • Hand detections keep their real sample and camera times.
  • The converter test reads its result through an in-memory connection.

The latest update adds bounds to all three ffprobe queries. No new actionable issues were found. The four earlier, unnumbered findings are addressed by the current code.

Reviews (12) · Last reviewed commit: "fix: bound frame metadata probes" · Reviewed by Greptile

Comment thread src/hflow/mediapipe_hands.py Resolved
greptile-apps[bot]
greptile-apps Bot previously approved these changes Oct 29, 2026

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you cover and fix non-integer sampling rates? On an intact three-second, 30 fps testsrc2 clip, rgb_frames(..., frames_per_second=3.0000049) returns only sample 0 here; main returns nine frames. At 3.0 fps, both return all nine.

@greptile-apps
greptile-apps Bot dismissed their stale review October 30, 2026 05:30

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

greptile-apps[bot]
greptile-apps Bot previously approved these changes Oct 30, 2026
@greptile-apps
greptile-apps Bot dismissed their stale review October 30, 2026 05:53

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

@harsh21234i

Copy link
Copy Markdown
Contributor Author

Hi, thanks for the review. I’ve updated the branch to:

  • Stream sampled RGB frames without spooling full pixel data to temporary storage.
  • Keep only source frame indices for MediaPipe timestamp mapping.
  • Wait for FFmpeg to exit normally after the hash stream reaches EOF.
  • Cover non-integer sample rates, including 3.0000049, using the same effective rate for FFmpeg and source-index mapping.
  • Give the DuckDB test its own connection to prevent shared connection state from breaking the full suite.

greptile-apps[bot]
greptile-apps Bot previously approved these changes Oct 30, 2026

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you add coverage through the MediaPipe path for a missing sampling tick? The decoder regression checks the retained frame indices, but it doesn't verify that later detector timestamps and absence intervals keep their original times.

@greptile-apps
greptile-apps Bot dismissed their stale review November 1, 2026 19:32

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

@harsh21234i

Copy link
Copy Markdown
Contributor Author

Thanks for the feedback. I added coverage through the MediaPipe path for a missing sampling tick. The test verifies later detector timestamps stay at their original times and the absence interval uses the original source timestamps. The full suite passes: 2,628 passed, 9 skipped.

greptile-apps[bot]
greptile-apps Bot previously approved these changes Nov 1, 2026

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also exclude the concealed picture after the lost keyframe? On the existing IDR-100 fixture, sample 108 is labelled as source frame 109 but its pixels read as 111. #767 now rejects that picture in the episode accessors.

@greptile-apps
greptile-apps Bot dismissed their stale review November 2, 2026 12:46

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

Copy link
Copy Markdown
Contributor Author

Thanks for the follow-up. I’ve pushed commit 30f1791, which applies the keyframe-chain check to sampled RGB frames and omits the concealed picture after the lost IDR. The IDR-100 regression now checks that samples 100–108 are omitted and verifies the pixels of every retained sample against its source timestamp. The earlier fractional-rate and MediaPipe timing requests are also covered on this branch. Full suite: 2,665 passed, 9 skipped. Could you please re-review?

Copy link
Copy Markdown
Contributor Author

I also fixed the opening-gap case in commit 64075eb. The decoded hash trace now keeps absolute source positions, and the streaming sampler starts from the first corresponding tick on the original sampling grid, so valid frames after a damaged opening keyframe retain their labels. Added regression coverage for nonzero decoded positions and the sample-grid offset. Full suite passes: 2,669 passed, 9 skipped. Please re-review.

greptile-apps[bot]
greptile-apps Bot previously approved these changes Nov 2, 2026

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you keep no_hand_detected intervals out of skipped sampling ticks? In test_missing_sampling_tick_keeps_detector_and_absence_times, tick 1 has no decoded sample, but the asserted interval runs from 0 to 3 seconds and includes it.

@greptile-apps
greptile-apps Bot dismissed their stale review November 6, 2026 09:30

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

Copy link
Copy Markdown
Contributor Author

I fixed the skipped-tick interval issue in commit 1bac45e. Absence runs now break at gaps in the original sample indices, so the interval cannot claim a skipped tick. The regression now expects the absence span to resume at the next consecutive sampled tick. Full suite passes: 2,670 passed, 9 skipped. Please re-review.

greptile-apps[bot]
greptile-apps Bot previously approved these changes Nov 6, 2026

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you bound the new frame-hash decode and cover a stalled FFmpeg process? _decoded_frame_hashes blocks while reading stdout and waits without a timeout, so a hung decoder can leave the measurement running indefinitely.

@greptile-apps
greptile-apps Bot dismissed their stale review November 11, 2026 08:56

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

harsh21234i commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor Author

I addressed this in commit c47e0ad. The frame-hash decode now runs through HFlow’s bounded media-command runner, with a 600-second timeout and a 256 MiB output limit; a timed-out child is killed and reaped. Added a stalled-process regression. Ruff, formatting, ty, and the focused process-boundary tests pass (8 passed). The full suite last passed on the preceding commit (2,670 passed, 9 skipped); Please re-review.

greptile-apps[bot]
greptile-apps Bot previously approved these changes Nov 11, 2026
@greptile-apps
greptile-apps Bot dismissed their stale review November 11, 2026 09:13

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

harsh21234i commented Nov 11, 2026 •

Copy link
Copy Markdown
Contributor Author

The CI failure was the package-boundary test; _video_measurements cannot import hflow.ffmpeg. Commit cb5b138 moves the bounded subprocess helper into the measurement package. The rerun is now green on Python 3.14 and 3.11; Python 3.14 lint, format, typecheck, tests, and minimum-NumPy compatibility all pass.

greptile-apps[bot]
greptile-apps Bot previously approved these changes Nov 11, 2026

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you also bound the new ffprobe calls in _probe_frame_rate and _probe_keyframe_indices? Both use subprocess.run without a timeout or output limit, so a stalled probe can still leave the measurement running indefinitely. Please cover a stalled probe too.

@greptile-apps
greptile-apps Bot dismissed their stale review November 12, 2026 18:25

Dismissed because a newer commit was pushed; Greptile will re-review the current head.

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working through this! Ruff, formatting, ty, and the full suite pass locally (2,637 passed, 47 skipped).

@kstonekuan
kstonekuan merged commit c073afa into Hebbian-Robotics:main Nov 13, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: rgb_frames(frames_per_second=…) samples come from after the message mediapipe_hands labels them with, and repeat across a lost keyframe

2 participants