Skip to content

Video: GStreamer subsystem overhaul + audit fixes#14499

Open
HTRamsey wants to merge 4 commits into
mavlink:masterfrom
HTRamsey:video/gst-source-factory
Open

Video: GStreamer subsystem overhaul + audit fixes#14499
HTRamsey wants to merge 4 commits into
mavlink:masterfrom
HTRamsey:video/gst-source-factory

Conversation

@HTRamsey

@HTRamsey HTRamsey commented Jun 8, 2026

Copy link
Copy Markdown
Member

Summary

Architectural overhaul of the GStreamer video subsystem plus Qt-6.10-validated audit fixes.

  • GstSourceFactory extracted from GstVideoReceiver; GstAppSinkAdapter split into focused units (GStreamerEnvironment, GStreamerFrameMap, GStreamerLogging, GStreamerHelpers).
  • HwBuffers expanded into per-platform zero-copy GPU paths (DMABuf, GL, IOSurface, D3D11, D3D12, AHardwareBuffer) sharing GstHwFrameTexturesBase + GstGlFrameTextures, with GstContextBridgeRegistry for cross-thread context handoff and GstHwPathTelemetry for per-format fallback counters. CpuVideoFramePool (QByteArray slab recycler) backs the CPU fallback.
  • Custom gstqgc plugin (qgcqvideosink + sink bin) with QGC-side QRhi capture and shader routing.
  • CMake: GStreamer support reorganized under cmake/GStreamer/ (Orchestrator, Probe, Components, Download, Install, Layout, Link, PluginPolicy + per-platform modules + test harness). Obsolete FindGStreamerMobile/FindQGCGStreamer/gst_ios_init shims removed. Unused Android Java callback wrappers dropped.

Audit fixes (cross-checked against Qt 6.10.3 source)

  • BGRx/BGRA DRM fourcc endian: ABGR8888ARGB8888 (LE) per fourccFromVideoInfo().
  • SMPTE 240M color matrix BT709 → AdobeRgb, transfer BT709 → Gamma22; FCC matrix BT601 → Undefined.
  • DMABuf GL cleanup matches ~QGstQVideoFrameTextures (drop QRhiGles2NativeHandles fallback).
  • Concurrency/lifetime hardening: GstContextBridgeRegistry snapshot-under-lock; D3D12 decoder-fence sync + slice copy moved off the QSGRenderThread to the streaming thread; per-frame dynamic_cast replaced with a sourcePath() path-tag.
  • Windows: demote the D3D decoder family that mismatches the active QRhi backend (QQuickWindow::graphicsApi()), so a D3D12 decoder can't outrank D3D11 on Qt's default Windows RHI.

Runtime fixes (caught via local run)

  • gpu-zerocopy is construct-only — construct the inner qgcqvideosink via gst_element_factory_make_full instead of a post-construction g_object_set that GLib rejected.
  • GstQgcVideoSinkBin now exposes a sync property proxied to the inner basesink, so GstVideoReceiver's sync setting actually applies.

Testing

  • Linux Release build clean; app launches and the pipeline builds. GStreamer 1.24.2.
  • No GLib-GObject property criticals remain.
  • Platform-gated paths (D3D11/D3D12, IOSurface, AHardwareBuffer) are compile-validated only — need Windows/macOS/Android CI + D3D12-capable hardware to exercise the zero-copy import.

Copilot AI review requested due to automatic review settings June 8, 2026 17:59

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot wasn't able to review this pull request because it exceeds the maximum number of lines (20,000). Try reducing the number of changed lines and requesting a review from Copilot again.

@github-actions

github-actions Bot commented Jun 8, 2026

Copy link
Copy Markdown
Contributor

⚠️ Build results unavailable — artifact download from one or more platform workflows failed (likely artifact retention expiry or transient API error). The combined report cannot be generated for this run.

See the Build Results workflow run for details.

@HTRamsey HTRamsey force-pushed the video/gst-source-factory branch 4 times, most recently from dbeb77c to 48dd85e Compare June 8, 2026 21:04
@codecov

codecov Bot commented Jun 8, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 18.57765% with 1122 lines in your changes missing coverage. Please review.
✅ Project coverage is 30.44%. Comparing base (f29efd3) to head (955b3f5).
⚠️ Report is 113 commits behind head on master.

Files with missing lines Patch % Lines
...anager/VideoReceiver/GStreamer/GstVideoReceiver.cc 0.00% 253 Missing ⚠️
...anager/VideoReceiver/GStreamer/GstSourceFactory.cc 31.09% 108 Missing and 138 partials ⚠️
.../VideoManager/VideoReceiver/GStreamer/GStreamer.cc 12.12% 116 Missing and 29 partials ⚠️
...er/VideoReceiver/GStreamer/GStreamerEnvironment.cc 10.89% 121 Missing and 18 partials ⚠️
src/Utilities/Platform/GraphicsSetup.cc 0.00% 85 Missing ⚠️
src/VideoManager/VideoManager.cc 17.34% 64 Missing and 17 partials ⚠️
...anager/VideoReceiver/GStreamer/GStreamerHelpers.cc 39.06% 43 Missing and 35 partials ⚠️
...anager/VideoReceiver/GStreamer/GStreamerLogging.cc 26.82% 35 Missing and 25 partials ⚠️
src/Settings/VideoSettings.cc 44.82% 2 Missing and 14 partials ⚠️
...Manager/VideoReceiver/GStreamer/GstVideoReceiver.h 0.00% 6 Missing ⚠️
... and 5 more

❌ Your patch check has failed because the patch coverage (18.57%) is below the target coverage (30.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

Impacted file tree graph

@@            Coverage Diff             @@
##           master   #14499      +/-   ##
==========================================
+ Coverage   25.47%   30.44%   +4.97%     
==========================================
  Files         769      786      +17     
  Lines       65912    66746     +834     
  Branches    30495    30914     +419     
==========================================
+ Hits        16788    20323    +3535     
+ Misses      37285    32532    -4753     
- Partials    11839    13891    +2052     
Flag Coverage Δ
unittests 30.44% <18.57%> (+4.97%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
src/LogManager/LogManager.h 100.00% <ø> (ø)
src/Settings/VideoSettings.h 100.00% <ø> (+85.71%) ⬆️
src/VideoManager/VideoManager.h 25.00% <ø> (+25.00%) ⬆️
...Streamer/HwBuffers/common/GstHwFrameTexturesBase.h 14.28% <ø> (ø)
...iver/GStreamer/HwBuffers/common/GstHwVideoBuffer.h 0.00% <ø> (ø)
...VideoReceiver/GStreamer/QGCQVideoSinkController.cc 33.08% <ø> (ø)
...ideoReceiver/GStreamer/gstqgc/GstQgcVideoFormats.h 66.66% <ø> (ø)
.../VideoReceiver/GStreamer/gstqgc/gstqgcqvideosink.h 100.00% <ø> (ø)
...ideoReceiver/GStreamer/gstqgc/gstqgcvideosinkbin.h 100.00% <ø> (ø)
...er/VideoReceiver/Offscreen/QGCOffscreenRenderer.cc 0.00% <ø> (ø)
... and 22 more

... and 403 files with indirect coverage changes


Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update a8a6cc4...955b3f5. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@HTRamsey HTRamsey force-pushed the video/gst-source-factory branch 6 times, most recently from d8a4d73 to 131dd86 Compare June 9, 2026 01:21
@HTRamsey HTRamsey force-pushed the video/gst-source-factory branch 5 times, most recently from 62e5067 to a9a9ed3 Compare June 18, 2026 01:33

steps:
- name: Harden Runner
uses: step-security/harden-runner@v2
@HTRamsey HTRamsey force-pushed the video/gst-source-factory branch 11 times, most recently from 5e9bf3b to 9274470 Compare June 18, 2026 21:41
@HTRamsey HTRamsey force-pushed the video/gst-source-factory branch 3 times, most recently from 4564817 to 3601fad Compare June 26, 2026 20:24
Extract GstSourceFactory from GstVideoReceiver. Split GstAppSinkAdapter into
focused components (GStreamerEnvironment, GStreamerFrameMap, GStreamerLogging,
GStreamerHelpers). Expand HwBuffers into per-platform zero-copy GPU paths
(DMABuf, GL, IOSurface, D3D11, D3D12, AHardwareBuffer) sharing
GstHwFrameTexturesBase + GstGlFrameTextures, with GstContextBridgeRegistry
for cross-thread context handoff and GstHwPathTelemetry for per-format
fallback counters. Add CpuVideoFramePool (WebRTC-style QByteArray slab
recycler) for the CPU fallback path.

HwBuffers is organized into per-backend subfolders (common, dmabuf, gl, d3d,
vulkan, apple, android, cuda) matching the CMake path-gating; intra-HwBuffers
unqualified includes resolve via the subfolders added to the GStreamer include
path, and external includers are repointed to their new subpaths.

Introduce custom gstqgc GStreamer plugin (qgcqvideosink + bin) with QGC-side
QRhi capture and shader pipeline routing.

CMake: reorganize GStreamer support into cmake/GStreamer/ (Orchestrator,
Probe, Components, Download, Install, Layout, Link, PluginPolicy + per-platform
modules + test harness). Delete obsolete FindGStreamerMobile, FindQGCGStreamer,
GStreamerHelpers, gst_ios_init shims.

Android: drop unused Java callback wrappers (GStreamer.java, GstAhcCallback,
GstAhsCallback, GstAmcOnFrameAvailableListener).

Audit fixes (Qt 6.10.3 source cross-checked):
- BGRx/BGRA DRM fourcc endian: was ABGR8888, now ARGB8888 (LE) per Qt
  fourccFromVideoInfo() at qgstvideobuffer.cpp:178-180
- SMPTE 240M color matrix: was BT709, now AdobeRgb per qgst.cpp:407-409
- SMPTE 240M transfer: was BT709, now Gamma22 per qgst.cpp:423-427
- FCC color matrix: was BT601, now Undefined (no Qt equivalent)
- DMABuf GL cleanup: drop QRhiGles2NativeHandles cast fallback, use only
  QOpenGLContext::currentContext() after makeThreadLocalNativeContextCurrent,
  matching ~QGstQVideoFrameTextures
- Inline kMaxDotFiles + kEosTimeoutNs into GstVideoReceiver.cc; delete
  GStreamerConfig.h (5 dead constants from removed pacing feature)
@HTRamsey HTRamsey force-pushed the video/gst-source-factory branch from 3601fad to f31cf72 Compare June 26, 2026 21:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants