feat: Add media foundation backend#180
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6d408abdfb
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c0e9858729
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| var rows = checked(height + (format == PixelFormats.NV12 ? (height + 1) / 2 : 0)); | ||
| var rgb = format is PixelFormats.RGB15 or PixelFormats.RGB16 or | ||
| PixelFormats.RGB24 or PixelFormats.RGB32 or PixelFormats.ARGB32; | ||
| var bottomUp = defaultStride < 0 || defaultStride is null && rgb; |
There was a problem hiding this comment.
When a capture media type does not include MF_MT_DEFAULT_STRIDE, ConfigureReader passes null into this code and the defaultStride is null && rgb branch assumes RGB buffers are bottom-up. Media Foundation's uncompressed-buffer guidance says to calculate the default stride when the attribute is absent because the stride sign carries top-down vs. bottom-up orientation; on RGB cameras that omit the attribute this will normalize frames with the wrong row order and produce vertically flipped images. Use MFGetStrideForBitmapInfoHeader or another explicit orientation source instead of treating null as bottom-up.
Useful? React with 👍 / 👎.
…ng COM COM Interfaces do not require dynamic code but unreferenced code. That means they are relevant for trimming as well, not just for aot
1 participant
I decided to follow your suggestion in #179 and implement the Media Framework backend in a separate PR before we go for complete AOT compatibility.
How it was implemented
The WMF backend is now implemented for
netstandard2.0,netstandard2.1,net48,net5.0, and newer.I am using CsWin32 with raw, unsafe bindings. That allows me not to use the
GeneratedComInterfaceI mentioned before, which should solve the issue of being restricted to >=net8.0🥳 (sry, I did not know before).Since we generate very low-level bindings, are you fine with using CsWin32 here? You can configure the output a lot to your requirements:
{ "$schema": "https://aka.ms/CsWin32.schema.json", "allowMarshaling": false, // The Win32 generator emits only raw, blittable structs with COM interfaces being emitted as interfaces. This gives us AOT compatibility "friendlyOverloads": { "enabled": true, // Enables generation of overloads with e.g., out parameters, ... "comOutPtrGenericOverloads": false // We don't require the generic overloads so this reduces compilation overhead }, "useSafeHandles": false, // Enables (or requires :D) explicit control over cleanup, ... "comInterop": { "preserveSigMethods": [ "*" ] // Methods return their native HRESULT, ... } }Short Disclaimer: While I reviewed the code and the architecture, and the higher-level code comes from me, the details of the Media Framework interop were implemented by AI. I tested the code in both the Avalonia app on
net48andnet8.0, as well as our production project published with AOT, and so far it seems to work.Additional libraries
The
netstandard2.0andnet48targets now depends onSystem.MemoryandSystem.Runtime.CompilerServices.Unsafe. This requirement comes straight from CsWin32. If you would like to avoid this, I suggest dropping thenetstandard2.0/net48support for WMF.Breaking change
There are no breaking changes I am aware of. However, right now, we annotate
OnEnumerateDescriptorswithRequiresUnreferencedCodeand then manually ignore that in theMediaFoundationCaptureDevicesoverwrite. This kinda works, but it's really not nice. Its probably best to think of a better API here in the future which allowes different backends to have different annotations regarding trimming capabilities.