Skip to content

Add vhost-device-media backend for virtio-media#944

Open
aesteve-rh wants to merge 7 commits into
rust-vmm:mainfrom
aesteve-rh:virtio-media
Open

Add vhost-device-media backend for virtio-media#944
aesteve-rh wants to merge 7 commits into
rust-vmm:mainfrom
aesteve-rh:virtio-media

Conversation

@aesteve-rh

@aesteve-rh aesteve-rh commented Mar 10, 2026

Copy link
Copy Markdown
Contributor

Summary of the PR

The virtio-media device first appeared in virtio-spec v1.4 [1].
It standardizes how guest applications communicate with media
devices on the host, such as encoders, decoders, and cameras,
using the V4L2 (Video4Linux2) protocol for guest-host
communication.

The low-level implementation of the virtio-media protocol is
provided by the device crate from the virtio-media repository [2].

The crate follows the same lib + binary structure as other
vhost-device crates: core logic lives in lib.rs, while main.rs
is a thin CLI entry point.

Each backend is gated by an optional cargo feature to minimize
binary size and the dependency footprint:

  • simple-capture: simple V4L2 capture device
  • v4l2-proxy: V4L2 proxy device (enabled by default)
  • ffmpeg: software video decoding via the virtio-media-ffmpeg-decoder
    crate [3], useful for testing without dedicated hardware
  • null: always-compiled no-op backend
  • xen: enables Xen platform support for rust-vmm dependencies

A V4l2DeviceType helper resolves the device path given on the
command line to the appropriate VFL_TYPE_* kernel constant.

The MediaAllocator, which manages shared memory regions for the
device, is adapted from AddressAllocator in the crosvm resources
crate.

[1] https://docs.oasis-open.org/virtio/virtio/v1.4/virtio-v1.4.html#x1-82200022
[2] https://github.com/chromeos/virtio-media
[3] https://github.com/chromeos/virtio-media/tree/main/extras/ffmpeg-decoder

Requirements

Before submitting your PR, please make sure you addressed the following
requirements:

  • All commits in this PR have Signed-Off-By trailers (with
    git commit -s), and the commit message has max 60 characters for the
    summary and max 75 characters for each description line.
  • All added/changed functionality has a corresponding unit/integration
    test.
  • All added/changed public-facing functionality has entries in the "Upcoming
    Release" section of CHANGELOG.md (if no such section exists, please create one).
  • Any newly added unsafe code is properly documented.

Comment thread vhost-device-media/Cargo.toml Outdated
@dorindabassey

Copy link
Copy Markdown
Collaborator

Thank you for this PR @aesteve-rh
IIUC, this vhost-device-media supersedes vhost-device-video, so will vhost-device-video be deprecated once virtio-media is fully supported?

@epilys

epilys commented Mar 12, 2026

Copy link
Copy Markdown
Member

@dorindabassey virtio-video is still under development (see virtio-comment list for v10). It's not replaced by virtio-media.

@aesteve-rh aesteve-rh marked this pull request as ready for review April 15, 2026 09:33
@aesteve-rh

Copy link
Copy Markdown
Contributor Author

Some rust-vmm-container dependencies may be missing for the builds in CI for the media crate. I will investigate.

@stefano-garzarella

Copy link
Copy Markdown
Member

Some rust-vmm-container dependencies may be missing for the builds in CI for the media crate. I will investigate.

@aesteve-rh yeah, we need to manually update rust-vmm-ci with the new container tag, then also update rust-vmm-ci submodule here. Sorry, we should document this somewhere.
See rust-vmm/rust-vmm-ci#186 for example.

@aesteve-rh

Copy link
Copy Markdown
Contributor Author

Ah, yes. Makes sense. I think I can manage. Thanks for the pointer!

@aesteve-rh

Copy link
Copy Markdown
Contributor Author

CI update: rust-vmm/rust-vmm-ci#205

@stefano-garzarella

Copy link
Copy Markdown
Member

@aesteve-rh it seems there is still something missing in the CI container. I suggest running it locally following https://github.com/rust-vmm/rust-vmm-ci#running-the-tests-locally

@aesteve-rh

aesteve-rh commented May 21, 2026

Copy link
Copy Markdown
Contributor Author

Another +1 for this one? rust-vmm/rust-vmm-ci#207

edit:
@epilys thanks!
Hopefully this would be the good one regarding container dependencies.

@aesteve-rh aesteve-rh force-pushed the virtio-media branch 2 times, most recently from e5a2e02 to 05a284d Compare May 21, 2026 14:58
@aesteve-rh

Copy link
Copy Markdown
Contributor Author

Looks much cleaner now. One of the remaining issues (with cargo-audit) is related with Gnurou/v4l2r#60, so it is a matter of time we get rid of it, hopefully.

The only other issue is with coverage. I will double check where's the gap and see if I can close it.

Thanks @stefano-garzarella and @epilys for the reviews and the patience!

@aesteve-rh aesteve-rh force-pushed the virtio-media branch 4 times, most recently from 7540e1a to 604b92c Compare June 29, 2026 12:55
@aesteve-rh aesteve-rh force-pushed the virtio-media branch 3 times, most recently from 937abba to 177bb16 Compare June 29, 2026 14:52
@aesteve-rh

Copy link
Copy Markdown
Contributor Author

The doc job in the pipeline seems to fail spuriously? I'd swear it passed in some of the re-runs.

The only other job failing (cargo-audit) is pending waiting on Gnurou/v4l2r#60. Maybe we can workaround it with a note/todo in the meantime? Even if the PR goes in, it will need to get a release pushed to crates.io, so we may as well temporarily workaround it.

Either way, the branch is probably in good position to get some early reviews? I understand that it is a big change and it is not easy to go through in one sit. I myself get a hard time when revisting changes in each commit and self-reviewing. I am open to suggestions in how to organize this to make reviews easier to handle.

aesteve-rh added a commit to aesteve-rh/qemu that referenced this pull request Jun 30, 2026
Add the QEMU side of the vhost-user-media device, which connects to a
virtio-media vhost-user backend daemon implementing the V4L2 API over
the virtio-media protocol.

The virtio-media device is specified in the VirtIO specification v1.4,
section 5.22:
    https://docs.oasis-open.org/virtio/virtio/v1.4/cs01/virtio-v1.4-cs01.html

Tested with the rust-vmm vhost-device-media backend [1]:

    cargo run -- -s /path/to/media.sock -d /dev/video0 --backend v4l2-proxy

Example invocation:

    qemu-system-x86_64 \
        -chardev socket,path=/tmp/media.sock,id=media \
        -device vhost-user-media-pci,chardev=media,id=media

[1] rust-vmm/vhost-device#944

Signed-off-by: Albert Esteve <aesteve@redhat.com>
Comment thread vhost-device-media/src/main.rs Outdated

@epilys epilys left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aesteve-rh maybe we could squash all the commits into one? Not sure if it makes a difference in review, for me viewing all the changes at once looks very readable as well. WDYT?

Comment thread vhost-device-media/src/main.rs Outdated
Comment thread vhost-device-media/src/main.rs Outdated
Comment thread vhost-device-media/src/main.rs Outdated
@aesteve-rh

Copy link
Copy Markdown
Contributor Author

@aesteve-rh maybe we could squash all the commits into one? Not sure if it makes a difference in review, for me viewing all the changes at once looks very readable as well. WDYT?

@epilys Some of the commits have some explanation/reasoning for the changes which would be lost otherwise. Maybe for those that only touch the code would make sense to squash. On the one hand, reviewing commit by commit has its problems as there is some back and forth with the structure, so a reviewer can comment on something that changed later on. Squashing would solve that.

On the other hand, reviewing only the final state is an option available for reviewers in github, just checking the Files Changed tab. So it is not really required?

So not sure, but it may make sense given that the changeset is so big, to just focus on the final state and add a single commit to the repository.

@epilys

epilys commented Jul 1, 2026

Copy link
Copy Markdown
Member

My suggestion: squash everything together, except for maybe the unit tests and CI if you wanna keep the commits small. You can merge parts of the commit texts from each squashed commit. As I said, the code changes are easy to read (for me), but the changing between commits is making it unnecessarily confusing. Just my 2 cents

@aesteve-rh

Copy link
Copy Markdown
Contributor Author

Done the squash, left the test, CI and some other build deps update separated. Also, the new additions that came from comment on this branch I left separated so that chages are easier to spot and because they can stand on their own anyway (i.e., changing xtask deserves a separate commit anyway as it is touching stuff outside the crate boundaries).

@aesteve-rh

Copy link
Copy Markdown
Contributor Author

CI is only failing due to a required anyhow update, which has already a PR. So all is good.

Btw, I already sent the frontend patch series to QEMU: https://lore.kernel.org/all/20260630112310.552606-1-aesteve@redhat.com/
In case anyone want to test it (instructions are in the cover letter).

@epilys

epilys commented Jul 1, 2026

Copy link
Copy Markdown
Member

OK went through everything again, LGTM overall. Let's wait for whatever CI/dependency todos we have and I think we're OK for merge, generally.

@epilys

epilys commented Jul 1, 2026

Copy link
Copy Markdown
Member

@aesteve-rh anyhow was fixed, you can now rebase on top of main

The virtio-media device first appeared in virtio-spec v1.4 [1].
It standardizes how guest applications communicate with media
devices on the host, such as encoders, decoders, and cameras,
using the V4L2 (Video4Linux2) protocol for guest-host
communication.

The low-level implementation of the virtio-media protocol is
provided by the device crate from the virtio-media repository [2].

The crate follows the same lib + binary structure as other
vhost-device crates: core logic lives in lib.rs, while main.rs
is a thin CLI entry point.

Each backend is gated by an optional cargo feature to minimize
binary size and the dependency footprint:

- `simple-capture`: simple V4L2 capture device
- `v4l2-proxy`: V4L2 proxy device (enabled by default)
- `ffmpeg`: software video decoding via the virtio-media-ffmpeg-decoder
  crate [3], useful for testing without dedicated hardware
- `null`: always-compiled no-op backend
- `xen`: enables Xen platform support for rust-vmm dependencies

A `V4l2DeviceType` helper resolves the device path given on the
command line to the appropriate VFL_TYPE_* kernel constant.

The `MediaAllocator`, which manages shared memory regions for the
device, is adapted from `AddressAllocator` in the crosvm `resources`
crate.

[1] https://docs.oasis-open.org/virtio/virtio/v1.4/virtio-v1.4.html#x1-82200022
[2] https://github.com/chromeos/virtio-media
[3] https://github.com/chromeos/virtio-media/tree/main/extras/ffmpeg-decoder

Signed-off-by: Albert Esteve <aesteve@redhat.com>
Add unit tests to existing code for vhost-device-media covering:

- descriptor_chain: DescriptorChainReader/Writer read/write paths
  and boundary behaviour.
- vhu_media_thread: VhostUserMediaThread queue processing helpers.
- vhu_media: VuMediaBackend construction, handle_event dispatch,
  and error-type conversions.
- lib: CLI argument parsing and socket-binding smoke tests for
  each backend.

Assisted-by: Claude <noreply@anthropic.com>
Signed-off-by: Albert Esteve <aesteve@redhat.com>
vhost-device-media depends on v4l2r, which uses bindgen in its build
script to generate Rust bindings from /usr/include/linux/videodev2.h.
That header includes <sys/time.h>, which on Debian/Ubuntu aarch64 is
placed under the multiarch path /usr/include/aarch64-linux-gnu/ rather
than directly under /usr/include/. Clang, invoked by bindgen, does not
add that directory to its default include search path, causing the build
to fail with:

  /usr/include/linux/videodev2.h:60:10: fatal error: 'sys/time.h' file
  not found

Set BINDGEN_EXTRA_CLANG_ARGS to pass the multiarch include directory to
clang for all GNU build and test steps. The path is derived from
$(uname -m) so it resolves correctly on x86_64 and aarch64 alike, and
is a no-op on any platform where the directory does not exist.

Additionally, the ffmpeg feature pulls in virtio-media-ffmpeg-decoder,
whose build script uses pkg-config to locate libavcodec. pkg-config
refuses to run for a musl cross-compilation target, and libavcodec is
not available as a musl-linked library regardless. Exclude
vhost-device-media from all musl build and test jobs, following the
same pattern already used for vhost-device-gpu.

Signed-off-by: Albert Esteve <aesteve@redhat.com>
Add a [patch.crates-io] override for v4l2r, redirecting every workspace
member that pulls v4l2r to the git revision 12afd03. That revision
replaces the unmaintained paste crate (RUSTSEC-2024-0436) with its
maintained successor pastey, resolving the cargo audit denial.

The direct dependency in vhost-device-media already pinned this
revision. The patch unifies the transitive pull through virtio-media
to the same source, eliminating duplicate crates-io entry from the
lockfile.

Signed-off-by: Albert Esteve <aesteve@redhat.com>
Move MediaArgs and BackendType into a dedicated args module so
that it can be reused by the xtask crate to generate a manpage
in a followup commit.

The From<MediaArgs> for VuMediaConfig conversion is moved to
lib.rs where both types are defined.

Signed-off-by: Albert Esteve <aesteve@redhat.com>
Add vhost-device-media support for the mangen task in xtask binary.

This generates a manpage in target/dist/man/vhost-device-media.1

Signed-off-by: Albert Esteve <aesteve@redhat.com>
This allows a service manager to start vhost-device-media with an
already listening socket. With this, the service manager can create
the socket in advance of starting any services, so there's no race in
between vhost-device-media being started and being ready to accept
connections.

Signed-off-by: Albert Esteve <aesteve@redhat.com>
@aesteve-rh

Copy link
Copy Markdown
Contributor Author

Passes the tests! Right now the only missing dependency is for vhost and vhost-user-backend that are overriding the workspace dependency (https://github.com/aesteve-rh/vhost-device/blob/faf6e0ef09495ce81dd30911c2db371f87a2b035/vhost-device-media/Cargo.toml#L30).

I can handle it in a follow up or wait for the change to propagate here. For it to land it requires virtio-queue release, then update vhost and vhost-user-backend PR in rust-vmm/vhost#348 (which I can help with after virtio-queue is released), and finally we can update them here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants