drm: Ensures that we can use the DRM format on the vulkan device

[marysaka]

Gamescope now select the DRM format by checking if the vulkan driver
can support those formats with the required image usage bits.

This fixes NVK where nouveau kernel module reports support for
XRGB2101010 and ARGB2101010 but the hardware cannot support those
formats as image storage

[misyltoad]

Shouldn't we just check if the format can do STORAGE?

[marysaka]

Shouldn't we just check if the format can do STORAGE?

Should be good now, I initially thought it would be way harder than that

[oSoMoN]

Vulkan-specific code should probably be moved into a function of its own in rendervulkan.{h,c}pp.

There is very similar code in the vulkan_init_format() function, can it be factored out?

[marysaka]

Vulkan-specific code should probably be moved into a function of its own in rendervulkan.{h,c}pp.

There is very similar code in the vulkan_init_format() function, can it be factored out?

Looking at it more, couldn't we just base it on sampledDRMFormats values?

If we modify vulkan_init_format to also check support for storage image and transfer src features and call vulkan_init_formats in init_drm we could just do that I think.

Of course vulkan_init_formats would need to exit early if sampledDRMFormats is already filled.

What do you think about that approach?

[matte-schwartz]

Closes #1717

[itsnoa04]

Tested this on an RTX 5070 (GB205) with nouveau/NVK, where I was hitting exactly the problem this PR describes.

Without this PR, the embedded session (gamescope-session-plus steam) crash-loops at startup: Assertion 'modifiers.size() > 0' failed in CVulkanTexture::BInit (rendervulkan.cpp), while gamescope creates its output image as XBGR2101010, the plane advertises the 10-bit formats but image creation for them fails on NVK.

With this PR applied on top of master (73950ef), the session starts cleanly and falls back to XR24 as intended. Steam Deck UI comes up and works.

Setup: Arch Linux, kernel 7.1.3, Mesa/NVK 26.1.4, nouveau with GSP 570.144 and nouveau.atomic=1, gamescope embedded DRM backend, Steam Deck client.

Full disclosure: I'm a hobbyist and I debugged this with help from an AI assistant, but the test itself is a real boot on real hardware, crash-loop without the patch, working session with it. Happy to re-test future revisions of this PR on this machine.