Support YUV 4:4:4 streaming

[ns6089]

Add support for YUV 4:4:4 streaming

moonlight-common-c pull request: moonlight-stream/moonlight-common-c#91 merged
sunshine pull request: LizardByte/Sunshine#2533 draft

Todo, in no particular order:

• Add Vulkan headers to CI environment
  • Probably the easiest way is to add them to moonlight-qt-prebuilts submodule, collected from https://github.com/KhronosGroup/Vulkan-Headers
• Test decoding on intel (i)gpus
• Implement "YUV 4:4:4" option on settings page
  • implemented as dumb switch, no intelligent graying out
• Generate and include missing blobs of encoded video data used in decoder availability auto-detection
  • may want to revise the methodology
• Change plvk renderer heuristics to support 10-bit SDR video decoding on SDR displays, currently it tests for VK_COLOR_SPACE_HDR10_ST2084_EXT and it fails when display is not in HDR mode
  • Removed this check altogether, libplacebo might need some reconfiguration to handle tone mapping gracefully
• Investigate and possibly resolve color conversion errors
  • Full Range had issues on sunshine's side, fixed in the same 4:4:4 PR
  • Rec.601 should never be requested on third-party renderers because it has different primaries from sRGB

[ns6089]

@cgutman I probably can't do more on moonlight side (moonlight-common-c included) without feedback. Format selection logic appears overcomplicated to me, so I've probably missed some corner cases, and you might prefer to do some things differently altogether. Also if you prefer, you can simply take this over after 5.1 is released, this may prevent the needless back-and-forth.

[ns6089]

• switched from fork to official moonlight-common-c submodule
• rebased against 6.0 reverting library updates that got rid of vulkan/libplacebo
• added command line option for YUV 4:4:4
• extended 4:4:4 format selection logic to cover forced codecs

[moi952]

Hi @ns6089, this development is planned to be developed on moonlight-android?

[ns6089]

this development is planned to be developed on moonlight-android?

Not anytime soon, no hardware support.

[ns6089]

• rebased against latest master
• added 2x multiplier to default bitrate when 4:4:4 option is enabled

[Decbuan]

• 根据最新的 master 进行重新定位
• 启用 4:4:4 选项时，默认比特率增加 2 倍乘数

Keep it up! Looking forward to the success of YUV444!! For high-end computer users and those with WiFi 6 or above, we hope for higher bitrate requests and better picture encoding and transmission.

[Decbuan]

• 根据最新的 master 进行重新定位
• 启用 4:4:4 选项时，默认比特率增加 2 倍乘数

I have obtained the version you uploaded on AppVeyor, and I know that this requires support for synchronous encoding by Sunshine, which is not currently in effect. What I want to point out is that when I enable the 444 button, my bitrate can only go up to 114mbps. If 444 is implemented, it shouldn't be limited to this range, right?

[ns6089]

• disabled H.264 4:4:4 for now due to lack of hardware decoders and incomplete (at least to my understanding) software decoding support on plvk backend

[moi952]

@ns6089 Thank you for your work !
I see that the application is not building for Windows, is this normal?

[ns6089]

I see that the application is not building for Windows, is this normal?

The repository with binary dependencies needs to be updated, I'm trying to get this sorted out.

[ns6089]

Can't be done before 6.0.1 hotfix goes live it seems.

[moi952]

Even recovering v6 it doesn't build?

[ns6089]

Even recovering v6 it doesn't build?

If you're trying to build outside of CI on your own computer, this PR currently contains commits that revert binary deps to somewhat functional state. But you also need local vulkan headers, with %VULKAN_SDK% env variable pointing at them. installing SDK from https://vulkan.lunarg.com/sdk/home#windows should be enough.

[cgutman]

@ns6089 I've updated the binary deps, so this should be ready to land once you rebase and resolve the conflicts.

[ns6089]

@cgutman Great, I will get onto it.
In the meantime, what are we going to do with vulkan headers on Windows in general, and in CI in particular?
We can grab a copy from https://github.com/KhronosGroup/Vulkan-Headers and put it into build-deps as well. Or just add it as a submodule, but not sure if anything other than Windows can make use of this.

[ns6089]

Rebased. Everything appears to be working including software decoding.
Now we just need to decide what to do with vulkan headers.

[ns6089]

on linux where they aren't shipping the video extensions

True I suppose, the merge request to hasvk got stalled (I misread current support status in the corresponding mpv discussion)

Actually, no (something something gaslighting is not real). ANV driver seems to have support, while HASVK is stuck in a limbo. No idea about 4:4:4 though.

@Kobain-Seo

It showed green screen.

Yeah, windows driver at this moment seem to have this problem. Intel is actively working on this driver though.

[KWottrich]

Try opening it in in mpv https://mpv.io/ while forcing vulkan decoding with the following args mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan --gpu-context=winvk that_file.mp4

Yep, file fails to play. Log output from mpv --vo=gpu-next --gpu-api=vulkan --hwdec=vulkan --gpu-context=winvk --msg-level=vd=v that_file.mp4:

● Video  --vid=1  (hevc 1920x1080 30 fps) [default]
File tags:
 Artist: Blender Foundation 2008, Janus Bager Kristensen 2013
 Comment: Creative Commons Attribution 3.0 - http://bbb3d.renderfarming.net
 Composer: Sacha Goedegebure
 Genre: Animation
 Title: Big Buck Bunny, Sunflower version
[vd] Container reported FPS: 30.000000
[vd] Codec list:
[vd]     hevc - HEVC (High Efficiency Video Coding)
[vd]     hevc_qsv (hevc) - HEVC video (Intel Quick Sync Video acceleration)
[vd]     hevc_cuvid (hevc) - Nvidia CUVID HEVC decoder
[vd] Opening decoder hevc
[vd] Looking at hwdec hevc-vulkan...
[vd] Trying hardware decoding via hevc-vulkan.
[vd] Selected decoder: hevc - HEVC (High Efficiency Video Coding)
[vd] Pixel formats supported by decoder: vaapi cuda vulkan yuv444p
[vd] Codec profile: Rext (0x4)
[vd] Requesting pixfmt 'vulkan' from decoder.


(program exits here without playing any video)

I referred to @vadash's driver versions that produce a working example, and after doing a full clean reinstall of 537.58 after using DDU to clean my updated driver on my laptop, I can confirm that this is now fully working for me as well. This confirms that NVidia must have introduced a bug in their drivers. Thank you!

[Erfboom]

hey guys, i'm eager to test this but it seems the api limit for downloading artifacts in appveyor was exceeded. can anyone share the appimage? or i can be patient and wait?

[RedTib]

537.58

Using this driver worked for me as well, but it did not work with HDR and 4:4:4 enabled at the same time.

[ns6089]

hey guys, i'm eager to test this but it seems the api limit for downloading artifacts in appveyor was exceeded. can anyone share the appimage? or i can be patient and wait?

https://github.com/ns6089/moonlight-qt/releases/tag/pr1282
Exceeds 25MB attachment limit, hence the roundabout way,

[Erfboom]

hey guys, i'm eager to test this but it seems the api limit for downloading artifacts in appveyor was exceeded. can anyone share the appimage? or i can be patient and wait?

https://github.com/ns6089/moonlight-qt/releases/tag/pr1282 Exceeds 25MB attachment limit, hence the roundabout way,

You're too kind, but would you happen to have the AppImage? Am on Linux.

[ns6089]

No, sorry.

[Decbuan]

Steamdeck oled does not support 444 decoding
Bad news

[Erfboom]

Needs to be software for AMD I believe.

[KWottrich]

hey guys, i'm eager to test this but it seems the api limit for downloading artifacts in appveyor was exceeded. can anyone share the appimage? or i can be patient and wait?

https://github.com/ns6089/moonlight-qt/releases/tag/pr1282 Exceeds 25MB attachment limit, hence the roundabout way,

You're too kind, but would you happen to have the AppImage? Am on Linux.

@Erfboom download limit reset: https://ci.appveyor.com/project/cgutman/moonlight-qt/builds/50226576/job/sf7l969npglkxddn/artifacts

If you can't get it, I was able to download and can figure out how to share/send it

[ns6089]

I've added it to the same temporary release page https://github.com/ns6089/moonlight-qt/releases/tag/pr1282

[Erfboom]

Thanks guys, I was able to pull it from appveyor.

Tests on AMD: works with software only. bitrate requirements go up. windows 11 with a 3080 and sunshine got wonky and i experienced hella latency. rebooting the machine fixed it. (may have helped color banding some :(, but still there). Things do look better though, sharper. can't do 4k@60 without hardware decoding. capped at 1440p@60 for stable frames, but with the bitrate lowered and no ability to change it, i started to see artifcating.

Tests on Intel: lol. I don't know if my iGPU is too old (10th gen comet lake), and I admit i'm running it in a distrobox, but after some fiddling I got hardware decoding to be detected by moonlight. hw decoding works, but not with YUV4, says gpu doesn't support it. vdpauinfo backs that up, don't know how to read vainfo to tell if that's the case. software decoding allows 4:4:4, but i can't get in stream, don't know what I'm missing.

This was all done using the AppImage, opensuse Aeon (tumbleweed distrobox) and opensuse tumbleweed plasma6.

I can whip up a quick vm with an nvidia card to test moonlight there too, but i don't want to be heartbroken when i see that it works great on hw decoding.

[ns6089]

@Erfboom

Tests on AMD

AMD lacks the support for native 4:4:4 on the hardware level. I will add a special "recombined" 4:4:4 mode a bit later, where full 4:4:4 data is spread across a larger 4:2:0 frame.

Tests on Intel

Intel does support 4:4:4 decoding on gen10 cpus and newer, so it's a driver issue.
Currently moonlight only tries vulkan hardware decoding to not contaminate the testing. This should be Intel ANV driver in your case and it may still have issues with vulkan video extensions.
I guess we can try enabling vaapi too (it has 4:4:4 support) later if vulkan driver is in a bad state. But the process will be somewhat troublesome (decoder priorities and a myriad of different linux configurations).

[Erfboom]

@Erfboom

Tests on AMD

AMD lacks the support for native 4:4:4 on the hardware level. I will add a special "recombined" 4:4:4 mode a bit later, where full 4:4:4 data is spread across a larger 4:2:0 frame.

Tests on Intel

Intel does support 4:4:4 decoding on gen10 cpus and newer, so it's a driver issue. Currently moonlight only tries vulkan hardware decoding to not contaminate the testing. This should be Intel ANV driver in your case and it may still have issues with vulkan video extensions. I guess we can try enabling vaapi too (it has 4:4:4 support) later if vulkan driver is in a bad state. But the process will be somewhat troublesome (decoder priorities and a myriad of different linux configurations).

The latency I was talking about is definitely because my CPU can't decode that much data coming through on a higher bitrate, but I do definitely see a difference now between YUV4:4:4 and YUV4:2:0. Wish I could use it!

I suspected it was a driver issue. Intel iGPUs are not in a good place in linux for media. They're fine for the most part but when you get into this stuff you can see the cracks.

I tried VAAPI, so that explains it. I did notice Vulkan errors. Moonlight kept on trying to use GL. I'm not familiar with the Intel ANV driver, but maybe I can give that a shot :). Thanks for your hard work! This is really cool stuff.

[mirh]

You need ANV_VIDEO_DECODE=1

[Erfboom]

More tests.

Test on Steamdeck:

Crisp and clear with YUV4:4:4, with software decoding. can handle 1080p@60 just fine though and looks great!

Wish i could figure out what filter gamescope / steam is using to add noise so i could turn it off.

More Intel Tests:

Software decoding works with or without ANV_Video_decode. After doing some digging, it looks like that's related to the new IRIS driver? If so, that only supports GEN11 and newer. I have the UHD driver for GEN10.

A note on the HDR. On KDE6 Plasma, with an HDR monitor, moonlight could not properly match the inbound hdr colorspace to what the intel igpu could decode resulting in a crash. I believe that was related to libplacebo or mesa itself.

This time around, the HDR button was able to be toggled without gamescope, on gnome (no hdr), probably because it's running in a distrobox and tonemapping was applied properly. Not sure what's going on there.

[mirh]

Iris is the opengl driver for gen8-9, it has nothing to do with this.
And vulkan video decode is possibly going to ship for gen7 too.

Gen11 gpus are required for 444 because they are the first to have the required hardware to begin with.

[cgutman]

It looks like the FFmpeg bug is being tracked upstream as https://trac.ffmpeg.org/ticket/10847

I'll merge this, but the Nvidia driver crash is probably a ship blocker until we can get a patch to address it (preferably upstreamed).

[cgutman]

@ns6089 Thanks for all your hard work on this!

[PmNz8]

Hello all, using MoonlightPortable-x64-r2552 I can not use HW for decoding 4:4:4 despite having i5 11400T with HD 730 that should support 4:4:4 decode of H265. I can only get 4:4:4 stream from Sunshine https://github.com/LizardByte/Sunshine/actions/runs/10206571176 when using software decode in Moonlight.
Moonlight claims there is no HW support:

Maybe some issue with detection of HW decoder capabilities?
Moonlight-1722769990.log

[cgutman]

It's because D3D11VA doesn't have YUV 4:4:4 support yet since Microsoft just updated the DXVA specification to support it and the Intel Windows driver seems to have broken Vulkan Video decoding support. The Intel drivers do support the new DXVA profiles for YUV 4:4:4 but it also requires support in FFmpeg to work.

[mirh]

Actually it's the other way around.
Since forever the intel drivers already support the old profiles they rolled out on their own, and it's unclear just how much further back in time such support would go (regardless, yes, you need patched ffmpeg)

[cgutman]

It's both. Modern Intel drivers expose the old Intel-specific profiles and the new Microsoft-defined official profiles. Unless FFmpeg is willing to accept Intel-specific profiles, I'm going to only support the official profiles which leaves out Ice Lake.

[mirh]

I don't see why they wouldn't. It's not like they aren't already supporting proprietary extensions.. and presumably this is just a matter of trying an additional guid, if the official one fail?

[nyanmisaka]

The two GUIDs correspond to different dxva structures (the bits are in different order, so it's not just as simple as switching GUIDs). Intel's dxva structure is dumped from the driver DDI and is not in any public documentation. Only Intel knows the details. Unless you want to hardcode it in FFmpeg instead of including the dx header files from Microsoft.

[KWottrich]

It looks like the bitrate slider isn't working when YUV 4:4:4 is enabled. It calculates a default value, but changing the slider doesn't persist when you close the settings with the back arrow in the upper left and immediately reopen the menu. Since this is already merged, I created a bug report for this: #1387