Add planar YUV texture support

[EIREXE]

Describe the project you are working on

Video decoding for godot

Describe the problem or limitation you are having in your project

While you could make the video decoder give out a bunch of textures and sample them individually, this is cumbersome and costly, I think it would make more sense for it to be exposed as a texture directly using the appropiate vulkan conversion methods.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Add the ability to load planar textures of the YUV format, particularly the common YUV4:2:0 type.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

(Probably) Add a new texture resource, PlanarTexture, which can be given images or textures to place at each plane, together with the planar format that will be sampled.

There's some information of planar conversions in here: https://themaister.net/blog/2019/12/01/yuv-sampling-in-vulkan-a-niche-and-complicated-feature-vk_khr_ycbcr_sampler_conversion/

I believe this extension should be available on most platforms that support vk1.1

If this enhancement will not be used often, can it be worked around with a few lines of script?

Not really.

Is there a reason why this should be core and not an add-on in the asset library?

It's pretty much a core-only feature

[QbieShay]

Would they work with a 3D texture?

[EIREXE]

Would they work with a 3D texture?

Yes, but godot expects a 2d texture from VideoStreamPlayback. (sorry for the late reply btw)

[voylin]

I don't think that this should be a part of the Godot editor as I see little use for this to game developers. For video decoding this would be helpful, I'm making GoZen, a video editor in Godot, and it would be beneficial to have. However, I see more use in this being a GD Extension instead.

A GD Extension which would support different video types would be helpful, but as many games don't use video, and for the couple which do, the already included one would be good enough in most cases. I do agree that we should have options to use multiple video formats, but I also think that the current VideoStreamPlayback will be lacking for use with other codecs.

My suggestion/proposal: Remove the VideoStreamPlayback out of the Godot project. Most game developers don't use it, and the ones who do want to use it would prefer to use mp4 files anyway. Those have licensing issues because of h.264 and h.265, so it can't be included in Godot without problems. So let's remove VideoStreamPlayback, and let's create a GDExtension which creates the functionality instead.

Result would be, better performance, better codec compatibility and reduce the size of Godot by a tiny amount.

This all is something I'll be working on in the coming months whilst figuring out how to do this for my video editor. When I have these things figured out I'd like to make GD Extensions for everybody to use so people can use videos in their projects which won't cause performance issues.

[Calinou]

Remove the VideoStreamPlayback out of the Godot project.

This is already being tracked in #3286, but now that we have Theora support still in 4.x, we can't remove it without breaking compatibility. I also think Godot should offer some kind of built-in video playback functionality that is free of patent restrictions, even if quality isn't the best. Many developers are located in countries where software patents are enforced after all. It's all about getting something working as quickly as possible.

Regarding YUV sampling/conversion, it's something that needs to be very fast for performance to be acceptable, so core changes may be needed regardless (even if we don't add support for more video formats ourselves).