Move video playback out of core and into an officially supported GDExtension

[Calinou]

Note: This change was discussed with reduz and others and is probably good to implement.

Describe the project you are working on

The Godot editor 🙂

Describe the problem or limitation you are having in your project

Video playback in Godot currently leaves a lot to be desired:

• It has a lot of bugs (see open bug issues for Theora, WebM).
• There are performance issues due to not being able to benefit from hardware video decoding, especially on mobile hardware.
• It takes valuable space in the binary due to video decoding libraries being large. (libtheora and libvpx also have their own dependencies that could be removed from core, such as libopus.)
• It lacks features that are a deal-breaker in some projects such as seeking.
• The number of supported formats is fairly limited, with patent-encumbered formats not being supported.
• Library updates are difficult for maintainers to perform. In general, libvpx (used for VP8 and VP9 decoding) isn't exactly known to be easy to work with.
• It's difficult to compile on unconventional platforms, making the build process unnecessarily complicated there.

We have very few contributors knowledgeable with video decoding libraries, so bug fixes and improvements are rarely seen nowadays.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

With GDExtension (the replacement of GDNative in 4.0), we can move video decoding to an officially supported add-on. This add-on will likely use FFmpeg like godot-videodecoder currently does, but it may also use another library depending on code size, maintenance quality and licensing.

There are many benefits to moving video playback out of core:

• The binary size penalty is removed from the Godot editor and export template binaries. Instead, the size cost is moved to the add-on's compiled libraries.
• Hardware video decoding could be used on compatible hardware and software.
• Support for patent-encumbered formats can be exposed optionally. (If this is implemented, this will be disabled by default.)
  • Users will need to check their country's regulations and possibly acquire licenses, especially for commercial use. It is possible for open source software to support patent-encumbered formats – otherwise, VLC would not be able to play back .mp4 videos 🙂

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Perform a change like godotengine/godot#52003, but for VideoPlayer.

If this enhancement will not be used often, can it be worked around with a few lines of script?

No.

Is there a reason why this should be core and not an add-on in the asset library?

Video decoding needs to have hooks in the engine to be efficiently implemented, so it needs dedicated GDExtension work.

[fire]

I'll contact anyone still on the video decoder team at https://github.com/kidrigger/godot-videodecoder.

@kidrigger @jamie-pate

[jamie-pate]

For the video extension, i just noticed that one of my devs is running Apple silicon but we didn't notice the gdnative dylib failed to load because it falls back to the built in playback (graceful degradation since it doesn't support stuff like steam position), so it would be best to support building for all platforms when the built in video player support is removed.

We also have no build support currently for web or mobile.

To do this properly we need yuv texture support because that's a how all the hw accelerated decoders work.

I've seen shaders that do it so maybe it just needs to be an option on the spatial material

[kidrigger]

I'm working on the GDExtension support now. I'll add a new proposal but here's the gist of it.

VideoStream and VideoStreamPlayback classes need work to be extended from GDExtension.
There are two options

1. Keep our playback and let plugins add decoders
   • This is the way GDNative plugin currently works.
   • This will be faster to implement as we just need the old Decoder Interface to be updated for GDExtension.
2. Make the playback classes GDExtension capable and let the Extension implementation set their own.
   • This option adds flexibility for the extension developers
   • Reduces the playback logic on Godot core
   • Bypasses the performance constraints caused due to design of the GDNative implementation. Will improve performance by allowing the implementation to decide playback logic
   • This will take longer as the new Playback extensions will need to be developed

I have started work on Option 2 unless Option 1 is asked for due to time constraints.

[Catchawink]

@kidrigger any updates on this or ways I could possibly help? I'm looking to address #2553 and it appears to be contingent on this one.

[kidrigger]

@Catchawink godotengine/godot#62737
The PR needs to be merged.
The WebM plugin is working pretty well otherwise.

Resolving conflicts right now: ETA < 1 hr.
All conflicts have been resolved.

Who should I asked to review this quickly?

[kidrigger]

So, how does one go about making a plugin officially supported?

This plugin has all the WebM code in working condition

So perhaps it might be good to know where to put it so others can weigh in and fix some bugs.

[Calinou]

So, how does one go about making a plugin officially supported?

We can transfer it to the @godotengine organization, as done for https://github.com/godotengine/webrtc-native (for example).

I can't do it myself but @akien-mga should be able to.

[makemefeelgr8]

It lacks features that are a deal-breaker in some projects such as seeking.

Yup, it just killed one more project. I wonder what is the actual "kill count" in 2 years the issue have been open.
So, instead of chasing the impossible and unimplementable original approach, I suggest the easier way to get things done. At least for desktop PC platforms.

OpenCV.

Workflow would be like:

1. Use OpenCV to decode some frames.
2. Draw those frames on a texture.

Just how cool is that! Any video format is supported.
It will even deal with seeking, rewinds and so on.
And the effort to implement it is like non-existant. Compile the library with some flags and you're done.

[Calinou]

OpenCV.

As mentioned above, large libraries like FFmpeg and OpenCV won't be considered for inclusion in core due to their binary size. What an extension uses is less important, but if you're able to integrate FFmpeg in an extension, integrating OpenCV isn't really worth the effort.

That said, an extension can be smaller and easier to build if it focuses on integrating only essential patent-unencumbered formats (VP8, VP9, perhaps AV1). FFmpeg and OpenCV are notoriously difficult to build from source after all.

An alternative would be to use Vulkan Video to rely on hardware-accelerated decoding, but it's poorly supported across the board and there is no OpenGL equivalent (for the Compatibility rendering method).

In the meantime, you can try using this GDExtension: https://github.com/EIRTeam/EIRTeam.FFmpeg

[makemefeelgr8]

It turned out to be way easier than expected. Here's the OpenCV PoC:

using Godot;
using OpenCvSharp; // Install https://www.nuget.org/packages/OpenCvSharp4.Windows
using System;

public partial class VideoSprite : Sprite2D
{
    [Export]
    public string VideoPath { get; set; }
    private VideoCapture _capture;
    private double _playbackPosition;

    public override void _Ready()
    {
        _capture = new VideoCapture(this.VideoPath);
        if (!_capture.IsOpened())
            throw new Exception($"Failed to open {this.VideoPath}");
        _playbackPosition = 0;
    }

    public override void _Process(double delta)
    {
        _playbackPosition += delta;
        var currentFrame = (uint)(_playbackPosition * _capture.Fps);
        if (_capture.PosFrames >= currentFrame)
            return; // It's already displaying the correct frame

        var frame = new Mat();
        _capture.Read(frame);
        if (frame.Empty())
            return; // Video is over

        // Convert frame data to godot boilerplate
        var bmpBytes = frame.ToBytes(ext: ".bmp");
        var boilerplate = new Godot.Image();
        boilerplate.LoadBmpFromBuffer(bmpBytes);
        var texture = ImageTexture.CreateFromImage(boilerplate);
        this.Texture = texture;
    }
}

It results in about 45fps now, because of the stupid conversions, that go like:

Video Frame -> Raw OpenCV -> OpenCV mat -> bmp bytes -> godot image -> godot texture -> Sprite2D.

If one were to shave a few steps from the list, it would yield an acceptable performance for sure.

Also, one should decouple video frame processing from the rendering thread (for obvious reasons). So, the real code would be more like:

using Godot;
using OpenCvSharp; // Install https://www.nuget.org/packages/OpenCvSharp4.Windows
using System;
using System.Threading;
using System.Threading.Tasks;

public partial class VideoSprite : Sprite2D
{
    [Export]
    public string VideoPath { get; set; }
    private VideoCapture _capture;
    private ImageTexture _texture;
    private Task _playVideoTask;
    private Image _image;

    public override void _Ready()
    {
        // Pefrorm the 1st draw and init textures
        _capture = new VideoCapture(this.VideoPath);
        if (!_capture.IsOpened())
            throw new Exception($"Failed to open {this.VideoPath}");
        _image = new Image();
        var frame = new Mat();
        _capture.Read(frame);
        _image.LoadBmpFromBuffer(frame.ToBytes(ext: ".bmp"));
        _texture = ImageTexture.CreateFromImage(_image);
        this.Texture = _texture;

        // Start processing task
        _playVideoTask = new Task(() => PlayVideo());
        _playVideoTask.Start();
    }

    private void PlayVideo()
    {
        var startTime = DateTime.UtcNow;
        while (true)
        {
            double playbackPosition = (DateTime.UtcNow - startTime).TotalSeconds;
            var currentFrame = (int)(playbackPosition * _capture.Fps);
            if (_capture.PosFrames >= currentFrame)
            {
                // It's already displaying the correct frame. Wait for half a frame time and try again
                Thread.Sleep((int)(1000 / (_capture.Fps/2)));
                continue;
            }

            var frame = new Mat();
            _capture.Read(frame);
            if (frame.Empty())
                return; // Video is over, exit

            // Convert frame data to godot boilerplate
            var bmpBytes = frame.ToBytes(ext: ".bmp");
            var boilerplate = new Image();
            boilerplate.LoadBmpFromBuffer(bmpBytes);
            Interlocked.Exchange(ref _image, boilerplate);
        }
    }

    public override void _PhysicsProcess(double delta)
    {
        _texture.Update(_image);
    }
}

[jamie-pate]

To get decent performance (hw acceleration) you will want to write yuv video frames directly to the texture using opengl or vulkan and then use a shader to convert to rgb. Last time I looked there was no path to do that in godot.. Any other path for decoding video from standard formats will incur a lot of overhead. How much overhead is acceptable depends on the size of your video frames vs the size of your cpu...

…

On Wed., Oct. 25, 2023, 1:51 a.m. makemefeelgr8, ***@***.***> wrote: It turned out to be way easier than expected. Here's the OpenCV PoC: using Godot;using OpenCvSharp; // Install https://www.nuget.org/packages/OpenCvSharp4.Windowsusing System; public partial class VideoSprite : Sprite2D{ [Export] public string VideoPath { get; set; } private VideoCapture _capture; private double _playbackPosition; public override void _Ready() { _capture = new VideoCapture(this.VideoPath); if (!_capture.IsOpened()) throw new Exception($"Failed to open {this.VideoPath}"); _playbackPosition = 0; } public override void _Process(double delta) { _playbackPosition += delta; var currentFrame = (uint)(_playbackPosition * _capture.Fps); if (_capture.PosFrames >= currentFrame) return; // It's already displaying the correct frame var frame = new Mat(); _capture.Read(frame); if (frame.Empty()) return; // Video is over // Convert frame data to godot boilerplate var pngBytes = frame.ToBytes(ext: ".png"); var boilerplate = new Godot.Image(); boilerplate.LoadPngFromBuffer(pngBytes); var texture = ImageTexture.CreateFromImage(boilerplate); this.Texture = texture; }} It results in about 15fps now, because of the stupid conversions, that go like: Video Frame -> Raw OpenCV -> OpenCV mat -> png bytes -> godot image -> godot texture -> Sprite2D. If one were to shave a few steps from the list, it would yield an acceptable performance for sure. Also, one would have to decouple video frame processing from the rendering thread (for obvious reasons). — Reply to this email directly, view it on GitHub <#3286 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAFGVIRGTVRIXYSVTUL6VC3YBDHJJAVCNFSM5D3SOA62U5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TCNZXHA4DANZQGIYQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[makemefeelgr8]

To get decent performance (hw acceleration) you will want to write yuv video frames directly to the texture using opengl or vulkan and then use a shader to convert to rgb. Last time I looked there was no path to do that in godot.. Any other path for decoding video from standard formats will incur a lot of overhead. How much overhead is acceptable depends on the size of your video frames vs the size of your cpu...

There's an easier way. I just wish godot exposed some raw opengl functions. I could pass the pointer to OpenCV output, bypassing the whole conversion flow.

glTexImage2D(
  GL_TEXTURE_2D,
  0,
  GL_RGB,
  image.cols,
  image.rows,
  0,
  GL_BGR,
  GL_UNSIGNED_BYTE,
  image.ptr()
);

Video Frame -> Raw OpenCV -> OpenCV mat -> bmp bytes -> godot image -> godot texture -> Sprite2D -> OpenGL

would turn into

Video Frame -> Raw OpenCV -> OpenGL

But (unlike Unity where GL is exposed) Godot faced no AAA gamedev, so, I assume there was no need for opengl functions in c#. Or for video playback.

[fire]

There's an external texture code, @BastiaanOlij but I don't know if it's usable for this purpose.

[BastiaanOlij]

Is this Godot 3 or 4?

I don't have any experience with C#, only C++/GDExtension when it comes to this. Seeing how OpenGL works, you can just include OpenGL and you should be running within the correct context. There may be some threading issues and we do plan on improving how OpenGL is exposed in Godot 4.

In Godot 3 you can use VisualServer.texture_get_texid to retrieve the OpenGL texture ID for an existing texture, then just use OpenGL commands to load in data.
In Godot 4 you can use RenderingServer.texture_get_native_handle to do the same.

In both cases you would use the VisualServer/RenderingServer to create a new texture object, and then load data in.

There was more logic I was working on for ARCore but that never got merged as we never managed to get ARCore working as a plugin.

[DeeJayLSP]

The only advantage I can see in OpenCV over FFmpeg might be licensing (permissive over copyleft).

However, unless you plan to export to consoles (or any platform with NDAs) I don't see the issue with FFmpeg as its license allows not disclosing source in case of shared libs.

Now somewhat breaking the topic

When it comes to multiplatform exporting that includes consoles (most of them have some built-in media libraries you can link against) I'd say a custom wrapper is a better option than any of the above.

[makemefeelgr8]

RenderingServer.texture_get_native_handle

I'm talking Godot 4. Thanks for trying to help. I tried using the RenderingServer and RenderingDevice combo, except none of it worked. For example, this one always returns an empty array:

var txData = RenderingServer.GetRenderingDevice().TextureGetData(_texture.GetRid(), 0);

This one always throws:

RenderingServer.GetRenderingDevice().TextureUpdate(_texture.GetRid(), 0, _rawData);

And method description is as cryptic as they make them:

        // Summary:
        //     Returns the texture data for the specified layer as raw binary data. For 2D textures
        //     (which only have one layer), layer must be 0.
        //     Note: texture can't be retrieved while a draw list that uses it as part of a
        //     framebuffer is being created. Ensure the draw list is finalized (and that the
        //     color/depth texture using it is not set to Godot.RenderingDevice.FinalAction.Continue)
        //     to retrieve this texture. Otherwise, an error is printed and a empty System.Byte[]
        //     is returned.
        //     Note: texture requires the Godot.RenderingDevice.TextureUsageBits.CanCopyFromBit
        //     to be retrieved. Otherwise, an error is printed and a empty System.Byte[] is
        //     returned.

It's an anti-pattern. Ensure the draw list is finalized. How? texture requires the Godot.RenderingDevice.TextureUsageBits.CanCopyFromBit to be retrieved. How?

It should be like Ensure the draw list is finalized by calling X.EnsureFinalized().

I also tried switching the engine to opengl, and writing some texture bytes using OpenTK, like:

        _textureHandle = RenderingServer.TextureGetNativeHandle(_texture.GetRid());
        GL.BindTexture(TextureTarget.Texture2D, (int)_textureHandle);
        GL.TexSubImage2D(TextureTarget.Texture2D, 0, 0, 0, frame.Cols, frame.Rows, PixelFormat.Bgra, PixelType.UnsignedByte, _rawData);
        GL.BindTexture(TextureTarget.Texture2D, 0);

But it just threw some access violations & memory errors from the unmanaged part. The context might be the reason, though, as I init the library like:

        GL.LoadBindings(new OpenTK.Windowing.GraphicsLibraryFramework.GLFWBindingsContext());

I assume there's a need to grab GL context from godot somehow, for it to actually draw something.

[makemefeelgr8]

The only advantage I can see in OpenCV over FFmpeg might be licensing (permissive over copyleft).

With OpenCV you're also getting a bunch of powerful tools for image editing, object recognition, machine learning, and so on.

[DeeJayLSP]

With OpenCV you're also getting a bunch of powerful tools for image editing, object recognition, machine learning, and so on.

Yes, but the discussion here is mainly about video decoding. Currently there's no huge demand for those features. And if someone really needs it, they can always extend the engine (for video decoding as Calinou mentioned, there's already a FFmpeg-based video decoder by EIRTeam).

Also, to correct my previous statement, I peeked into OpenCV's source code and it seems to make use of FFmpeg itself for video decoding, therefore the copyleft licensing part also applies to OpenCV. If one wants to do video decoding with it, why not just use FFmpeg directly?

[makemefeelgr8]

there's already a FFmpeg-based video decoder by EIRTeam

But the implementation is horrible! It's no different from my code above. It's even worse. Have you seen their code? Those guys copy a video frame line by line in a loop. And then they use image.set_data and image texture. update. They lock a mutex 3 times per frame.

[JanWerder]

Wouldn't including libvlc be an alternate solution? (LGPL2.1 license)
There's one project that has done that: https://github.com/RancidMilkGames/Godot-VLC

[Calinou]

Wouldn't including libvlc be an alternate solution? (LGPL2.1 license) There's one project that has done that: RancidMilkGames/Godot-VLC

LGPL libraries can't be integrated in core, as per Best practices for engine contributors:

Libraries must use a permissive enough license to be included into Godot. Some examples of acceptable licenses are Apache 2.0, BSD, MIT, ISC, and MPL 2.0. In particular, we cannot accept libraries licensed under the GPL or LGPL since these licenses effectively disallow static linking in proprietary software (which Godot is distributed as in most exported projects). This requirement also applies to the editor, since we may want to run it on iOS in the long term. Since iOS doesn't support dynamic linking, static linking is the only option on that platform.

While they could be used in an official extension (as extensions are dynamically linked), dynamic linking isn't allowed on all platforms. Also, I think Godot should remain fully permissively licensed, including official extensions to avoid licensing pitfalls that other projects have encountered (such as Qt). Complying with the LGPL is possible for many projects, but that doesn't mean it's easy.

[lostminds]

As finding a decoder library with suitable features, license and platform support seems difficult, could an alternative be to instead rely on platform specific built in system services for this? At least for MP4 / H264 there seems to be support for this in Windows via Microsoft Media Foundation, macOS/iOS via CoreVideo, Android via MediaCodec and possibly in some linux versions as well?

The tradeoff would of course be that more platform specific code would be needed, and feature support will vary. And for web deployments for example this might not be possible. But even a lowest common denominator feature set will likely be a big improvement over the current support, and while not ideal I'm sure it won't be the only feature that is not available on all platforms. One issue though is that the currently only supported format OGV/Theora is likely not supported by any of these system services, since it's so unusual. So moving exclusively to using system decoders would break compatibility with existing projects using these videos.

[voylin]

@Calinou I have successfully made a GDExtension which has video and audio playback with seeking. Not certain if this could be turned into an official extension though. I know a lot more work would be necessary to make it more user friendly. It uses FFmpeg

If I were to be allowed to work on an official gdextension, how would the process be? I'd probably also look into #8049 to see if it's possible for me to also implement that as I both need it for my video editor and as it could improve performance for people using the GDExtension.

[voylin]

A little update on this, I got a GDExtension working for creating video playback. It is not a drop in replacement as you still need to write the code yourself for displaying the frames in order. But it is a beginning. https://github.com/VoylinsGamedevJourney/gde_gozen

Only question would be is if I can turn this into an officially supported GDExtension as I don't know how it can become "officially supported" ^^" So if you think I'm on the right track, let me know and I can try to create a Player node ;)

[Calinou]

Great work 🙂

Only question would be is if I can turn this into an officially supported GDExtension

I think the extension would need to be production-ready first, which means it should be able to work as a drop-in replacement for the existing built-in video playback system (other than reencoding your videos to the desired format).

[voylin]

I think the extension would need to be production-ready first, which means it should be able to work as a drop-in replacement for the existing built-in video playback system (other than reencoding your videos to the desired format).

If it were to use FFmpeg as a library, is this okay for an official GDExtension?