Video: Hardware Acceleration

[alexmercerind]

How is current video rendering?

Currently, whenever you show a video using Video widget from package:dart_vlc, it produces redundant load on your CPU.
It is because every frame is copied from GPU back to RAM through CPU for rendering inside Flutter i.e. no hardware-acceleration.
This load on CPU increases significantly with higher resolution or higher FPS videos.

This still works fine -ish with 720p videos (bearable CPU load).

What is hardware acceleration?

From Google:

Hardware acceleration is the use of computer hardware designed to perform specific functions more efficiently when compared to software running on a general-purpose central processing unit.

Playing a video should make use of dedicated hardware i.e. GPU (and not CPU). CPU is for general-purpose computation, not copying frames/buffers or rendering things efficiently. As discussed above, since every video frame is copied from GPU back to RAM through CPU for rendering inside Flutter, redundant CPU load is seen. This also increases the power usage (battery consumption).

Currently, 720p videos result in bearable CPU load & work fine -ish. Higher than that, high CPU usage.

NOTE: If you are showing the Video in a small part of the UI, you can limit the dimensions of each frame manually.
This will reduce CPU usage & give you finer control to tweak the performance.

Player player = Player(id: 0, videoDimensions: const VideoDimensions(640, 360));

Why is hardware acceleration not supported (yet)?

First of all, because this is way too complex to be part of someone's hobby. A funding is necessary here.

Both, due to Flutter & libVLC's limitations at that time.

• libVLC 3.x.x does not have any API to handle hardware accelerated video rendering in client code. libVLC 4.x.x will have an API to render into a client defined texture through OpenGL or Direct3D. However, it is still not possible to use it since shipment of dependencies on Linux will be a big challenge & I don't think consumers will receive the new VLC anytime soon (even after the release).
• Flutter recently added the new Direct3D texture interop capability for Windows. See my testing playground.
  This is hardware-accelerated & uses Direct3D. This was not present earlier.
• Flutter recently added texture support (both pixel-buffer & GL) for Linux.
  This is also hardware-accelerated. This was not present earlier.

When & how will be hardware accelerated video playback implemented?

Fortunately, package:dart_vlc & my work in this regard is now sponsored by Stream.

This is a concern for Windows & Linux currently.
Scope of work may be increased as required, however package:video_player already seems to have hardware acceleration.
Maybe I will add support to package:video_player itself at some point.

Rapidly ship in-app messaging with Stream's highly reliable chat infrastructure and feature-rich SDKs, including Flutter!

Try the Flutter Chat tutorial

I made a POC implementation (posted on my Twitter) showing hardware-accelerated video rendering inside Flutter.
It is using libmpv & ANGLE based Direct3D & open-gl interop.
Notice minimal CPU 2 % - 5% usage. Slight usage can still be seen due to logging/flushing etc. for debugging. It will be improved.
Same would have caused about 20-25% with package:dart_vlc (even more with higher resolution & FPS videos).

I have decided to use libmpv because it has API for rendering video output using open-gl.
Since, this package has everything specifically around VLC, it makes no sense to work on it here.

Thus, expect a new package really-really soon with hardware accelerated video playback. Other than just hardware-accelerated video embedding, it has things like metadata / tags reader, shifting support, pitch shifting, volume boost, dynamic playlists & native OS controls etc. out-of-the-box along-side basic features. Audio-only support is ready & used in Harmonoid: A Material Design-led music player to play & manage music library.
Most importantly: It is far smaller in size compared to package:dart_vlc, stable & even supports Windows 7.

Switching to libmpv does not mean that I do not like VLC or the awesome work that developers have done on it. It is still the best of best (& my preferred) media player, which I have installed on every device I own.
However, from the library / plugin standpoint, I'm not quite satisfied.

libmpv+flutter.mp4

Thanks!

---

You may subscribe to the issue for updates.

[alexmercerind]

package:media_kit is now public!