GUI embedding for Wayland in libmpv

[MaxPerl]

Hello everybody,
I saw this very old issue report and wanted to ask, whether it would be possible to make wayland-embedding in libmpv possible, too? This would be great!!! The vlc guys think about this, too (see here ;-) ) a function as mpv_set_wl_surface or similar would be great. Rendering through OpenGL doesn't work for me and a simpler alternative in Wayland would be very important!
Thanks in advance,
Max

[Dudemanguy]

It's not really possible. At least, the --wid option currently does not work on wayland and probably will never work (the wayland devs don't like embedding to arbitrary windows). Your only real choice is probably to use the render api in libmpv.

[MaxPerl]

Thank you very much for your answer. I am trying the render api (see my issue report here). Nevertheless would a simpler way wonderful. But I see that this would a long-term goal (the milestone for the vlc devs is Dec 31, 2022 ;-) ).

[polarathene]

For reference/tracking, two projects (Qt based) that embed MPV with devs stating that not much can be done for broken embedding on Wayland:

kokoko3k/xt7-player-mpv#32 (comment)

Unfortunately xt7 will never support wayland, because mpv does not support the way xt7 embeds his window under the main app;

smplayer-dev/smplayer#369 (comment)

Install smplayer using a snap or flatpak package. Select "shared memory" as video output in preferences -> general -> video.
This is the best integration I could make.

---

Not sure if the situation has changed in 2022, I have noticed some work going on with MPV mentioning Wayland and not to use Shared Memory / SHM by @Dudemanguy

and displays them using software shared memory renderer

is not really acceptable. One could hook hwdec into vo_wlshm (please don't do this though) and it would accomplish the same thing.
To efficiently share the buffer between mpv and the compositor, you need to use the linux-dmabuf protocol.
shm should not be used at all.

---

I noticed there was ongoing work with Vulkan and Vulkan Video for GPU-Next VO, not sure if that will change anything? Years ago someone requested support in the embedding API for getting Vulkan output, and I've seen maintainers here talk about using DMA-BUF + Wayland protocols (not sure if that helps at all with the embedded use for frontend players).

Or would PipeWire make sense at all if nothing else is useful? MPV could presumably share the video output through that to another app to display, with the app still able to control the playback via UI controls it offers interacting with libmpv? Or is that no better than the suggestions to use SHM (SMPlayer dev advice via snap/flatpak/appimage) or x11egl (xt7-player chooses to use XWayland I think)?

[Dudemanguy]

I might have been wrong earlier. Using xdg-foreign, exporting the toplevel, and then the client importing the toplevel and setting it as its parent may suffice? I'm not 100% clear this actually would work for arbitrary window embedding though like X11. Might be worth a try later.

Otherwise, the situation has not changed at all. Also I would really not recommend using SHM at all. You're better off with x11egl via xwayland.

Edit: Actually, I don't think you can place the toplevel anywhere with xdg-foreign so I don't really think anything would change at all?

[polarathene]

Edit: Actually, I don't think you can place the toplevel anywhere with xdg-foreign so I don't really think anything would change at all?

I am just an end-user so not able to contribute much to the discussion at a technical level. Thanks for chiming in though! :)

Is the embedding use-case for frontend interfaces like SMPlayer different from getting the video output as a stream from PipeWire and only using libmpv to control playback?

I assume screen recording software is doing something like this (and apparently PipeWire was helpful getting support working well on Wayland for that type of software). I'm not familiar with the lower level implementation, but assume libmpv would need to provide these frontend clients with a way to get that video stream through PipeWire?

I have heard of software using DMA-BUF sharing to avoid extra copies for this type of thing, which I think PipeWire manages.

---

Otherwise, the situation has not changed at all.

Do you know if there are any plans to tackle this in future? Or is it not a concern for MPV? (I am not entirely sure, but I think the project tries to provide a common base that other players can build around, in addition to it's own player/interface)

Also I would really not recommend using SHM at all. You're better off with x11egl via xwayland.

The advice is appreciated, but for the downstream projects such as SMPlayer, it might be helpful if you can elaborate a bit more what is wrong with the SHM option, otherwise they may prefer not to switch their implementation if it seems fine to them.

• Is it just a performance issue, or other limitations/concerns? No HW decode?
• Is x11egl via xwayland likewise a degraded experience vs alternatives usable on a X11 desktop instead of a Wayland one?

[Dudemanguy]

Is the embedding use-case for frontend interfaces like SMPlayer different from getting the video output as a stream from PipeWire and only using libmpv to control playback?

It's extremely different and not really related to PipeWire at all. Basically what other applications would like is direct control over the mpv window in whatever platform they are using (x11, macos, windows, etc.). The --wid option provides a way to do this (the details differ depending on the platform). The problem is that on Wayland, I don't think this is actually possible. The closest thing we have is the xdg-foreign protocol, but it looks pretty limited. Basically a different wayland client can get some access to a toplevel from another client (AKA mpv), and then all I think you can do is choose whether to stack it or not. You don't actually get any type of more fine-tuned positioning control or anything so it doesn't seem useful for our purposes. Maybe that's why I said it was impossible in my first reply; been a while since I looked at this.

Do you know if there are any plans to tackle this in future? Or is it not a concern for MPV?

Maybe someone could extend the xdg-foreign protocols to do these things. There's not really anything we can do about it though. It mostly needs upstream support from wayland somewhere first. For now, the only option for embedding on wayland is through the render api. Admittedly, not exactly ideal since it's still opengl-only, but it'll work.

The advice is appreciated, but for the downstream projects such as SMPlayer, it might be helpful if you can elaborate a bit more what is wrong with the SHM option, otherwise they may prefer not to switch their implementation if it seems fine to them.

To be honest, I have no clue how they got SHM to work. Maybe they worked around wayland or something. Anyways, shm is a software rendering VO. It's nice as a very "minimal" wayland VO, but in general it's not something to be used. It has several limitations, including no hardware decoding, but it's also not a hardware accelerated output at all to begin with. You'll have worse quality and much less available options as a result.

Is x11egl via xwayland likewise a degraded experience vs alternatives usable on a X11 desktop instead of a Wayland one?

It kind of depends. Using the x11 backends on wayland work just fine in general and will be hardware accelerated and all that jazz (except maybe on nvidia, dunno if they fixed that or not yet). There's some details that would make using xwayland worse than native wayland on wayland or native x11 on x11 though. If you're doing any kind of hidpi scaling (i.e. anything other than scale 1), it'll be strictly worse. This is because the compositor will scale the mpv surface server-side which, as an xwayland client, there's nothing mpv can do about of course. That would potentially degrade picture quality since mpv's scaling algorithms will, most likely, be a lot more sophisticated than whatever the compositor is using. Although, if you are doing non-integer scaling in wayland (i.e. fractional scaling) the current behavior is pretty bad in native wayland clients anyway so I guess maybe it's not that much of difference. If you care about presentation timings, native X11 on X11 has presentation as well as native wayland on wayland (well minus nvidia drivers but that's just a common thing). xwayland on wayland does not. So technically xwayland is a bit worse on that front as well although that's relatively minimal in the big picture.

[Dudemanguy]

Oh so it turns out there was an issue about this wayland-protocols a while ago. The tl;dr is: don't expect this anytime soon.

[polarathene]

For now, the only option for embedding on wayland is through the render api. Admittedly, not exactly ideal since it's still opengl-only, but it'll work.

Is the render API an alternative to embedding where there is no MPV window to control by a client app, which can instead have access to texture/stream (or equivalent) to render/display?

I have heard with Vulkan memory management is different, and it sounds like some sort of buffer sharing can be used between apps (not sure if Wayland negatively impacts that though).

I imagine DMA-BUF and PipeWire also fit into this type of sharing, and is what I usually read as solutions with Wayland for sharing displays (QEMU-KVM has a new host-side option with D-BUS for exchanging a DMA-BUF fd I think for clients to show VM guest displays, Firefox is using DMA-BUF for video and WebGL AFAIK, and screen sharing apps PipeWire).

Is any of that an approach that might be suitable for MPV to support in future if it better supports clients on Wayland like SMPlayer? (I think Gnome has an MPV based GTK player app, but I've not looked into how they approached it or if it has similar issues with Wayland)

I think that would be a "headless" approach without an MPV window to control, if that's possible? MPV in that context may need more information (eg: dimensions to render for?)

---

shm is a software rendering VO. It's nice as a very "minimal" wayland VO, but in general it's not something to be used. It has several limitations, including no hardware decoding, but it's also not a hardware accelerated output at all to begin with. You'll have worse quality and much less available options as a result.

Using the x11 backends on wayland work just fine in general and will be hardware accelerated and all that jazz (except maybe on nvidia, dunno if they fixed that or not yet). There's some details that would make using xwayland worse than native wayland on wayland or native x11 on x11 though. If you're doing any kind of hidpi scaling (i.e. anything other than scale 1), it'll be strictly worse.

This is because the compositor will scale the mpv surface server-side which, as an xwayland client, there's nothing mpv can do about of course. That would potentially degrade picture quality since mpv's scaling algorithms will, most likely, be a lot more sophisticated than whatever the compositor is using.

Awesome, that's really helpful, thanks for clarifying!

---

Oh so it turns out there was an issue about this wayland-protocols a while ago. The tl;dr is: don't expect this anytime soon.

If that's the only/proper way to go forward with better Wayland support that's all good. I was just after a clearer picture of where the issue was, and how it could be resolved 👍

Thanks for taking the time to chime in, really appreciated! 😀

[Dudemanguy]

Is the render API an alternative to embedding where there is no MPV window to control by a client app, which can instead have access to texture/stream (or equivalent) to render/display?

That's right. It lets an application use libmpv directly and then they can do pretty much whatever they want. Of course, for applications that are already built around window embedding, it's not necessarily feasible for them to incorporate such a scheme into whatever their existing structure happens to be.

I imagine DMA-BUF and PipeWire also fit into this type of sharing

Again, this isn't really related at all. PipeWire works around wayland by handling video streams via dbus APIs. That's great, but this isn't what GUI embedded applications actually want. They want direct control over the window on a wayland/x11/whatever level. Like in X11, mpv can just attach itself to another window and then the other application can do whatever it wants from there. They need that kind of level of control. Right now, the only way to accomplish that on wayland is with the render API.

I think Gnome has an MPV based GTK player app, but I've not looked into how they approached it or if it has similar issues with Wayland

Celluloid uses the render API the last time I checked.

If that's the only/proper way to go forward with better Wayland support that's all good. I was just after a clearer picture of where the issue was, and how it could be resolved

This use case will likely never be supported by Wayland so I wouldn't count on it. Likely, the best approach would be to use the render api and improve any deficiencies with it (one obvious one would be the lack of vulkan support; not impossible or anything, just no one has done it). Of course if an application is already structured around --wid (many are and they work perfectly fine), this isn't really the greatest thing to hear but realistically it's the best way for this to work. The application can then handle the wayland subsurfaces and everything internally and it should be fine.