cocoa: layer backed view

[Akemi]

this is a WIP so there is still some debugging shit in the commits and the commit messages don't conform to the standard yet.

i am doing it this way since it potentially fixes 7 issues and i kinda want a central place where issues can be reported and fix confirmations can be posted. the issues in question that most likely will be fixed are:
#2217
#2392
#2873
#3739 - confirmed fixed
#3746 - confirmed fixed
#3944
#3978 - confirmed fixed (most likely)

known issues of this WIP branch:

• the layer is not in sync with the resizing on live-resize, so black edges can occur on the right and bottom. this makes the fullscreen animation look a bit choppy.
• if display aspect ratio and video aspect ratio are different the video will be stretched for the time of the animation from window to fullscreen and vise versa.

since there is a dislike for animations and the native fullscreen (#4014) i will also add a none native fs after everything has been worked out.

[AirPort]

I did a couple of tests, and generally it works very well (it seems to solve the flickering overlays issue, the fs animation looks really smooth and has a "native" feel) but I seem to have stumbled upon a remarkable regression playing this clip: https://a.pomf.cat/kdhdza.mp4
Current master, with display-resample or audio sync + opengl-early-flush=no, plays the clip smooth both in windowed and in fullscreen mode.
The cocoa_animation branch plays the clip smooth with player windowed, but in fullscreen playback is glaringly choppy. opengl-early-flush=no has no effect on it.
The "strange" thing is that, unlike the old performance issue, the player reports no dropped frames whatsoever.
I tested the issue with a "regular" 24 fps video file and it's not nearly as glaring, playback seems fine, only I think sometimes I noticed some little "jump" during pans.
I'll re-test tomorrow with the morning and a clearer mind, if you need anything or you have some specific test you need me to run please tell me.

(For the sake of thoroughness, I can play the clip smooth in fs with this branch using the dGPU or a barebones configuration (basically cutting out opengl-hq), yet the Iris Pro should be more than able to handle this stress, as it's shown by the fact it's capable to do it with the current master).

[ghost]

For testing display-sync, check the usual frame statistics like vsync-jitter and vsync-ratio.

[Akemi]

a performance regression isn't something i would have expect tbh. though since it replaces the backing storage with the 'layer', it could be that the layer itself is just slower since it can be animated via Core Animation. i was thinking of replacing the CALayer with a CAOpenGLLayer/NSOpenGLLayer, though the latter wants that all rendering is done in its drawInCGLContext/drawInOpenGLContext method which makes it really hard to use. theoretical it might be possible to get mpv to sync to it or to use a shared context, a frame buffer and render that frame buffer to the OpenglLayer's context. for the latter i have no clue how do it (yet).

[Akemi]

@AirPort i made a small test app with libmpv and a CAOpenGLLayer. so i was wondering if you can build and test it. if you built mpv with libmpv you just need to call make and then ./test file.mkv to play a file in the directory of that repository. does it also have the performance regression you described?

[AirPort]

Sure, as soon as I'll get back home I'll try it and report.
I had also collected some data in the past couple of days, but unfortunately I had no time yesterday to post it in an useful way.

[AirPort]

Unfortunately I have to say that the test program changes nothing over the previous results.

Yesterday though, I tried to collected some numerical data with the clip I linked before.
I usually set the player with video-sync=audio, and in that mode the total frame timings property values are ~06000-0900 on average, and occasionally 11000-15000 on peaks with both master and cocoa_animation branch.
The player doesn't report dropped frames in any case, but with cocoa_animation in fullscreen (or with a resized, big, window) playback is stuttery.
In video-sync=display-resample mode the vsync-ratio is stable at 1.000 and vsync-jitter is ~0.02-0.03 (with occasional peaks) with both branches, both windowed and in fullscreen. Again, master is smooth, cocoa_animation is stuttery enlarging the window or going fullscreen. No frames are dropped or mistimed, every now and then a frame get delayed.

So in the end it seems that, whatever performance regression is happening, mpv is actually unaware of it. Maybe there are some system tools that could help me shedding some light on what's happening behind the scenes?

[Akemi]

i first thought it might be related to the layer not 'syncing' properly as seen in one of the issues i mentioned. though i am pretty sure that my example app, which uses CAOpenGLLayer, properly syncs and since you still get that problem it is probably something else. which is kinda bad since now i have no idea what can be done about it.

i pushed something to my cocoa_animation branch, removing the kCGLPFAAllowOfflineRenderers and kCGLPFABackingStore attributes from the context creation. i doubt they cause the performance regression but can't hurt trying without them. if you could test it again it would be appreciated.

[tmkk]

On an external display attached to retina MBP, a video is rendered inside a quarter of a player window. After resizing the window (or switching to fullscreen playback) it renders properly. Playback in built-in retina display is fine.
Screenshot:

[Akemi]

very weird issue. anyway my recent commit should fix it.

[tmkk]

Getting better, but still not perfect. It switches to a proper rendering almost instantly, but getting a quarter rendering at the beginning of a playback.

Calling [s->nsgl_ctx update] once when vo_cocoa_swap_buffers is called for the first time (i.e. before the first CGLFlushDrawable call) completely solves the problem. It seems [s->nsgl_ctx update] in vo_cocoa_config_window doesn't take effect at all for some reason.

[Akemi]

It seems [s->nsgl_ctx update] in vo_cocoa_config_window doesn't take effect at all for some reason.

hence why i said it's weird and why i change it.

[Akemi]

unfortunately i can also reproduce some performance regressions. vsync jitter drastically increases from under 0.03 to 0.03-0.3 (very unstable) causing quite a lot mistimed and delayed frames. though visually it's really hard to spot so i wonder if the layer might mess with the statistics somehow.

[Akemi]

@AirPort would you mind testing this again? i made a few changes.

[AirPort]

Unlike before, as far as I can see now with the test app and the 60 fps clip the playback is perfectly smooth both in fs and resizing the window!

(Just to be sure and not lead you astray, since I have no clue about the libmpv API and I have no way to check: it does pick up the VO options from my config, right?)

[Akemi]

yeah i made sure it loads the config file from ~/.config/mpv. if you have stats.lua in your scripts folder you can toggle it visible via echo 'script-binding stats/display_stats_toggle' | socat - /tmp/mpvsocket in a separate terminal. though you need to install socat from brew for it. i also kinda forgot to mention, that test only works properly on a the main screen for now and assumes 60hz as initial value for the display refresh rate.

since it works for you and it also improved the performance for me drastically this seems to be the only way to go. we have to use CAOpenGLLayer somehow and also have to use a 'discouraged' way of updating that layer to make it work properly. the big problem here though is that opengl-cb/libmpv use a different different way of rendering, which is the opposite of how mpv internally works. so it's sadly non-trivial to use a CAOpenGLLayer and might not work at all.

[Akemi]

going to close this, since i too see some bad performance regressions and we need a completely different approach with a CAOpenGLLayer.