platform/windows: release frame early during mouse movement

[psyke83]

Resolves visual stutter during mouse movement in certain applications.

Proposed resolution for: #227

It seems that waiting for the frame to be released between consecutive snapshot functions calls introduces visual stutter (only in certain applications & during mouse movement - see above linked issue). This may not be the ideal way to implement the fix, so feedback is appreciated.

[psyke83]

I updated the patch to only release the frame early when mouse movement occurred. I'm still not confident this is the best way to fix the issue in the code, but perhaps someone can test and point out a better solution.

[AetherCollective]

I've been wondering about this issue. Any ideas when it might get merged?

[psyke83]

This PR dramatically improves the problem to the extent that I'm using it on my machine, but I'm not convinced this is the ideal way to fix the issue.

Without this PR, any mouse movement during capture of certain content would cause frame_info.AccumulatedFrames to increase to 2, meaning that a screen update already occurred since the capture was taken. The patch set greatly reduces the problem so that AccumulatedFrames remains at 0/1 (when mouse/screen updates occur, respectively), but with mouse movement I still see occasional stutters (i.e. AccumulatedFrames can still increase to 2).

Microsoft's documentation states:

For performance reasons, we recommend that you release the frame just before you call the IDXGIOutputDuplication::AcquireNextFrame method to acquire the next frame.

Sunshine already does this, but it holds the frame for the delay interval between frames (std::chrono::nanoseconds { 1s } / framerate, i.e. 16.6ms for 60fps). I'm not certain if Microsoft's guidance includes the assumption that you aren't continuously capturing without delay, then waiting the requisite interval before picking frames in a separate thread.

[resech]

I'm getting similar errors to what the failed appveyor build shows, are there any extra pre-requisites/commands that need to be run to build this?

[psyke83]

@resech

It looks like the mingw headers were updated recently, so this PR itself is not the cause. #296 should fix it.

[TheElixZammuto]

Thank you for the heads up @psyke83, I was also having the same problem that I couldn't reproduce on my local build but only on GitHub actions

[psyke83]

After some more research, I found this: https://docs.microsoft.com/en-us/windows/win32/direct3ddxgi/desktop-dup-api

When AccumulatedFrames is >1, AcquireNextFrame does not return a desktop image, but instead provides update regions (dirty/move rectangles). Since sunshine doesn't yet support these updates, it considers these frames as empty, which results in visible stutters.

This PR was only helping to mask the issue by giving AcquireNextFrame less opportunity to construct dirty/move regions by placing ReleaseFrame() in a less optimal position. As I suspected, this PR is not the proper solution, so I'll convert this a draft until I've had time to investigate the real fix.

[psyke83]

Unfortunately, processing dirty rects* doesn't seem to affect slight stutter effect noticeable when AccumulatedFrames is >1. I'm struggling to think of another way to eliminate the remaining stutters (which I'll admit are quite rare, and mostly only occur when dealing with the desktop or a game running in windowed mode). I'll leave this in draft status, but I'm not sure what else I can do to alleviate these remaining rare stutters.

*If anyone is interested, here's a test branch: master...psyke83:dirty_rects
I'm not certain if this realistically improves performance, as each frame received by the encoder is still the complete image.

[hksdpc255]

I recommend to always send frames to the client even though the screen is not changed. Only for Windows.

As far as I know, a new screen capture mechanism was introduced in 7ddf8bb, while the old one works fine on Windows.

See: #253

It seems that current screen capture method is introduced in this commit:

7ddf8bb

[psyke83]

@hksdpc250

I've tested older builds without the capture callback change, but it has its own issues compared to master. The positive difference is that there isn't any lag in simple desktop applications such as Notepad (which is closer related to your bug report than this, I imagine). Unfortunately, performance for 3D applications, especially in windowed mode, is very choppy. Calling ReleaseFrame() doesn't help as much with that problem compared to latest master with the callback changes.