Mouse movement causes entire stream to stutter

[psyke83]

Host: Windows 10 x64 21H1, Ryzen 2700, Radeon RX 570, H264 software encoding, built against latest master as of d332f11
Client: Moonlight 3.1.4 on Ubuntu 21.10 x64 & Windows 10 x64

Issue: when moving the mouse on the client during movement on-screen, the entire stream (not just the mouse pointer) will stutter at what seems subjectively half speed - but the Moonlight encoder statistics doesn't show any framerate drop. The effect persists until the mouse cursor stops moving.

Steps to reproduce:

1. Launch a 2D game that features smooth horizontal scrolling without need for user input; e.g. Sonic Mania in the menu attract mode.
2. Without moving the mouse, observe the fluidity of the stream to get a baseline.
3. Start moving the mouse continuously and watch for any change to the subjective framerate/fluidity.

Notes:

• The h264 software and amdvce encoders all exhibit this issue.
• Hardware and software decoding via Moonlight makes no difference.
• Testing Moonlight on Linux via Wayland may not be recommended, as I notice it is generally much choppier than Xorg during mouse movement unless I manually patch mutter with the following merge request: https://gitlab.gnome.org/GNOME/mutter/-/merge_requests/1915. In order to rule out this issue being the cause, I tested only on Xorg as well as running the Windows client on the same client machine to rule out an OS issue.
• Hiding the cursor via Ctrl + Shift + N or using absolute mouse mode doesn't resolve the stutter.
• As a comparison, Steam's Remote Play doesn't exhibit any stutter on Windows or Xorg. Stutter does occur via Wayland, but it is completely solved with the mutter patch.
• Stutter while controling the camera via mouse aim in modern games is harder to notice, but is still visible in certain scenarios - e.g. opening an ingame (pseudo-)2D map and scrolling horizontally, you can see stutter if you move the mouse while scrolling via the keyboard.
• The mouse polling rate doesn't seem to affect the severity of stutter. I tried my Logitech G305 at 1000Hz, 500Hz and also tried the built-in touchpad which reports at ~80Hz, but the stutter effect seemed to be roughly equal in all cases.

Edit: It seems that the same stutter is present if I move the host's mouse while observing the client stream. Also, to be clear: no stutter is visible on the host screen during mouse movement when the host or client mouse is moving.

[psyke83]

It seems that when the mouse is being moved, the capture FPS is exceeding the proper client constraint.

Steps to reproduce:

• On host, run a game, making sure that is capped to 60fps via vsync.
• On the client, run Moonlight with FPS set to the "unsupported" 90 or 120fps setting.
• Observe the Moonlight encoder statistics.

Result:

• The same stutter issue is present.
• The encoder statistics will report 60fps if the mouse is not in motion, but now shoots up to 90fps during movement (even if FPS is configured to 120 in Moonlight).

So, my guess is that the encoder could be getting overwhelmed by excessive framerate even when Moonlight is set to 60fps, even if the statistics don't show the problem if not using the unsupported rates.

[psyke83]

It seems that the issue I'm experiencing only affects some games, and in certain cases, it depends on what is being rendered for the issue to trigger.

One obvious example is Red Dead Redemption 2 (Vulkan). The starting menu and actual ingame works fine, with the capture/render FPS always constrained to 60fps according to Moonlight's statistics. If you enter the pause menu or open the map, however, the Moonlight fps will only stay at 60fps when the mouse is not moved, but shoots up during movement, which also triggers the stuttering effect while the map scrolls. Note that Steam's FPS counter never goes above 60fps during these stuttering incidents.

I haven't yet found an open source game or application to highlight this problem, but I can at least point to the free "Dagon" visual novel on Steam. The loading screen and main menu runs at 60fps, but exceeds that when the mouse is moved, whereas ingame will stay locked to 60fps at all times. Unfortunately this is not a great example to show the stutter effect, as the main menu itself is static.

More examples:
Control: stutter whenever cursor is visible, ingame is fine
Cyberpunk 2077: not affected
FTL: not affected
Dying Light: not affected
Hollow Knight: start menu affected, ingame is fine (but mouse doesn't control movement)
Risk of Rain: not affected
Shadow of the Tomb Raider: stutter whenever cursor is visible (start/pause menu), ingame is fine.
South Park: The Fractured But Whole: stutter at all times cursor is visible (which includes ingame).
Witcher 3: not affected. I tried toggling "hardware cursor" in the options, but it made no difference.

For the affected games, the vsync constraint is somehow ignored when feeding sunshine frames if the cursor is visible, which causes the stutter effect (even when Moonlight is set to 60fps), but there is no problem when the cursor is hidden/in mouselook mode.

[TrueZhuangJia]

happening on both AMD & Nvidia GPU (tested by RX550, 1050Ti), when having a remote desktop session instead of gaming. Run Google Earth to travel between locations then move your mouse will easily reproduce this issue.