Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unified DXVK/VKD3D lag timebomb in most games if compiled from git #4436

Closed
SheMelody opened this issue Nov 8, 2024 · 91 comments
Closed

Unified DXVK/VKD3D lag timebomb in most games if compiled from git #4436

SheMelody opened this issue Nov 8, 2024 · 91 comments

Comments

@SheMelody
Copy link

SheMelody commented Nov 8, 2024

There's a literal "lag timebomb" when pulling the latest git versions of DXVK and VKD3D and compiling them. These lag timebombs are unified, meaning that they happen in both DXVK git and VKD3D git, and they trigger after roughly ~25 to ~40 minutes of gameplay and generally only trigger if the user is providing input to the game. This problem exists since roughly 2 months and I felt like it would be good to report it here. Since the problem is unified (happens on both DXVK and VKD3D) I actually didn't know where to report it.

Affected games mostly use Unreal Engine. Insomniac Engine games, for example, didn't seem to be affected.

Using a "Release" version of DXVK + a "Release" version of VKD3D seems to solve the problem.

More specifically, the game is smooth before the "timebomb" triggers, then consistent and repeated heavy stutters begin to happen. If needed, I can record a video.

EDIT: We have found out that, when this triggers, it only happens when moving the mouse.

Software information

Any game that uses DXVK or VKD3D pulled from git and compiled, this problem is unified and common in both DXVK and VKD3D

System information

  • GPU: Most likely any GPU, both NVIDIA and AMD (tested: 4070 SUPER, 4080 SUPER, 7900XTX)
  • Driver: Most likely any driver
  • Wine version: Tried multiple Proton versions, including GE and LFX2
  • DXVK version: straight from git

Apitrace file(s)

  • Can provide if needed, but this problem can easily be reproduced on any machine

Log files

  • Again I can provide this if it's really necessary, but this can be easily reproduced on any machine
@SheMelody
Copy link
Author

We have found out that, when this triggers, it only happens when moving the mouse, I'll be posting a video soon.

@doitsujin
Copy link
Owner

doitsujin commented Nov 8, 2024

DXVK does not interact with mouse input in any way, I strongly doubt it's even remotely possible that this is our bug.

Also haven't seen anything like this on my machine even in extended play sessions. As for Unreal Engine titles, I ran Hellblade and Days Gone for ~an hour not too long ago to test some unrelated changes.

@SheMelody
Copy link
Author

DXVK does not interact with mouse input in any way, I strongly doubt it's even remotely possible that this is our bug.

Also haven't seen anything like this on my machine even in extended play sessions. As for Unreal Engine titles, I ran Hellblade and Days Gone for ~an hour not too long ago to test some unrelated changes.

I understand that they don't interact with mouse input, but when I run a game that natively uses Vulkan (ie. Quake 2 Remastered, Saints Row) they're not affected by this problem.

Switching to any of your releases paired with a vkd3d release completely fixes this mouse problem. I'm not any more expert than you, but the only thing dxvk and vkd3d have in common is dxgi.

I don't know if I'm being helpful but I can post a video on YouTube or logs.

@doitsujin
Copy link
Owner

doitsujin commented Nov 8, 2024

Well what I'm saying is that this functionally doesn't make any sense and I can't reproduce it.

First step would probably be to provide detailed info about your build environment, including compiler version, MinGW version if applicable, and the exact command lines you're using to build the whole thing.

Another question would be what kind of environment you're running on as well, any sort of input issue points more towards wine/compositor issues than us.

@SheMelody
Copy link
Author

SheMelody commented Nov 8, 2024

Well what I'm saying is that this functionally doesn't make any sense and I can't reproduce it.

First step would probably be to provide detailed info about your build environment, including compiler version, MinGW version if applicable, and the exact command lines you're using to build the whole thing.

Another question would be what kind of environment you're running on as well, any sort of input issue points more towards wine/compositor issues than us.

I'm using fully updated Arch Linux, MinGW 13.1.0-1.

I tried with X11, X11 + gamescope, Wayland and Wayland + gamescope. Same problem happens after roughly 25 to 40 minutes of gameplay.

There's an intermittent heavy stutter that runs at its own pace, but only when playing with keyboard and mouse apparently, and it gets worse if you move your mouse once the "timebomb" triggers.

I'm using your package-release.sh script to build dxvk, using vkd3d's package-release.sh script to build vkd3d as well. Nothing goes wrong while compiling. Switching to "old" versions of vkd3d + dxvk completely gets rid of the issue.

Running a game that natively uses Vulkan or OpenGL never triggers this problem (under the same proton prefix and under the same circumstances of course).

I'm going to upload a video of the problem on YouTube (recorded by a friend of mine) in order to showcase the problem.

@SheMelody
Copy link
Author

Video uploaded, manually switch to 1080p if it starts at 360p. This is a footage after roughly 30 minutes of playtime. The lag "timebomb" shows up at 0:44 timestamp:

https://youtu.be/PNa_DGexFAs

@Blisto91
Copy link
Contributor

Blisto91 commented Nov 8, 2024

Does this also happen with the builds that ship with Proton Bleeding Edge or downloaded from GitHub CI or only with self compiled?

Also are you able to bisect which commit it started with?

@doitsujin
Copy link
Owner

doitsujin commented Nov 8, 2024

I'm using fully updated Arch Linux, MinGW 13.1.0-1.

This is precisely what the release builds are from. MinGW stuff hasn't been updated in a year or so, so the past couple of stable releases were compiled using exactly this in release mode.

Switching to "old" versions of vkd3d + dxvk completely gets rid of the issue.

Does "old versions" include self-compiled builds of stable releases?

I'm not running anything special here, just plain old KDE Wayland with (generally) no Steam overlay and (generally) no Gamescope, and RADV on a 6900XT. If this is an issue that supposedly affects any (UE4) game on any setup then what exactly is it that makes my system the special snowflake that this doesn't happen on?

Sorry for the rather annoyed tone but we have a release planned soon, so if this issue actually does affect a majority of users then I need a reliable way to reproduce it in order to actually do somehting about it.

@SheMelody
Copy link
Author

I'm using fully updated Arch Linux, MinGW 13.1.0-1.

This is precisely what the release builds are from. MinGW stuff hasn't been updated in a year or so, so the past couple of stable releases were compiled using exactly this in release mode.

Switching to "old" versions of vkd3d + dxvk completely gets rid of the issue.

Does "old versions" include self-compiled builds of stable releases?

I'm not running anything special here, just plain old KDE Wayland with (generally) no Steam overlay and (generally) no Gamescope, and RADV on a 6900XT. If this is an issue that supposedly affects any (UE4) game on any setup then what exactly is it that makes my system the special snowflake that this doesn't happen on?

Yes, to also answer @Blisto91 the "old versions" included my own self-compiled. I have self-compiled versions of both dxvk and vkd3d saved, but I don't know where to start, honestly, maybe we can guess something:

  • 21st of July 2024
  • 23rd of July 2024
  • 26th of July 2024
  • 27th of July 2024
  • 30th of July 2024
  • 10th of August 2024
  • 16th of August 2024
  • 22nd of August 2024
  • 30th of August 2024
  • 9th of September 2024
  • 10th of September 2024
  • 19th of September 2024
  • 21st of September 2024
  • 25th of September 2024
  • 1st of October 2024
  • 6th of October 2024
  • 12th of October 2024
  • 16th of October 2024
  • 18th of October 2024
  • 6th of November 2024
  • 7th of November 2024
  • Today

@SheMelody
Copy link
Author

For the sake of having a complete background, I'm actually going to test this whole thing without using ecores on intel and by using only one ccx on amd. It's probably unrelated and nonsensical but it doesn't harm me to test, my boyfriend is also testing with me. We've been trying to troubleshoot this for days to no avail.

@Etaash-mathamsetty
Copy link
Contributor

Etaash-mathamsetty commented Nov 8, 2024

could you send a log with DXVK_LOG_LEVEL=debug

also could u try windowed mode

@SheMelody
Copy link
Author

SheMelody commented Nov 8, 2024

could you send a log with DXVK_LOG_LEVEL=debug

also could u try windowed mode

Yes, I'll attempt both things after testing the ecores/ccx thingy.

Edit: I'm sorry because these tests take a very long time, it's 25 to 40 minutes per-test. Yet, reproducibility should be easy (not timewise of course) as long as you're playing with a mouse and a keyboard on newly compiled dxvk + vkd3d.

@doitsujin
Copy link
Owner

@Etaash-mathamsetty
Copy link
Contributor

could you send a log with DXVK_LOG_LEVEL=debug
also could u try windowed mode

Yes, I'll attempt both things after testing the ecores/ccx thingy.

Edit: I'm sorry because these tests take a very long time, it's 25 to 40 minutes per-test. Yet, reproducibility should be easy (not timewise of course) as long as you're playing with a mouse and a keyboard on newly compiled dxvk + vkd3d.

funny story, I can't reproduce it either

@SheMelody
Copy link
Author

Can you try https://github.com/doitsujin/dxvk/tree/no-occlusion?

I'm building it while my boyfriend is testing without ecores.

@SheMelody
Copy link
Author

could you send a log with DXVK_LOG_LEVEL=debug
also could u try windowed mode

Yes, I'll attempt both things after testing the ecores/ccx thingy.
Edit: I'm sorry because these tests take a very long time, it's 25 to 40 minutes per-test. Yet, reproducibility should be easy (not timewise of course) as long as you're playing with a mouse and a keyboard on newly compiled dxvk + vkd3d.

funny story, I can't reproduce it either

Is it actually git cloned and compiled or just picked from releases? If nobody can reproduce this here, it could actually be related to a change that affects ecores (to-be-confirmed since my boyfriend is still testing, he's playing since 31 minutes without the problem triggering as of now).

@SheMelody
Copy link
Author

For the sake of having a complete background, I'm actually going to test this whole thing without using ecores on intel and by using only one ccx on amd. It's probably unrelated and nonsensical but it doesn't harm me to test, my boyfriend is also testing with me. We've been trying to troubleshoot this for days to no avail.

Does not seem to be the problem. Will try no-occlusion straight away @doitsujin

@SheMelody
Copy link
Author

Can you try https://github.com/doitsujin/dxvk/tree/no-occlusion?

Timebomb triggered at 33 minutes with this fork. Some walls and corners were invisible. My boyfriend will test this again with debug logging, I'm going to sleep soon. Make a list of more things that I should try and I'll do it right after waking up (I don't have work so I'm free the whole day).

@doitsujin
Copy link
Owner

Well, that rules out the only DXGI change since 2.4.1, so....

@SheMelody
Copy link
Author

Well, that rules out the only DXGI change since 2.4.1, so....

Could it be a shared change between DXVK and VKD3D? Because we don't experience that problem with neither Proton GE and Experimental, yet we used to not have that problem at all and I always compiled the latest git releases for both of us.

I strictly believe that, at some point in time, something changed, and it affected both dxvk and vkd3d, and specifically "mouse handling" (probably not the right term) somehow.

It's weird that anything that runs on native API (Vulkan / OpenGL) does not suffer from this issue on any Proton prefix, yet the moment we try using git compiled dxvk + vkd3d this starts triggering.

It's interesting that games that run, for example, on Insomniac Engine and its forks, don't even suffer from this issue, while other games (notably Unreal Engine) have this timebomb as long as they use DXVK or VKD3D and as long as you play them with a mouse and keyboard.

My boyfriend will provide debug logging until the problem starts happening, and if you can 100% conclude that this is certainly not a DXVK + VKD3D issue, I'll have to look somewhere else. Most worrying scenario would be this problem affecting the majority of people but they have no idea. We really need more people testing this.

@Blisto91
Copy link
Contributor

Blisto91 commented Nov 8, 2024

Been playing Hogwarts Legacy for around 110 mins now but haven't been able to reproduce.
Proton Experimental Bleeding Edge (the beta branch), Ryzen 7950x, RTX 4080 (driver is 565.57.01), KDE X11, Steam overlay turned on and even obs open in the background.

@SheMelody
Copy link
Author

Been playing Hogwarts Legacy for around 110 mins now but haven't been able to reproduce. Proton Experimental Bleeding Edge (the beta branch), Ryzen 7950x, RTX 4080 (driver is 565.57.01), KDE X11, Steam overlay turned on and even obs open in the background.

That's not exactly the steps, I meant manually compiling dxvk and vkd3d straight away and putting them in a compatibility tool.

@Blisto91
Copy link
Contributor

Blisto91 commented Nov 8, 2024

I asked above if the issue shows with the builds included in Proton Bleeding Edge or from the CI.

Edit: Could you share the ones you have compiled?

@SheMelody
Copy link
Author

I asked above if the issue shows with the builds included in Proton Bleeding Edge or from the CI.

This will take a long time to test, I was mainly testing git compiled dxvk and vkd3d. Bleeding Edge breaks too much stuff, I'd rather live with the lag timebomb.

@Blisto91
Copy link
Contributor

Blisto91 commented Nov 8, 2024

I would like to try the ones you have compiled yourself to test with. Just package them up and attach here

@SheMelody
Copy link
Author

I would like to try the ones you have compiled yourself to test with. Just package them up and attach here

All.tar.gz

@SheMelody
Copy link
Author

I've got to sleep now, I'll keep myself updated when I wake up. File attached in the previous message.

@Blisto91
Copy link
Contributor

Blisto91 commented Nov 9, 2024

Played Hogwarts Legacy for 90 minutes more, this time with the Bleeding Edge DLLs replaced with vkd3d-proton, dxvk and dxvk-nvapi from your archive. Still nothing sadly

@Etaash-mathamsetty
Copy link
Contributor

Etaash-mathamsetty commented Nov 9, 2024

since you can build your own dxvk and know that the issue has been happening for 2 months, could you bisect it?

@SheMelody
Copy link
Author

SheMelody commented Nov 9, 2024

Played Hogwarts Legacy for 90 minutes more, this time with the Bleeding Edge DLLs replaced with vkd3d-proton, dxvk and dxvk-nvapi from your archive. Still nothing sadly

My boyfriend actually tried experimental bleeding edge overnight (without my DLLs) and the timebomb triggers there as well. Are you sure you were actually playing with a keyboard and a mouse? The problem only seems to happen when using m+k. Regular Proton Experimental is completely fine and unaffected by the way. And again: native Vulkan/OpenGL games running under any of these compatibility tools do not expose the problem, so it's definitely a dxvk+vkd3d thing.

since you can build your own dxvk and know that the issue has been happening for 2 months, could you bisect it?

See my message above where I list my older compiled dxvk+vkd3d combos, I don't know where to start. If you have an idea, maybe a date, I can start from there and test.

@SheMelody
Copy link
Author

What if you also manually disable steam input for the game? Maybe it also (intentionally or not?) loads the overlay stuff even if you disable it.

Steam Input was already turned off because that game is an aim trainer, so I turned it off to not add useless layers on top of the game.

@SheMelody
Copy link
Author

Yes, but please open a new issue about it referencing this one over at https://github.com/ValveSoftware/steam-for-linux/issues

I probably won't do it until tomorrow because I'm strained by all these days of troubleshooting and I just don't want to deal with more log grabbing and stuff. You can feel free to post an issue, or wait until I do it, it's the same to me.

@SheMelody
Copy link
Author

I'm closing this specific issue because it's caused by Steam's LD_PRELOADs, thanks everybody who helped me find the culprit.

I will probably open an issue at https://github.com/ValveSoftware/steam-for-linux/issues tomorrow, but you can feel free to do so as well.

@netborg-afps
Copy link

Actually, I just played overwatch 2 and did run into the same issue.

@SheMelody
Copy link
Author

SheMelody commented Nov 10, 2024

Actually, I just played overwatch 2 and did run into the same issue.

Were you able to fix it with the LD_PRELOAD method?

LD_PRELOAD="" %COMMAND%

If you need mangohud:

LD_PRELOAD="" VK_LOADER_LAYERS_ENABLE=VK_LAYER_MANGOHUD_overlay_x86_64 %COMMAND%

If you also need gamemode for some reason:

LD_PRELOAD="/usr/lib/libgamemodeauto.so.0" VK_LOADER_LAYERS_ENABLE=VK_LAYER_MANGOHUD_overlay_x86_64 gamemoderun %COMMAND%

(you still need gamemoderun for gamemode to actually trigger)

@netborg-afps
Copy link

Haven't tried yet, but I'll probably play again today, have already setup LD_PRELOAD="" in the launch parameters.
I wonder how long this bug has been around, probably not for too long, since it's the first time I experienced it, but haven't been playing for a month or so previously.

Thanks for being persistent in trying to solve this issue, as it's definitely a rough one.

@SheMelody
Copy link
Author

SheMelody commented Nov 10, 2024

Haven't tried yet, but I'll probably play again today, have already setup LD_PRELOAD="" in the launch parameters. I wonder how long this bug has been around, probably not for too long, since it's the first time I experienced it, but haven't been playing for a month or so previously.

Thanks for being persistent in trying to solve this issue, as it's definitely a rough one.

For me, it definitely started around the same time Valve implemented the "Record your game" thingy on Steam. And yeah I would say I'm quite persistent with things when it's not my fault, probably due to the way my work environment is (where colleagues accuse each others of having broken some code all the time), kinda reflects to other things I do.

@me01205
Copy link

me01205 commented Nov 10, 2024

Ran into the same issue about 3 days ago as well - Thanks for the LD_PRELOAD fix!

@Plagman
Copy link

Plagman commented Nov 10, 2024

When you say your overlay is completely disabled, how do you mean? Have you taken any steps for that other than using the UI option in the Steam client to disable the overlay?

@SheMelody
Copy link
Author

When you say your overlay is completely disabled, how do you mean? Have you taken any steps for that other than using the UI option in the Steam client to disable the overlay?

Completely disabled through Steam's UI.

@tfredett
Copy link

Thank you for this thread @SheMelody. I thought I was going bonkers for the last few weeks looking for whatever it may have been. As I experienced this on a number of different games but not all games.

I appreciate the effort you went through on this.

@Keenky
Copy link

Keenky commented Nov 15, 2024

Can confirm I have also been experiencing the same issue, but only since upgrading to Fedora 41. Couldn't find anything online and was thinking I was going crazy. Thanks for making the thread.

@ammuench
Copy link

Having the same issue, stuttering occurred in Saitsfactory. Thought it was gamescope but turning gamescope off didn't resolve it. Happened in DX12 and Vulkan mode. Disabled steam overlay. Switched from Proton Experimental to GE-Proton-9-16

It initially seems to have been resolved by adding LD_PRELOAD="" to my launch commands.

Went stuttering in Satisfactory every 30-40m on mouse input that required a reboot of the game, to playing for almost an hour and half last session after adding the command without issue (wasn't able to test further).

@jcsstelar
Copy link

Thank you so much guys this issue was driving me crazy!
LD_PRELOAD=""

@Moudoux
Copy link

Moudoux commented Nov 16, 2024

Setting LD_PRELOAD="" seems to introduce input lag for me in Borderlands 2

@mhmarf
Copy link

mhmarf commented Nov 17, 2024

Alan Wake 2 too had been showing some weird mouse lags recently (past month) on both Nobara 40 and Mint 22. It happens only if at least 15 minutes has passed. It simply will not happen sooner. I noticed this issue here, and tried to run the game on Lutris to see if lags go away and it was definitely fixed. I did not use LD_PRELOAD workaround for neither Steam nor Lutris.

I never compiled anything though... playing with default Proton 9/Experimental versions on both Steam and Lutris. I shall now try other games on Steam to see if they too have this mouse lag bug after 30 mins (tried Remnant 2 yesterday but I always rage quit it before 10 mins passed, despite trying hard not to. Now going to try Empire of the Ants and possibly Hellblade 2).

@mhmarf
Copy link

mhmarf commented Nov 17, 2024

Ah! And god of war Ragnarok also has this problem!!
Empire of the Ants also has this problem.

@OneWiseKnight
Copy link

OneWiseKnight commented Nov 26, 2024

I had this issue with Guild Wars 2. I want to also add what temporary fixed the time bomb was using the release version of DXVK. However this was my experience 2 weeks ago, currently experiencing mouse lag immediately in GW2. The fix for the 40 min lag time bomb was to use a release version of DXVK, however the immediate cursor lag I'm experiencing now is unrelated to DXVK. The current lag I am experiencing is worse with outside Value tree's of Proton. I've tested Experimental and GE Proton versions which have a 5-10 fps drop moving the cursor, however Proton or Wine compiled without value tree's shows a bigger fps drop moving the cursor. Again want to state this was only for Guild Wars 2, and not other games.

@marcelo-j-B
Copy link

I had relief knowing this issue was made clear when noticed in CS2 massive lag spikes after playing 30 min , warframe too when doing nothing but not in Space Engineers (maybe because crashes before experiencing this lag).
Testing currently
LD_PRELOAD="/usr/lib/libgamemodeauto.so.0" VK_LOADER_LAYERS_ENABLE=VK_LAYER_MANGOHUD_overlay_x86_64 gamemoderun %COMMAND%

@nexitem
Copy link

nexitem commented Dec 9, 2024

I had relief knowing this issue was made clear when noticed in CS2 massive lag spikes after playing 30 min , warframe too when doing nothing but not in Space Engineers (maybe because crashes before experiencing this lag). Testing currently LD_PRELOAD="/usr/lib/libgamemodeauto.so.0" VK_LOADER_LAYERS_ENABLE=VK_LAYER_MANGOHUD_overlay_x86_64 gamemoderun %COMMAND%

Did you mean Counter Strike 2 or Cities Skylines 2?

I seem to have a similar issue that after around 40-60 minutes of gameplay suddenly the game starts stuttering. Not sure if its this but Im gonna try LD_PRELOAD="" as well.

EDIT: Actually seems to have helped... no more stutters.

AMD Radeon 7900 XTX
AMD Ryzen 7950X3D
32GB DDR5 Ram

OS:
Archbased (CachyOS) Kernel 12.2
Plasma KDE 6.2
Wayland

@pyrotiger
Copy link

Having same issue with Star Trek Online

@netborg-afps
Copy link

Also happens in Path of Exile 2 with its Vulkan renderer, which would mean this is unrelated to dxvk?

@J-eremy
Copy link

J-eremy commented Dec 11, 2024

I am seeing this also on Manjaro KDE running Fallout76 using GE-Proton9-20. The game runs perfectly smooth for about exactly 30 minutes or so then the jitter line in mangohud just goes crazy and the only fix is to restart the game.

Currently using this launch parameter:
gamemoderun mangohud %command%

image

I will try some of the above fixes and report back.

@J-eremy
Copy link

J-eremy commented Dec 11, 2024

I can report that after over an hour it has not returned. The only thing added to the run command was LD_PRELOAD="" for a total command of:
LD_PRELOAD="" gamemoderun mangohud %command%

Mangohud works and so does gamemoderun without anything additional.

@ThePyrotechnic
Copy link

This was occurring to me in Path of Exile 2 with both DX12 and Vulkan.
With Steam overlay disabled, LD_PRELOAD="" did not work
With Steam overlay enabled, LD_PRELOAD="" worked
However, I did not test steam overlay enabled without LD_PRELOAD, so I cannot say whether that or enabling Steam overlay (or both) did the trick.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests