N64 games crash or freeze In-Game when using the latest (v139) version. (Windows 10)

[Irityan]

Describe the bug
After updating I noticed that all my N64 games stopped working on my Win10 machine.
With any game that I tried, the game boots but then stops working.
OpenGL driver would crash the software entirely, while DirectX would make it freeze at some frame.

Rollback to previous version (v138) resolves the issue.

In comparison, SNES games, for example, seem to be working as intended.

To Reproduce
Steps to reproduce the behavior:

1. Open any N64 ROM on the most recent version (v139).
2. Wait for a bit while the intro plays.
3. Emulator crashes (or freezes, if DirectX is selected).

[bsdcode]

I can confirm:

I'm using FreeBSD 14.1-RELEASE-p2 with packages from the latest repository, with the nvidia driver version 550.54.14 for my GTX 1080 TI. I compiled the latest version of ares from the main branch of this repository.

Emulator freezes after starting any N64 rom. The exact time when the freeze happens is always different (even for the same game), but it will happen after a short while. dmesg is logging the following when it happens:

NVRM: Xid (PCI:0000:01:00): 13, pid='<unknown>', name=<unknown>, Graphics Exception: SKEDCHECK04_LOCAL_MEMORY_CRS_SIZE failed
NVRM: Xid (PCI:0000:01:00): 13, pid='<unknown>', name=<unknown>, Graphics Exception: ESR 0x407020=0x800 0x407028=0x300 0x40702c=0x57316 0x407030=0x0
NVRM: Xid (PCI:0000:01:00): 13, pid='<unknown>', name=<unknown>, Graphics Exception: ChID 002d, Class 0000c197, Offset 00000418, Data 00000000

The only other core I tested is the Sega Mega Drive core. It works without problems.

The previous version v138 doesn't have this problem with N64 roms. I can do a git bisect to identify the offending commit when I have a little bit more time.

[bsdcode]

The problem starts with this commit: f31d209

This commit has many changes, so in order to identify the root of the problem we probably have to make some more observations. And indeed, I realized the freeze doesn't happen as randomly as it first seems. Trying and restarting the same N64 rom again and again the game lasts longer on each try before it freezes again. Therefore I guessed it probably has something to do with an "OpenGL/Shader-video driver-cache issue" (nothing to do with librashader!). It tries to create/precompile/load/save some shader, fails and the emulator freezes. The next time the requested precompiled shader is available and the game can continue further than the last time, until the next new shader must be compiled.

And lo and behold: disabling the NVIDIA shader disk cache (by default it lives under ~/.cache/nvidia/GLCache) with __GL_SHADER_DISK_CACHE=0 ./ares prevents this problem. I have played the very first areas of Resident Evil 2 and Shadow Man and I didn't encounter a freeze again during this short test. I also tested this with the latest version of ares, i.e. compiled from HEAD.

@Irityan: could you try something similiar, i.e. try to disable the shader cache and see if that's fixing your issue too?

But we still have to check if ares is doing something wrong here, because other programs have no problems with the cache.

[LukeUsher]

That commit is purely an update to parallel-rdp and nothing more; have you checked if other emulators that use parallel-rdp are impacted?

[Irityan]

@bsdcode can confirm, disabling shader cache in Nvidia's control panel indeed makes the games playable again.

[Irityan]

(my bad, I rarely use github's ui and closed the issue by complete accident)

[remutro]

If you could check on another emulator such as simple64 ( https://github.com/simple64/simple64/releases/tag/v2024.06.2 ) which uses parallel-rdp also and you can reproduce the issue there, then I would open an issue on parallel-rdp's GitHub: https://github.com/Themaister/parallel-rdp

[Irityan]

If you could check on another emulator such as simple64 ( https://github.com/simple64/simple64/releases/tag/v2024.06.2 ) which uses parallel-rdp also and you can reproduce the issue there, then I would open an issue on parallel-rdp's GitHub: https://github.com/Themaister/parallel-rdp

Yep, same exact issue and same exact solution. I guess this issue here can be marked as closed then?

[remutro]

It's good information to have for sure, but I would open it up on parallel-rdp's github for resolution. I would definitely mention it occurs both on simple64 and ares, and the workaround you found to address the issue.

https://github.com/Themaister/parallel-rdp

[bsdcode]

Thank you @Irityan for confirming the suggested workaround and for testing other emulators. I tried to compile simple64 here on FreeBSD, but it's not as easy to compile as ares.

[Themaister]

Can someone confirms it works on Turing cards? (2000 series or 1600 series).

[kev4cards]

Windows 11 Home 23H2 Build 22631.4037 laptop with Intel Core i7-9750H CPU, NVIDIA GTX 1660 Ti GPU (driver 556.12), and 16 GB DDR4 SDRAM. Works fine here.

I have noted about 15-20 of these freezes over the past two years from ares, simple64, and RMG. Out of the few that I could obtain information from, there were definitely a few with drivers newer than 535.xx. So, I am not sure if NVIDIA may a full fix or something partial or what. I just know getting enough useful hardware and software information has been hard to come by. Also, December 9th, 2022 is the date where paraLLEl-RDP works for most people. I think there was one that did not have success with that build, though. I am fuzzy on that as with next to all of these reports because user response has not been good and most were around the time of the sync2 commits.

There is at least one active ares user and ares Discord member with a GTX 1060 and driver 536.23 that encounters the crashes.

[Themaister]

Well, I pushed an update to parallel-rdp-standalone with updated Granite which includes a workaround for Pascal GPUs and earlier.

[bsdcode]

Thanks! I tested your update with ares and it solves the issue here with the GTX 1080 Ti and driver version 550.54.14 on FreeBSD 14.1-Release.