Resistance 1: freeze with error VKGSRender::patch_transform_constants

[digant73]

Quick summary

Got the following error causing a freeze on Resistance 1:

F {RSX [0x231ea9c]} SIG: Thread terminated due to fatal error: Verification failed (object: 0x0)
(in file D:\a\1\s\rpcs3\Emu\RSX\VK\VKGSRender.cpp:2481[:3], in function 'void VKGSRender::patch_transform_constants(rsx::context *,uint,uint)') (error=0x578)

Not sure if it could be a regression of the recent merged PRs

Details

No response

Attach a log file

RPCS3.log

Attach capture files for visual issues

No response

System configuration

No response

Other details

No response

[kd-11]

The only thing that changed is that your old cache is now invalid, I'm guessing this is just a corner case when running the game with no shader cache.

[digant73]

is there something I can do to provide more info?

[kd-11]

I just have to review it by eye and hope to catch the problem. I'm not able to reproduce the problem.

[digant73]

It happened after some time for two times. I will clear again the cache and test it again. Just to verify it occurs under some circumstances

[digant73]

Here again the same error after just 2 minutes in the same place and when compiling shaders. I cleared all caches before.
Made a backup of the game save for testing in case a bugfix is released.

RPCS3.log

[kd-11]

Check if #16540 does anything for this issue. If the problem is still there I'll create a debugging branch with some extra logging added. I cannot see how this can happen unless the fill is being skipped.

[digant73]

ok I will test it

[digant73]

No, the issue is not fixed by #16540

[Darkhost1999]

I was able to reproduce this issue this morning and discussed it with Jonathan. He quoted the other issue he has open for Resistance without looking at the file and function the error is produced in. This is a separate regression. Meanwhile Resistance Fall of Man still has an active regression on a set mission I was able to reproduce the issue before getting Health regeneration and again before the first mission was even completed.

windows 11
gtx 1660 super
ryzen 7 7700x

I tried with clean cache and resetting setting to default with the only change I intentionally used was atomic rsx fifo accuracy. Nothing else was configured.

[digant73]

It is not a regression of #16466. It is present also on the previous release.
It is a permanent error I get in the same place so I cannot advance in the game (I started the game from the beginning and now I always get this error in the same place). The freeze is shown in the picture below.
@Darkhost1999 which are the other regressions mentioned by Jonathan?

[Darkhost1999]

#14809 The other regression for Resistance Fall of Man still active is this 1. But the error message is different showing that this is a different issue.

[digant73]

Made some testing with 10 runs for each Sahder mode. Below the results:

10 runs (4 KO) - Legacy (single threaded)

• 2 freezes with error:

F {RSX [0x234680c]} SIG: Thread terminated due to fatal error: Verification failed (object: 0x0)
(in file D:\a\1\s\rpcs3\Emu\RSX\Common\texture_cache.h:387[:7], in function 'std::pair<bool,vk::render_target *> rsx::texture_cache<vk::texture_cache,vk::texture_cache_traits>::sampled_image_descriptor::is_expired<vk::surface_cache,vk::render_target*>()')

• 2 freezes with error:

F {RSX [0x1e3dcf4]} SIG: Thread terminated due to fatal error: Assertion Failed! Vulkan API call failed with unrecoverable error: Device lost (Driver crashed with unspecified error or stopped responding and recovered) (VK_ERROR_DEVICE_LOST)

10 runs (all KO) - Async (multi threaded)

• 9 freezes with error:

F {RSX [0x231ea9c]} SIG: Thread terminated due to fatal error: Verification failed (object: 0x0)
(in file D:\a\1\s\rpcs3\Emu\RSX\VK\VKGSRender.cpp:2481[:3], in function 'void VKGSRender::patch_transform_constants(rsx::context *,uint,uint)') (error=0x578)

• 1 freezes with error:

F {RSX [0x1e23ff8]} SIG: Thread terminated due to fatal error: Verification failed (object: 0x0)
(in file D:\a\1\s\rpcs3\Emu\RSX\Common\texture_cache.h:387[:7], in function 'std::pair<bool,vk::render_target *> rsx::texture_cache<vk::texture_cache,vk::texture_cache_traits>::sampled_image_descriptor::is_expired<vk::surface_cache,vk::render_target*>()') (error=0x578)

10 runs (2 KO) - Async with Shader Interpreter

• 1 freezes with error:

F {RSX [0x1e3dcf4]} SIG: Thread terminated due to fatal error: Assertion Failed! Vulkan API call failed with unrecoverable error: Device lost (Driver crashed with unspecified error or stopped responding and recovered) (VK_ERROR_DEVICE_LOST)

• 1 freezes with error:

F {RSX [0x1e56444]} SIG: Thread terminated due to fatal error: Verification failed (object: 0x0)
(in file D:\a\1\s\rpcs3\Emu\RSX\Common\texture_cache.h:387[:7], in function 'std::pair<bool,vk::render_target *> rsx::texture_cache<vk::texture_cache,vk::texture_cache_traits>::sampled_image_descriptor::is_expired<vk::surface_cache,vk::render_target*>()') (error=0x578)

All the issues happen in the same place.
If the device-lost could be related to something corrupted propagated by RSX (e.g. an invalid buffer) then it could be enough to investigate and hopefully fix the other issues.

Attached the log of the issues for all the runs above

R1_crash.zip

[digant73]

#14809 The other regression for Resistance Fall of Man still active is this 1. But the error message is different showing that this is a different issue.

That bug is related to texture_cache also reported in the testing's results in my previous post. I'm currently also in the main chapter Bristol - Devil at the door but the freezes are in general present in all the game.

[kd-11]

Try this: https://github.com/kd-11/rpcs3/tree/constants-crash-debug
It should dump the vertex program layout at the point of the crash. I have a hunch what is happening, I just need to prove the theory.

[kd-11]

As for the other issues:

1. Device lost - we found some issues already, I forgot to merge the patches into master though (they're still on gtx900 branch). I'll include those in the current PR. However, problems continued without much info in the logs, so paused for now.
2. Crash in texture cache (is_expired check). I can fix this if I can reproduce. I really suck at these PS3 shooters , let me know if there are hacks/cheats to skip to the crashing location quickly, or if I can make the crash happen by standing in one spot.

[Darkhost1999]

2. Crash in texture cache (is_expired check). I can fix this if I can reproduce. I really suck at these PS3 shooters , let me know if there are hacks/cheats to skip to the crashing location quickly, or if I can make the crash happen by standing in one spot.

There's no easy way even with the savedata provided. All I can think of is defeat the enemies that are before the "devil's gate" door then create a savestate and see if the savestate helps.
Will test in a moment for this issue though that I also haven't found an easy way to make crash. Although someone else said they had a consistent spot to cause crashing.

[digant73]

Try this: https://github.com/kd-11/rpcs3/tree/constants-crash-debug It should dump the vertex program layout at the point of the crash. I have a hunch what is happening, I just need to prove the theory.

Got this output on the GUI followed by the freeze at the first run:

E RSX: Crash imminent. Unexpected constants layout. Index=46, Count=4, Prog=0000000000000000, Vtx=0000000169b41fb8
E RSX: Vertex constants indices:
0, 8, 9, 10, 11, 22, 32, 33, 34, 46, 47, 48, 
>> <<

S RSX: Program compiled successfully
F {RSX [0x1e15478]} SIG: Thread terminated due to fatal error: Verification failed (object: 0x0)
(in file M:\console\git\rpcs3_kd-11\rpcs3\Emu\RSX\VK\VKGSRender.cpp:2505[:3], in function 'void VKGSRender::patch_transform_constants(rsx::context *,uint,uint)') (error=0x578)

RPCS3.log

Same issue at second run:

E RSX: Crash imminent. Unexpected constants layout. Index=46, Count=4, Prog=0000000000000000, Vtx=00000000308e6ef8
E RSX: Vertex constants indices:
0, 8, 9, 10, 11, 22, 32, 33, 34, 46, 47, 48, 
>> <<

S RSX: Program compiled successfully
F {RSX [0x2314e7c]} SIG: Thread terminated due to fatal error: Verification failed (object: 0x0)
(in file M:\console\git\rpcs3_kd-11\rpcs3\Emu\RSX\VK\VKGSRender.cpp:2505[:3], in function 'void VKGSRender::patch_transform_constants(rsx::context *,uint,uint)') (error=0x578)

Always on the same point/place

[kd-11]

I see. It's a skipped draw call. I'll push the fix.

[digant73]

Error on patch_transform_constants is present only with Async (multi threaded) on all my testing (if it can help).
Attached a recording with the point of the freeze (it is when I'm close to the barrier which I'm shooting to before the end of the video).

https://mega.nz/file/K6p1iS6D#15Zde7e9sl1jOZq9xp04kXE-HRSXY9YMf8iozH3CWuA

[kd-11]

It's missing with legacy though which never has a missing shader. That was the hint.

[digant73]

Latest update on #16540 seems to fix the error on patch_transform_constants. Made 10 runs with Async and got 1 error but only on texture_cache ( as it was before). Made also 3 runs, as no-regression, with Legacy and Async + Interpreter only. Similarly to previous tests, I got 1 error on texture_cache and 1 device-lost. They happen in the same point, just before the freeze related to patch_transform_constants. They are possibly related.

[digant73]

@kd-11 as indicated by @Darkhost1999, the error on texture_cache was reported in #14809. In that bug report a user also reported that the bug was present since build 0.0.28-15302 (#14104). I made some testing with that PR and the previous build 0.0.28-15277 and I confirm that this error is present since #14104.

[kd-11]

For the texture cache crash, run with https://github.com/kd-11/rpcs3/tree/debug-is-expired-crash and attach log. I think I know why it's crashing, just need evidence.

[digant73]

ok. Compiling and testing now

[digant73]

here the log with the crash

RPCS3.log

[digant73]

a second crash

RPCS3.log

Both logs report an imminent crash. Something like:

·E 0:15:18.542487 {RSX [0x1ea8b40]} RSX: A crash is imminent. Ref_addr=0xc1683e80, image_handle=000000007f678480, external_handle=0000000000000000, op=0, simplified=No

[kd-11]

@digant73 What happens if you disable MSAA? Does the crash go away?

[digant73]

do you mean setting Anti-Aliasing in the GPU tab from Auto to Disabled?

[kd-11]

do you mean setting Anti-Aliasing in the GPU tab from Auto to Disabled?

Yes

[digant73]

same crash

RPCS3.log

[digant73]

@kd-11 let me know if there is something else I can do on this remaining bug. Also, did you integrate the fixes you made in the gtx900 branch? if maybe it could make sense to test with also those fixes

[kd-11]

I'll add gtx900 changes to the PR but that won't fix the expiration problem. Unfortunately I just have to see that one for myself.
A texture comes in claiming to be from the surface cache but it is not a valid surface cache object and the surface cache denies ever having a texture at the address we're probing. Something is very wrong there.
That said, I believe the is_expired crash has it's own ticket, so I'll close this one with the PR anyway.

[digant73]

yes, I will open a ticket for the crash related to is_expired

[Darkhost1999]

yes, I will open a ticket for the crash related to is_expired

That'd be #14809

[digant73]

yes, I will open a ticket for the crash related to is_expired

That'd be #14809

In that case, it should be be better to provide in the detail section the info about the regressed PR. So:

build 0.0.28-15302 (#14104)

Anyway, I think that it is hard for developers to book, see and fix old reported bugs. Possibly the user that opened the bug is no more available to provide further info or to test a possible bug fix