2D Physics Run on Seperate Thread Crash

[DenysAshikhin]

Tested versions

• Reproducible in :
  v4.3.beta2.mono.official.b75f0485b
  v4.2.stable

System information

Godot v4.3.beta2.mono - Windows 10.0.22631 - Vulkan (Mobile) - dedicated NVIDIA GeForce RTX 4090 (NVIDIA; 31.0.15.5222) - AMD Ryzen 9 7900X 12-Core Processor (24 Threads)

Issue description

When enabling Run on Seperate Thread for Physics 2D, the game run from the editor crashes without any error. However, disabled it, the game runs perfectly fine.

Video below to demonstrate: https://youtu.be/VzHw3-da53U

Please let me know if there's any way I can get more logs/information about such crashes

Steps to reproduce

Video demonstrating it:
https://youtu.be/VzHw3-da53U

Please let me know if there's any way I can get more logs/information about such crashes

Minimal reproduction project (MRP)

mrp (2).zip

[DenysAshikhin]

Any update regarding how I can get some more logs of the crash for troubleshooting?

[AThousandShips]

You need to use a debug build of the engine, and run from console, and get the error logs from that

But the best thing would be to have an MRP, or more info, check here for more on debugging and logs

[DenysAshikhin]

You need to use a debug build of the engine, and run from console, and get the error logs from that

But the best thing would be to have an MRP, or more info, check here for more on debugging and logs

I would be hard pressed to make an MRP without knowing which part is causing it - it's not a small project I have that I can condense that quickly unfortunately.

That being said, where can I get the debug build of the engine?

[AThousandShips]

Right where I linked are the instructions, and you'll need to build for your os the details are on the same page. there are no official builds with debugging, so you need to build your own

I would be hard pressed to make an MRP without knowing which part is causing it

Absolutely, but that makes it even harder for anyone else to identify and fix it, so without an MRP it's very hard to do anything

Creating a new project and recrating some conditions would be the way to go probably, but with debugging you can start pinpointing where the issue comes from at least

[DenysAshikhin]

That's my idea is to at least get an idea of what I need to recreate - the only thing for sure I know it's related to a lot of areas/collisions happening at once. Because if I tone down the number of units (the number in the video was used more as a performance testing/benchmark) it works no problem. I didn't scroll up to see the windows instructions my bad. That being said, this is what the 4.3 console shows:

I will follow up with what the dev build does

[DenysAshikhin]

================================================================
CrashHandlerException: Program crashed
Engine version: Godot Engine v4.3.beta.custom_build (f0d15bbfdfde1c1076903afb7a7db373580d5534)
Dumping the backtrace. Please include this when reporting the bug to the project developer.
[0] RID::operator== (C:\Users\denys\Documents\godot\source_code\godot\core\templates\rid.h:44)
[1] GodotArea2D::BodyKey::operator== (C:\Users\denys\Documents\godot\source_code\godot\servers\physics_2d\godot_area_2d.h:78)
[2] HashMapComparatorDefault<GodotArea2D::BodyKey>::compare (C:\Users\denys\Documents\godot\source_code\godot\core\templates\hashfuncs.h:399)
[3] HashMap<GodotArea2D::BodyKey,GodotArea2D::BodyState,GodotArea2D::BodyKey,HashMapComparatorDefault<GodotArea2D::BodyKey>,DefaultTypedAllocator<HashMapElement<GodotArea2D::BodyKey,GodotArea2D::BodyState> > )
[4] HashMap<GodotArea2D::BodyKey,GodotArea2D::BodyState,GodotArea2D::BodyKey,HashMapComparatorDefault<GodotArea2D::BodyKey>,DefaultTypedAllocator<HashMapElement<GodotArea2D::BodyKey,GodotArea2D::BodyState> > )
[5] HashMap<GodotArea2D::BodyKey,GodotArea2D::BodyState,GodotArea2D::BodyKey,HashMapComparatorDefault<GodotArea2D::BodyKey>,DefaultTypedAllocator<HashMapElement<GodotArea2D::BodyKey,GodotArea2D::BodyState> > )
[6] GodotArea2D::call_queries (C:\Users\denys\Documents\godot\source_code\godot\servers\physics_2d\godot_area_2d.cpp:269)
[7] GodotSpace2D::call_queries (C:\Users\denys\Documents\godot\source_code\godot\servers\physics_2d\godot_space_2d.cpp:1129)
[8] GodotPhysicsServer2D::flush_queries (C:\Users\denys\Documents\godot\source_code\godot\servers\physics_2d\godot_physics_server_2d.cpp:1323)
[9] PhysicsServer2DWrapMT::flush_queries (C:\Users\denys\Documents\godot\source_code\godot\servers\physics_server_2d_wrap_mt.cpp:71)
[10] Main::iteration (C:\Users\denys\Documents\godot\source_code\godot\main\main.cpp:4055)
[11] OS_Windows::run (C:\Users\denys\Documents\godot\source_code\godot\platform\windows\os_windows.cpp:1686)
[12] widechar_main (C:\Users\denys\Documents\godot\source_code\godot\platform\windows\godot_windows.cpp:180)
[13] _main (C:\Users\denys\Documents\godot\source_code\godot\platform\windows\godot_windows.cpp:206)
[14] main (C:\Users\denys\Documents\godot\source_code\godot\platform\windows\godot_windows.cpp:220)
[15] WinMain (C:\Users\denys\Documents\godot\source_code\godot\platform\windows\godot_windows.cpp:234)
[16] __scrt_common_main_seh (D:\a\_work\1\s\src\vctools\crt\vcstartup\src\startup\exe_common.inl:288)
[17] <couldn't map PC to fn name>
-- END OF BACKTRACE --
================================================================

Got a more proper stack-trace this time @AThousandShips any idea what this is related to?

[AThousandShips]

It's related to an Area2D but that's all I know

[DenysAshikhin]

Are there any illegal operations/calls when using physics in its own thread? Like I can't call x functions outside _phyiscs_process or something?

[AThousandShips]

You can't call a lot of physics things outside of _physics_process but they generally have warnings

But this pinpoints it to being related to areas, that should help narrowing it down

[DenysAshikhin]

Alright, I'll try to make an mrp...

[DenysAshikhin]

@AThousandShips
I manage to make it! How can I share it?

Edit:
Would a github repo work?

[AThousandShips]

You upload it, it should be small so best upload it directly, just drag it to the input field and keep it focused until it's done uploading, make sure to add it to the original post

[DenysAshikhin]

mrp.zip

[DenysAshikhin]

You upload it, it should be small so best upload it directly, just drag it to the input field and keep it focused until it's done uploading, make sure to add it to the original post

Let me know if it crashes for you - specifically the physics has to be on separate thread (on single it works fine)

[AThousandShips]

The MRP is not really working, it completely breaks down performance wise because you didn't copy all the code into the MRP, you removed parts that were necessary, for example the _disable_bullet method, please test your own MRP and see

It completely bogs down, but it does not crash

[DenysAshikhin]

mrp (2).zip

@AThousandShips I apologise - I got too excited that I could replicate so easily and zipped the wrong file.
I've also attached a video with the mrp working (single threaded) and crashing (with physics multi-threading)

https://github.com/godotengine/godot/assets/10791321/39564a6e-519b-48e9-b910-02d47aa3e544
https://github.com/godotengine/godot/assets/10791321/5be3f732-9464-4195-ba5d-9c8c0638207a

Edit:
Bogging down is 100% intentional as I was looking/trying different things to see how much extra fps it would net me in worst case scenarios. It's partly how I ran into this crash in the first place

[AThousandShips]

Will test as well when I have time!

[DenysAshikhin]

@AThousandShips any luck replicating?

[AThousandShips]

Haven't found the time, will see if I can or someone else can try replicating it

[DenysAshikhin]

@AThousandShips just gonna ping you one more time if you have a chance to confirm it's happening on your machine 😅

[AThousandShips]

Sorry I haven't found the time still, will try find some time once 4.3 is released

[AThousandShips]

I can confirm that it crashes specifically on threaded, but it only does this once it's broken down completely performance wise, I don't expect this to happen in any reasonable case, and the crash is almost certainly caused by the fact that things are bogged down completely

While it shouldn't crash, I'm not sure it's ever going to crash in any reasonable case given that it only crashes with this MRP in extreme slowdown and under an extreme load level (on non-threaded the game simply bogs down to freezing with <1 fps even on my reasonably powerful computer)

[DenysAshikhin]

That's fair point, however, even when performance is fine this is relatively repeatable. I have attached an mrp for convenience sake, but all I did was in main_scene.gd -> line 69 -> if (drones.size() < (75) and true): I changed the values from 225 to 75. The higher the value the more consistent the crash. I've had it crash at values as small as 5, but maybe 5% of the time. And it went up from there.

Also a video to show I'm not crazy 😅
mrp3.zip
https://youtu.be/SOUhyYcZz98

[AThousandShips]

Will test that case as well and see! Always important to provide good references for bugs and their scope

[DenysAshikhin]

Sounds good, this definitely feels like some sort of race condition that happens specifically during spawning. Because I notice it either crashes while something is being spawned, or doesn't crash once everything is spawned. It's also why higher numbers make this more consistent since there are more frames spent spawning units... but that's just my guess

[DenysAshikhin]

@AThousandShips I know it's been dead for a while but I'm still around and hoping this can be addressed/resolved 😅

P.S.
4.4 dev 3 also has this issue!

[AThousandShips]

Can't add any more details to this right now, so needs someone with physics expertise to look at it

[DenysAshikhin]

Are you able to replicate the issue on your end with reasonable performance now with the small changes I mentioned?

[AThousandShips]

I don't remember I will test soon and confirm