Add multi-threaded options to 3D physics

[Zylann]

There are 3 aspects to this:

• Make it thread-safe (at least for resources if not already)
• Make it possible to run simulation in a thread
• Avoid stalls when creating physic resources (which is my main problem at the moment)

The project you are working on:

I am working on a voxel engine, which is also used by other people's projects.

The problem or limitation you are having in your project:

As described in Zylann/godot_voxel#124
In a voxel terrain, many meshes are getting created dynamically at runtime. Creating mesh colliders is 10 times slower than creating visual meshes (mostly due to computing its BVH), and that takes a lot of time. Unlike VisualServer and PhysicsServer2D, PhysicsServer has no thread model option, so I assume it's basically not thread-safe and runs on the main thread (correct me if I'm wrong). So I currently have no choice but to create my shapes on the main thread, which limits how many I can process per frame before starting to stall the game.
Beyond my own case, this also has the broader issue of preventing scenes from being loaded in a thread, since creating shape resources they might contain would be unsafe too (if they actually are, please document!).

Describe the feature / enhancement and how it helps to overcome the problem or limitation:

The way it used to be done:

If I take the same design as VisualServer and PhysicsServer2D from back in Godot 3, here is how it would be a dropdown in ProjectSettings:

• Single-Unsafe would be the option for running physics on the main thread, without thread safety (basically how it is now).
• Single-Safe would still run physics on the main thread, but with thread-safety in case a thread calls its functions
• Multithreaded would run physics on a separate thread and also be safe to be called from other threads.

This would be a first improvement since the physics server would have its own thread time to spend without all the other stuff done in the main one.

If we go for that design, PhysicsServer would be wrapped into a PhysicsServerWrapMT class which pushes calls to its function into a ring buffer, and dequeues calls from that buffer from inside the physics thread (or the main thread in case of Single-Safe).

HOWEVER, there is a big flaw in this design, which is important for my use case: when I create a large amount of complex Shape resources, it needs to call Bullet/other impl. which in turn has to build its BVH for that shape. So a call to the server sounds unavoidable. If all calls are wrapped the same way, the server will dequeue calls to create shapes, and that can stall the simulation because they will be processed in the same thread that runs the simulation, even if those shapes don't intervene in the simulation yet. And if the result needs to be accessed just after creation, the queue has to be flushed immediately, which defeats the reason to run in a thread in the first place.

This problem might exist in other servers already, however that's mostly an assumption I'm making since I don't know their code in detail (especially with Vulkan rewrite). So perhaps not the whole server should be threaded the same. What I'm asking here is that creating resources involving computations which don't intervene in the frame logic should not slow it down.
Maybe resource creation could have another ring buffer and be processed in another thread? (Although I use several high-processing threads already so I'm still wary of creating new threads for too many things, as it may still lead to contention when exceeding CPU limits).
Or, a simple solution would be to not wrap those calls and let them be processed inside the calling thread, which would solve my problem best since I already do the call from a thread. It's only viable if the corresponding server code is officially safe for that kind of contract.

How it should be done:

So that leads to the following:

• Instead of a "thread model", we should be able to choose how many threads are dedicated to simulation of worlds, either statically (in ProjectSettings) or dynamically (via code or some policy option) because games have different uses for threads, and players have different CPUs. If no threads are used, the simulation may run on the main thread. If 1 or more threads are used, the simulation will use that amount of threads if it can.
• Functions for creating and cooking resources such as shapes must be thread-safe regardless of thread count, and without impacting framerate, similarly to the Vulkan renderer (contention must be low). Also, they must not be put in some message queue to be run later, they must be able to run in the caller's thread, because it gives control over the way their creation is dispatched and prioritised (in my case, my threaded tasks use a priority system and are not bound to frames)
• These things would likely be done in the implementation of each physics engine, not in a generic Godot wrapper.

This alternative could be at least as simple as a PhysicsServer API.

If this enhancement will not be used often, can it be worked around with a few lines of script?

I already had a workaround so far by doing everything on the main thread. It limits performance dramatically and complexifies the code because I have to scatter away the collider part and maintain a timed queue with the data myself. In many cases you can see terrain parts loading and you have to wait a while for those on the main thread, before you can start moving around and edit.

Is there a reason why this should be core and not an add-on in the asset library?:

This is already core for other servers, the feature is just missing in PhysicsServer. But I want to also emphasize on the resource creation case, which is the original reason why I bumped into this.
But it can technically be done with different physics engines once GDExtension gets an API to implement physics servers. Point being, it should not be something Godot implements on your behalf (and if so, must be an optional last resort).

[starwolfy]

I think that this feature would be a huge addon to Godot. PhysicsServers are often used for better performance and multi-threaded access; having more parallel options with 3D Physics Servers will definitely increase their potential.

[smix8]

It is not only an issue with mesh colliders but also 3D gameplay code that requires updated raycast positions from the physics server and/or run in _physics_process() in general.

Very common 3D game features like a characters head lookat or inverse kinematic hand and feet placement make the physics server stall the visual server so much that framerates are constantly halfed or worse with multithreading enabled while they run fine in singlethread mode.

[m4nu3lf]

Because of this issue when using godot_voxel in my project with a big terrain I was having consistent stuttering. To fix it I had to patch Godot in such a way that convex polygon shapes are created asynchronously within the BulletPhysicsServer.
I also had to add a way to flush the pending changes.
It's not a clean solution but it will allow me to keep working on my project.

[Malkverbena]

Any Update about this proposal?

[Calinou]

Any Update about this proposal?

For Godot 4.0, we no longer need to cater to Bullet since only GodotPhysics will be provided b y default.

However, we don't have an active physics maintainer anymore, so this feature will take a while to be implemented (and may not be done in time for 4.0).

[Zylann]

Here is an update for the problem I initially had:

I am still heavily bottlenecked by shapes having to be created on the main thread (and I'm already limiting ranges / simplifying meshes). I have a job system cooking many meshes really quickly but it ends up most of the time waiting for the main thread. The ideal (and minimal) solution I would need would be the ability to fully create shapes from my own threads, in a safe way. I don't even need physics to simulate on a different thread, yet.

To achieve this, the generic API might be affected a little. It depends if we want multithreading to be enforced by the generic API, or if it should be left to implementers. IMO it may depend on the chosen physics engine. I believe Godot Physics could have that feature.

Here are more detailed points I wrote earlier on RC:

1. Have create_shape_offline(all_data_necessary), or a way to support it in the API. The shape create implementation must not interact with simulation state until added to a body, I think that's common sense. Like, really just allocate whatever data is needed, precalculate whatever data structure, based on arguments given at the same time (like faces, BVH may be done here). Currently, several calls are needed sometimes (for example create, followed by set_faces). This API is a bit problematic because it opens the door to cases where the shape became attached to a body in the meantime. In a multithreaded scenario that means plenty of avoidable locking. Those functions could stay, but we should also be able to send all data in the create call. Because...

2. Have it run on the caller thread safely. For meshes and convex hulls, logic described in 1) is the expensive part and should run there, even if the physics engine simulates in its own thread. This will also help cases of loading a scene in parallel, as shape resources will simply be able to be built from that thread. At the end of shape creation, the function should return a RID. That probably means inserting it into a RidContainer of some sort. It could be protected with a simple mutex or lockless thing, this is fine because insertion should be really quick compared to the work done in 1).

3. Document the behavior per physic engine. If these things are not possible due to how the physics engine is implemented (for example the ReactPhysics library uses a custom allocator which doesnt work in multithreaded scenarios), then it must be documented because for these engine implementations, having thread-safety will mean to lock the entire physics for the full call instead, and can cause enough stalls to be a show-stopper.

---

Overall I think this proposal's title is only part of the problem, and should not be implemented the way Godot has it currently.
This is what actually needs to be done:

• We should be able to choose how many threads are dedicated to simulation of worlds, either statically or dynamically (because games have different uses for threads, and players have different CPUs)
• Functions for creating and cooking resources such as shapes must be thread-safe regardless of the model, and without impacting framerate, similarly to the Vulkan renderer. Also, they must not be put in some message queue to be run later, they must be able to run in the caller's thread.
• These things must be done in the implementation of each physics engine, it must not be done in a generic Godot wrapper.

[Malkverbena]

@Zylann I've seen your contributions, especially in the double precision support. Noce job!
Should this proposal be added to Physics 3D TRACKER?
Camille is not the physics maintainer anymore?

[Calinou]

Camille is not the physics maintainer anymore?

No, as he went on to work at Rockstar San Diego (and is no longer working on Godot as a result).

[Marzin-bot]

I too have performance issues. If I understood correctly, when you talk about multithreaded physics, the physics will be calculated on a separate thread from the main thread. But physics is still single-threaded. It will be necessary to add the possibility of allocating a certain number of threads where the calculation of the physics will be distributed. This would be useful for game servers. Alternatively, add an option where each world resource will have a dedicated physical thread independent of other worlds so that one instance of Godot can handle multiple games at the same time on a dedicated server.

[Zylann]

If I understood correctly, when you talk about multithreaded physics, the physics will be calculated on a separate thread from the main thread. But physics is still single-threaded.

I actually edited my proposal recently, check How it should be done.

Worlds each running simulation in their own thread is one way of distributing the workload when multiple threads are provided, although that would be an option, because not all games use multiple worlds and yet could need multithreading. For example, simulation in a world can be distributed between interacting islands. It's up to the implementation to support this.

[KnightNine]

Anyone know if there's a PR which implements 3D threaded physics for 3.X?

[Calinou]

Anyone know if there's a PR which implements 3D threaded physics for 3.X?

Not that I know of. Most of the development focus has been on 4.x lately, as it's where most larger-scale projects are being developed.

[warent]

Just for clarification, does World3D utilize threads such that each world may be in its own thread?

According to the documentation

When using multi-threaded physics, access is limited to Node._physics_process in the main thread.

And then looking at the setting;

It's all a bit ambiguous to me. If I had to guess, it sounds like every World3D still shares the same thread even with multi-threaded physics enabled, correct?

[smix8]

@warent
A World2D/3D is nothing more than an ID number that bundles together other ID numbers. It does not do anything by itself.

How a server decides to process all those parts that are relevant to it is a matter of each individual server implementation. PhysicsServer, RenderingServer, NavigationServer, AudioServer ... they all do threading differently or not at all.

Commonly the servers that use threads do so at a far lower level to speed things up, e.g. each individual item as a group task. Most projects have only 1 World2D/3D in use so having the threading at the world level would have no benefit for those projects.