Vulkan: Crash in scene with 128 OmniLights and 128 SpotLights rendered at the same time (or Decals/ReflectionProbes) #56657

Calinou · 2022-01-09T21:04:44Z

Godot version

4.0.dev (b6b81e8)

System information

Fedora 34, GeForce GTX 1080 (NVIDIA 495.46)

Issue description

The engine crashes when opening a scene with 128 OmniLights and 128 SpotLights rendered at the same time. If only the 128 OmniLights or 128 SpotLights (not both) are instanced, there is no crash.

The highest amount of combined lights I can render successfully is 103 OmniLights + 103 SpotLights. Past that, I always get a crash when the project starts.

This can also be reproduced with Decals and ReflectionProbes, both of which are clustered elements.

Decreasing Max Clustered Elements in the Project Settings to 128 or below does not prevent the crash from occurring.

Backtrace:

handle_crash: Program crashed with signal 11
Engine version: Godot Engine v4.0.dev.custom_build (eb469a72c642d11779512e67e1106f763e305526)
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] /lib64/libc.so.6(+0x3d320) [0x7fd30f65e320] (??:0)
[2] Map<RID, Set<RID, Comparator<RID>, DefaultAllocator>, Comparator<RID>, DefaultAllocator>::_find(RID const&) const (/home/hugo/Documents/Git/godotengine/godot/./core/templates/rid.h:47)
[3] Map<RID, Set<RID, Comparator<RID>, DefaultAllocator>, Comparator<RID>, DefaultAllocator>::find(RID const&) (/home/hugo/Documents/Git/godotengine/godot/./core/templates/map.h:598)
[4] RenderingDeviceVulkan::_free_dependencies(RID) (/home/hugo/Documents/Git/godotengine/godot/drivers/vulkan/rendering_device_vulkan.cpp:148)
[5] RenderingDeviceVulkan::free(RID) (/home/hugo/Documents/Git/godotengine/godot/drivers/vulkan/rendering_device_vulkan.cpp:8453)
[6] RendererSceneRenderImplementation::RenderForwardClustered::_setup_render_pass_uniform_set(RendererSceneRenderImplementation::RenderForwardClustered::RenderListType, RenderDataRD const*, RID, bool, int) (/home/hugo/Documents/Git/godotengine/godot/servers/rendering/renderer_rd/forward_clustered/render_forward_clustered.cpp:2407)
[7] RendererSceneRenderImplementation::RenderForwardClustered::_render_scene(RenderDataRD*, Color const&) (/home/hugo/Documents/Git/godotengine/godot/servers/rendering/renderer_rd/forward_clustered/render_forward_clustered.cpp:1469)
[8] RendererSceneRenderRD::render_scene(RID, RendererSceneRender::CameraData const*, PagedArray<RendererSceneRender::GeometryInstance*> const&, PagedArray<RID> const&, PagedArray<RID> const&, PagedArray<RID> const&, PagedArray<RID> const&, PagedArray<RID> const&, PagedArray<RID> const&, RID, RID, RID, RID, RID, RID, int, float, RendererSceneRender::RenderShadowData const*, int, RendererSceneRender::RenderSDFGIData const*, int, RendererSceneRender::RenderSDFGIUpdateData const*, RendererScene::RenderInfo*) (/home/hugo/Documents/Git/godotengine/godot/servers/rendering/renderer_rd/renderer_scene_render_rd.cpp:5080)
[9] RendererSceneCull::_render_scene(RendererSceneRender::CameraData const*, RID, RID, RID, unsigned int, RID, RID, RID, RID, int, float, bool, RendererScene::RenderInfo*) (/home/hugo/Documents/Git/godotengine/godot/servers/rendering/renderer_scene_cull.cpp:3221)
[10] RendererSceneCull::render_camera(RID, RID, RID, RID, Vector2, float, RID, Ref<XRInterface>&, RendererScene::RenderInfo*) (/home/hugo/Documents/Git/godotengine/godot/servers/rendering/renderer_scene_cull.cpp:2515)
[11] RendererViewport::_draw_3d(RendererViewport::Viewport*) (/home/hugo/Documents/Git/godotengine/godot/servers/rendering/renderer_viewport.cpp:171)
[12] RendererViewport::_draw_viewport(RendererViewport::Viewport*) (/home/hugo/Documents/Git/godotengine/godot/servers/rendering/renderer_viewport.cpp:233)
[13] RendererViewport::draw_viewports() (/home/hugo/Documents/Git/godotengine/godot/servers/rendering/renderer_viewport.cpp:659)
[14] RenderingServerDefault::_draw(bool, double) (/home/hugo/Documents/Git/godotengine/godot/servers/rendering/rendering_server_default.cpp:92)
[15] RenderingServerDefault::draw(bool, double) (/home/hugo/Documents/Git/godotengine/godot/servers/rendering/rendering_server_default.cpp:377)
[16] Main::iteration() (/home/hugo/Documents/Git/godotengine/godot/main/main.cpp:2703)
[17] OS_LinuxBSD::run() (/home/hugo/Documents/Git/godotengine/godot/platform/linuxbsd/os_linuxbsd.cpp:342)
[18] /home/hugo/Documents/Git/godotengine/godot/bin/godot.linuxbsd.tools.64.llvm(main+0x1c6) [0x45dcd76] (/home/hugo/Documents/Git/godotengine/godot/platform/linuxbsd/godot_linuxbsd.cpp:58)
[19] /lib64/libc.so.6(__libc_start_main+0xd5) [0x7fd30f648b75] (??:0)
[20] /home/hugo/Documents/Git/godotengine/godot/bin/godot.linuxbsd.tools.64.llvm(_start+0x2e) [0x45dcaee] (??:?)
-- END OF BACKTRACE --

103 OmniLights + 103 SpotLights success:

Steps to reproduce

Add a MeshInstance3D with a PlaneMesh.
With a @tool script, instance 128 OmniLight3D and 128 SpotLight3D on the PlaneMesh at any position (as long as the touch the PlaneMesh somehow).

Minimal reproduction project

test_clustering_crash.zip

The text was updated successfully, but these errors were encountered:

qarmin · 2022-01-10T18:21:47Z

For me maximal allowed value is 102, later it crashes with this backtrace

=================================================================
==90592==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x629000a18210 at pc 0x0000161602ff bp 0x7ffe1d2f58b0 sp 0x7ffe1d2f58a0
WRITE of size 4 at 0x629000a18210 thread T0
    #0 0x161602fe in ClusterBuilderRD::add_light(ClusterBuilderRD::LightType, Transform3D const&, float, float) servers/rendering/renderer_rd/cluster_builder_rd.h:305
    #1 0x160c4c6d in RendererSceneRenderRD::_setup_lights(PagedArray<RID> const&, Transform3D const&, RID, bool, unsigned int&, unsigned int&, bool&) servers/rendering/renderer_rd/renderer_scene_render_rd.cpp:3635
    #2 0x16110fce in RendererSceneRenderRD::_pre_opaque_render(RenderDataRD*, bool, bool, bool, RID, RID) servers/rendering/renderer_rd/renderer_scene_render_rd.cpp:4926
    #3 0x16625ef1 in RendererSceneRenderImplementation::RenderForwardClustered::_render_scene(RenderDataRD*, Color const&) servers/rendering/renderer_rd/forward_clustered/render_forward_clustered.cpp:1458
    #4 0x161187a7 in RendererSceneRenderRD::render_scene(RID, RendererSceneRender::CameraData const*, PagedArray<RendererSceneRender::GeometryInstance*> const&, PagedArray<RID> const&, PagedArray<RID> const&, PagedArray<RID> const&, PagedArray<RID> const&, PagedArray<RID> const&, PagedArray<RID> const&, RID, RID, RID, RID, RID, RID, int, float, RendererSceneRender::RenderShadowData const*, int, RendererSceneRender::RenderSDFGIData const*, int, RendererSceneRender::RenderSDFGIUpdateData const*, RendererScene::RenderInfo*) servers/rendering/renderer_rd/renderer_scene_render_rd.cpp:5080
    #5 0x175fff55 in RendererSceneCull::_render_scene(RendererSceneRender::CameraData const*, RID, RID, RID, unsigned int, RID, RID, RID, RID, int, float, bool, RendererScene::RenderInfo*) servers/rendering/renderer_scene_cull.cpp:3221
    #6 0x175d9e59 in RendererSceneCull::render_camera(RID, RID, RID, RID, Vector2, float, RID, Ref<XRInterface>&, RendererScene::RenderInfo*) servers/rendering/renderer_scene_cull.cpp:2515
    #7 0x1771332a in RendererViewport::_draw_3d(RendererViewport::Viewport*) servers/rendering/renderer_viewport.cpp:171
    #8 0x17716518 in RendererViewport::_draw_viewport(RendererViewport::Viewport*) servers/rendering/renderer_viewport.cpp:230
    #9 0x17728427 in RendererViewport::draw_viewports() servers/rendering/renderer_viewport.cpp:659
    #10 0x15b88c54 in RenderingServerDefault::_draw(bool, double) servers/rendering/rendering_server_default.cpp:91
    #11 0x15b93c65 in RenderingServerDefault::draw(bool, double) servers/rendering/rendering_server_default.cpp:375
    #12 0x24201ff in Main::iteration() main/main.cpp:2702
    #13 0x2297b0c in OS_LinuxBSD::run() platform/linuxbsd/os_linuxbsd.cpp:342
    #14 0x2285bb9 in main platform/linuxbsd/godot_linuxbsd.cpp:58
    #15 0x7feb631d0fcf in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58
    #16 0x7feb631d107c in __libc_start_main_impl ../csu/libc-start.c:409
    #17 0x22856ed in _start (/usr/bin/godot4s+0x22856ed)

0x629000a18210 is located 0 bytes to the right of 16400-byte region [0x629000a14200,0x629000a18210)
allocated by thread T0 here:
    #0 0x7feb641a1808 in __interceptor_malloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cc:144
    #1 0x18128141 in Memory::alloc_static(unsigned long, bool) core/os/memory.cpp:75
    #2 0x17774986 in ClusterBuilderRD::setup(Vector2i, unsigned int, RID, RID, RID) servers/rendering/renderer_rd/cluster_builder_rd.cpp:272
    #3 0x1609c98d in RendererSceneRenderRD::render_buffers_configure(RID, RID, int, int, int, int, float, float, RenderingServer::ViewportMSAA, RenderingServer::ViewportScreenSpaceAA, bool, unsigned int) servers/rendering/renderer_rd/renderer_scene_render_rd.cpp:3019
    #4 0x17653eab in RendererSceneCull::render_buffers_configure(RID, RID, int, int, int, int, float, float, RenderingServer::ViewportMSAA, RenderingServer::ViewportScreenSpaceAA, bool, unsigned int) servers/rendering/renderer_scene_cull.h:1163
    #5 0x17711a4f in RendererViewport::_configure_3d_render_buffers(RendererViewport::Viewport*) servers/rendering/renderer_viewport.cpp:142
    #6 0x17716160 in RendererViewport::_draw_viewport(RendererViewport::Viewport*) servers/rendering/renderer_viewport.cpp:224
    #7 0x17728427 in RendererViewport::draw_viewports() servers/rendering/renderer_viewport.cpp:659
    #8 0x15b88c54 in RenderingServerDefault::_draw(bool, double) servers/rendering/rendering_server_default.cpp:91
    #9 0x15b93c65 in RenderingServerDefault::draw(bool, double) servers/rendering/rendering_server_default.cpp:375
    #10 0x24201ff in Main::iteration() main/main.cpp:2702
    #11 0x2297b0c in OS_LinuxBSD::run() platform/linuxbsd/os_linuxbsd.cpp:342
    #12 0x2285bb9 in main platform/linuxbsd/godot_linuxbsd.cpp:58
    #13 0x7feb631d0fcf in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58

SUMMARY: AddressSanitizer: heap-buffer-overflow servers/rendering/renderer_rd/cluster_builder_rd.h:305 in ClusterBuilderRD::add_light(ClusterBuilderRD::LightType, Transform3D const&, float, float)

qarmin · 2022-01-10T18:28:09Z

There is also crash when I instanced 444 Omnilight, so this is not only problem when using both types of Light together

Calinou · 2022-01-22T21:31:41Z

I noticed that decreasing Max Clustered Elements in the Project Settings will cause this crash to happen with fewer lights. Disabling Depth Prepass > Enable in the Project Settings doesn't affect the number of lights at which the crash starts to occur too.

mrjustaguy · 2022-02-26T10:54:42Z

Reflection Probes decrease the number of lights till crash quite a bit..

Calinou · 2022-02-26T18:10:06Z

Reflection Probes decrease the number of lights till crash quite a bit..

This is because they're clustered elements (like Decals). I mentioned that in OP already 🙂

mrjustaguy · 2022-02-26T20:57:40Z

No, I mean Reflection probes act as a sort of multiplier for the Clustered elements they're rendering it seems.

Calinou · 2022-02-26T22:38:02Z

No, I mean Reflection probes act as a sort of multiplier for the Clustered elements they're rendering it seems.

This is likely because the reflection probes render the lights and decals, while other lights and decals won't render each other.

atirut-w · 2022-11-18T22:30:35Z

I am having a very similar issue with my project. I instanced a lot of lights at runtime according to GTA 3's item placement file and it crashes. Instancing non-lighting nodes instead fixes the crashes so I think it might be the same issue?

Calinou · 2022-11-19T02:05:39Z

I am having a very similar issue with my project. I instanced a lot of lights at runtime according to GTA 3's item placement file and it crashes. Instancing non-lighting nodes instead fixes the crashes so I think it might be the same issue?

This is indeed the same issue. Can you check if enabling Distance Fade on the lights (and ensuring the fade distance is low enough for some lights to be culled) fixes the crash?

atirut-w · 2022-11-19T05:44:41Z

Yup. Someone suggested that I disable out-of-range lights so I used distance fading(again, data from placement files) and it worked.

solek1312 · 2022-12-31T02:00:48Z

Yep, still there in Beta 10. Distance fading helps

Gamepro5 · 2023-05-22T20:03:35Z

Is it normal that I'm getting an errorless crash, and there isn't even anything logged using a developer build?

It crashes randomly after spawning a lot of decals.

poperigby · 2023-06-10T20:25:05Z

For me, my game crashes when I instantiate exactly 207 decals on a single model.

Calinou · 2023-06-10T22:23:54Z

Is it normal that I'm getting an errorless crash, and there isn't even anything logged using a developer build?

It crashes randomly after spawning a lot of decals.

Yes, as Godot currently doesn't have a GUI crash handler (until #61906 is merged).

For me, my game crashes when I instantiate exactly 207 decals on a single model.

This checks out with this quote from OP:

The highest amount of combined lights I can render successfully is 103 OmniLights + 103 SpotLights. Past that, I always get a crash when the project starts.

(103 + 103 resulting in 206 clustered items)

Calinou added bug topic:rendering crash topic:3d labels Jan 9, 2022

Calinou added this to the 4.0 milestone Jan 9, 2022

qarmin added the confirmed label Jan 10, 2022

This was referenced Jan 22, 2022

Decrease the default number of max clustered elements to 128 #56636

Closed

[TRACKER] Decal and OmniLight3D/SpotLight3D projector issues #57284

Open

Calinou changed the title ~~Vulkan: Crash in scene with 128 OmniLights and 128 SpotLights rendered at the same time~~ Vulkan: Crash in scene with 128 OmniLights and 128 SpotLights rendered at the same time (or Decals/ReflectionProbes) Sep 29, 2022

Calinou mentioned this issue Sep 29, 2022

[TRACKER] ReflectionProbe issues #66628

Open

Calinou mentioned this issue Oct 13, 2022

Decal crashes & culling behavior when placing at rapid pace #67353

Closed

clayjohn modified the milestones: 4.0, 4.x Feb 23, 2023

Calinou mentioned this issue Apr 28, 2023

[Tracker] Vulkan error unable to create swapchain (device lost) happening randomly #71929

Open

AThousandShips mentioned this issue May 22, 2023

678 OmniLight3D in a scene causes crash to desktop. #77372

Closed

bitsawer mentioned this issue Aug 21, 2023

Fix Vulkan crash with many Omni/SpotLights, Decals or ReflectionProbes #80845

Merged

akien-mga modified the milestones: 4.x, 4.2 Aug 21, 2023

akien-mga closed this as completed in #80845 Aug 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Vulkan: Crash in scene with 128 OmniLights and 128 SpotLights rendered at the same time (or Decals/ReflectionProbes) #56657

Vulkan: Crash in scene with 128 OmniLights and 128 SpotLights rendered at the same time (or Decals/ReflectionProbes) #56657

Calinou commented Jan 9, 2022 •

edited

Loading

qarmin commented Jan 10, 2022

qarmin commented Jan 10, 2022 •

edited

Loading

Calinou commented Jan 22, 2022

mrjustaguy commented Feb 26, 2022

Calinou commented Feb 26, 2022

mrjustaguy commented Feb 26, 2022

Calinou commented Feb 26, 2022

atirut-w commented Nov 18, 2022

Calinou commented Nov 19, 2022 •

edited

Loading

atirut-w commented Nov 19, 2022

solek1312 commented Dec 31, 2022

Gamepro5 commented May 22, 2023

poperigby commented Jun 10, 2023

Calinou commented Jun 10, 2023 •

edited

Loading

Vulkan: Crash in scene with 128 OmniLights and 128 SpotLights rendered at the same time (or Decals/ReflectionProbes) #56657

Vulkan: Crash in scene with 128 OmniLights and 128 SpotLights rendered at the same time (or Decals/ReflectionProbes) #56657

Comments

Calinou commented Jan 9, 2022 • edited Loading

Godot version

System information

Issue description

Steps to reproduce

Minimal reproduction project

qarmin commented Jan 10, 2022

qarmin commented Jan 10, 2022 • edited Loading

Calinou commented Jan 22, 2022

mrjustaguy commented Feb 26, 2022

Calinou commented Feb 26, 2022

mrjustaguy commented Feb 26, 2022

Calinou commented Feb 26, 2022

atirut-w commented Nov 18, 2022

Calinou commented Nov 19, 2022 • edited Loading

atirut-w commented Nov 19, 2022

solek1312 commented Dec 31, 2022

Gamepro5 commented May 22, 2023

poperigby commented Jun 10, 2023

Calinou commented Jun 10, 2023 • edited Loading

Calinou commented Jan 9, 2022 •

edited

Loading

qarmin commented Jan 10, 2022 •

edited

Loading

Calinou commented Nov 19, 2022 •

edited

Loading

Calinou commented Jun 10, 2023 •

edited

Loading