Godot crash while generating the NavigationMeshInstance

[Nutriz]

Godot version:
3.2.1 stable
OS/device including version:
Ubuntu Budgie 19.10
Issue description:
Godot crash while generating the NavigationMeshInstance, when 0 is set for NavigationMeshInstance/Detail/Sample Distance
Steps to reproduce:

• Add Navigation node
• Add NavigationMeshInstance node as child of Navigation
• In NavigationMeshInstance node, add a NavigationMesh
• Inside NavigationMesh, go to Detail section and drag Sample Distance to 0
• Add a MeshInstance with a cube (or whatever)
• Select NavigationMeshInstance node and click Bake NavMesh

~/Bureau handle_crash: Program crashed with signal 11                                                        13:36:44
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] /lib/x86_64-linux-gnu/libc.so.6(+0x46470) [0x7f857efe2470] (??:0)
[2] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x241eb6a] (??:?)
[3] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x8c55d9] (<artificial>:?)
[4] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x91b67b] (??:?)
[5] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x25f97d6] (<artificial>:?)
[6] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x25f9a75] (<artificial>:?)
[7] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x2898209] (??:?)
[8] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0xbe6033] (??:?)
[9] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0xc8c576] (??:?)
[10] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0xc8d570] (??:?)
[11] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x184b4a5] (<artificial>:?)
[12] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x184b9ba] (<artificial>:?)
[13] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x184bd41] (??:?)
[14] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x1f6be86] (??:?)
[15] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0xc01314] (??:?)
[16] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0xbce1c1] (??:?)
[17] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x18c8488] (<artificial>:?)
[18] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x18eaf7f] (<artificial>:?)
[19] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x18ec066] (??:?)
[20] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x18ec1be] (<artificial>:?)
[21] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x2725c26] (??:?)
[22] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0xbe6033] (??:?)
[23] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0xbce245] (??:?)
[24] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x1883923] (??:?)
[25] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x1886a4c] (??:?)
[26] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x28cd843] (??:?)
[27] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x28caaad] (<artificial>:?)
[28] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x2a0c4ab] (<artificial>:?)
[29] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x84ac3c] (??:?)
[30] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7f857efc31e3] (??:0)
[31] /home/nutriz/Bureau/Godot_v3.2.1-stable_x11.64() [0x857fde] (??:?)
-- END OF BACKTRACE --

Minimal reproduction project:
NavMeshCrash.zip

Open the project, select NavigationMeshInstance node and click Bake NavMesh, it will crash

[Calinou]

I guess we should have a non-zero minimum value for Sample Distance. What would be a good minimum value? Something like 0.1 or 1?

[Nutriz]

Default is 6, and actual range go from 0 to 16 (float). I don't exactly know what this value is for.

The class is undocumented, maybe I will try to contribute and write it.

[Calinou]

Keep in mind the navigation system was rewritten in the master branch, so you'll have to open your pull request against the 3.2 branch.

[Nutriz]

Chapter 2 🕵️‍♂️

While investigation for writing the NavigationMesh doc, I have found that a lib called Recast is used to generate it.

There is some doc inside code:

godot/thirdparty/recastnavigation/Recast/Include/Recast.h

Lines 256 to 258 in 7899b3e

	/// Sets the sampling distance to use when generating the detail mesh.
	/// (For height detail only.) [Limits: 0 or >= 0.9] [Units: wu]
	float detailSampleDist;

and the value is used like that:

godot/modules/gdnavigation/navigation_mesh_generator.cpp

Line 349 in 7899b3e

cfg.detailSampleDist = p_nav_mesh->get_detail_sample_distance() < 0.9f ? 0 : p_nav_mesh->get_cell_size() * p_nav_mesh->get_detail_sample_distance();

After testing again, it crashes not only when the Sample Distance is set to 0, but also if it is set betweens 0 and 0.89. I don't know exactly why it crashes when set to 0 (infinite loop ?).

So I propose to set minimum value to 1 to avoid crashes. Maybe I will try to make a PR.

Also, I have a better crash log with build from source code:

handle_crash: Program crashed with signal 11
Dumping the backtrace. Please include this when reporting the bug on https://github.com/godotengine/godot/issues
[1] /lib/x86_64-linux-gnu/libc.so.6(+0x46470) [0x7fc890bd2470] (??:0)
[2] vdistSq2(float const*, float const*) (/media/nutriz/App/Dev/Projects/godot_src/thirdparty/recastnavigation/Recast/Source/RecastMeshDetail.cpp:48)
[3] vdist2(float const*, float const*) (/media/nutriz/App/Dev/Projects/godot_src/thirdparty/recastnavigation/Recast/Source/RecastMeshDetail.cpp:55)
[4] /media/nutriz/App/Dev/Projects/godot_src/bin/godot.x11.tools.64() [0x1ceec25] (/media/nutriz/App/Dev/Projects/godot_src/thirdparty/recastnavigation/Recast/Source/RecastMeshDetail.cpp:604)
[5] /media/nutriz/App/Dev/Projects/godot_src/bin/godot.x11.tools.64() [0x1cef777] (/media/nutriz/App/Dev/Projects/godot_src/thirdparty/recastnavigation/Recast/Source/RecastMeshDetail.cpp:783)
[6] rcBuildPolyMeshDetail(rcContext*, rcPolyMesh const&, rcCompactHeightfield const&, float, float, rcPolyMeshDetail&) (/media/nutriz/App/Dev/Projects/godot_src/thirdparty/recastnavigation/Recast/Source/RecastMeshDetail.cpp:1300)
[7] EditorNavigationMeshGenerator::_build_recast_navigation_mesh(Ref<NavigationMesh>, EditorProgress*, rcHeightfield*, rcCompactHeightfield*, rcContourSet*, rcPolyMesh*, rcPolyMeshDetail*, Vector<float>&, Vector<int>&) (/media/nutriz/App/Dev/Projects/godot_src/modules/recast/navigation_mesh_generator.cpp:406)
[8] EditorNavigationMeshGenerator::bake(Ref<NavigationMesh>, Node*) (/media/nutriz/App/Dev/Projects/godot_src/modules/recast/navigation_mesh_generator.cpp:473 (discriminator 1))
[9] NavigationMeshEditor::_bake_pressed() (/media/nutriz/App/Dev/Projects/godot_src/modules/recast/navigation_mesh_editor_plugin.cpp:68 (discriminator 4))
[10] MethodBind0::call(Object*, Variant const**, int, Variant::CallError&) (/media/nutriz/App/Dev/Projects/godot_src/./core/method_bind.gen.inc:61 (discriminator 4))
[11] Object::call(StringName const&, Variant const**, int, Variant::CallError&) (/media/nutriz/App/Dev/Projects/godot_src/core/object.cpp:922 (discriminator 1))
[12] Object::emit_signal(StringName const&, Variant const**, int) (/media/nutriz/App/Dev/Projects/godot_src/core/object.cpp:1249 (discriminator 1))
[13] Object::emit_signal(StringName const&, Variant const&, Variant const&, Variant const&, Variant const&, Variant const&) (/media/nutriz/App/Dev/Projects/godot_src/core/object.cpp:1307)
[14] BaseButton::_pressed() (/media/nutriz/App/Dev/Projects/godot_src/scene/gui/base_button.cpp:135 (discriminator 2))
[15] BaseButton::on_action_event(Ref<InputEvent>) (/media/nutriz/App/Dev/Projects/godot_src/scene/gui/base_button.cpp:165)
[16] BaseButton::_gui_input(Ref<InputEvent>) (/media/nutriz/App/Dev/Projects/godot_src/scene/gui/base_button.cpp:64 (discriminator 3))
[17] MethodBind1<Ref<InputEvent> >::call(Object*, Variant const**, int, Variant::CallError&) (/media/nutriz/App/Dev/Projects/godot_src/./core/method_bind.gen.inc:775 (discriminator 12))
[18] Object::call_multilevel(StringName const&, Variant const**, int) (/media/nutriz/App/Dev/Projects/godot_src/core/object.cpp:764 (discriminator 1))
[19] Object::call_multilevel(StringName const&, Variant const&, Variant const&, Variant const&, Variant const&, Variant const&) (/media/nutriz/App/Dev/Projects/godot_src/core/object.cpp:865)
[20] Viewport::_gui_call_input(Control*, Ref<InputEvent> const&) (/media/nutriz/App/Dev/Projects/godot_src/scene/main/viewport.cpp:1669 (discriminator 2))
[21] Viewport::_gui_input_event(Ref<InputEvent>) (/media/nutriz/App/Dev/Projects/godot_src/scene/main/viewport.cpp:2049 (discriminator 3))
[22] Viewport::input(Ref<InputEvent> const&) (/media/nutriz/App/Dev/Projects/godot_src/scene/main/viewport.cpp:2825 (discriminator 2))
[23] Viewport::_vp_input(Ref<InputEvent> const&) (/media/nutriz/App/Dev/Projects/godot_src/scene/main/viewport.cpp:1445)
[24] MethodBind1<Ref<InputEvent> const&>::call(Object*, Variant const**, int, Variant::CallError&) (/media/nutriz/App/Dev/Projects/godot_src/./core/method_bind.gen.inc:775 (discriminator 12))
[25] Object::call(StringName const&, Variant const**, int, Variant::CallError&) (/media/nutriz/App/Dev/Projects/godot_src/core/object.cpp:922 (discriminator 1))
[26] Object::call(StringName const&, Variant const&, Variant const&, Variant const&, Variant const&, Variant const&) (/media/nutriz/App/Dev/Projects/godot_src/core/object.cpp:849)
[27] SceneTree::call_group_flags(unsigned int, StringName const&, StringName const&, Variant const&, Variant const&, Variant const&, Variant const&, Variant const&) (/media/nutriz/App/Dev/Projects/godot_src/scene/main/scene_tree.cpp:275)
[28] SceneTree::input_event(Ref<InputEvent> const&) (/media/nutriz/App/Dev/Projects/godot_src/scene/main/scene_tree.cpp:431 (discriminator 6))
[29] InputDefault::_parse_input_event_impl(Ref<InputEvent> const&, bool) (/media/nutriz/App/Dev/Projects/godot_src/main/input_default.cpp:416)
[30] InputDefault::parse_input_event(Ref<InputEvent> const&) (/media/nutriz/App/Dev/Projects/godot_src/main/input_default.cpp:260)
[31] InputDefault::flush_accumulated_events() (/media/nutriz/App/Dev/Projects/godot_src/main/input_default.cpp:679)
[32] OS_X11::process_xevents() (/media/nutriz/App/Dev/Projects/godot_src/platform/x11/os_x11.cpp:2676)
[33] OS_X11::run() (/media/nutriz/App/Dev/Projects/godot_src/platform/x11/os_x11.cpp:3241)
[34] /media/nutriz/App/Dev/Projects/godot_src/bin/godot.x11.tools.64(main+0x125) [0x13ffb4b] (/media/nutriz/App/Dev/Projects/godot_src/platform/x11/godot_x11.cpp:57)
[35] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf3) [0x7fc890bb31e3] (??:0)
[36] /media/nutriz/App/Dev/Projects/godot_src/bin/godot.x11.tools.64(_start+0x2e) [0x13ff96e] (??:?)
-- END OF BACKTRACE --

[GuidoRevers]

On a related note, setting the size of the MeshInstance cube to x: 0, y: 0 and z: 0, while leaving NavigationMeshInstance/Detail/Sample Distance at its default 6, will also crash the baking.

Recast doesn't seem to like it when you throw not so sane values at it.