Bevy Panics when trying to load a single mesh from gltf file containing armature

[PatrickChodowski]

Bevy version

0.15 (happened on 0.14 too)

Relevant system information

os: "Linux 22.04 Ubuntu"
kernel: "6.8.0-49-generic"
cpu: "Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz"
core_count: "4"
memory: "15.5 GiB"
AdapterInfo { name: "NVIDIA GeForce GTX 1050", vendor: 4318, device: 7309, device_type: DiscreteGpu, driver: "NVIDIA", driver_info: "535.183.01", backend: Vulkan}

What you did

Tried to use specific mesh/primitive from Gltf file, that also contains armature inside

What went wrong

Received wgpu error:

2024-12-22T13:38:16.509152Z ERROR wgpu::backend::wgpu_core: Handling wgpu errors as fatal by default    
thread 'Compute Task Pool (2)' panicked at /home/patrick/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wgpu-23.0.1/src/backend/wgpu_core.rs:2976:18:
wgpu error: Validation Error

Caused by:
  In RenderPass::end
    In a draw command, kind: Draw
      The BindGroupLayout with 'mesh_layout' label of current set BindGroup with 'model_only_mesh_bind_group' label at index 1 is not compatible with the corresponding BindGroupLayout with 'skinned_mesh_layout' label of RenderPipeline with 'pbr_opaque_mesh_pipeline' label
        Expected entry with binding 1 not found in assigned bind group layout


note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Encountered a panic in system `bevy_render::renderer::render_system`!
thread 'main' panicked at /home/patrick/.cargo/registry/src/index.crates.io-6f17d22bba15001f/wgpu-hal-23.0.1/src/vulkan/instance.rs:173:58:
Trying to destroy a SurfaceSemaphores that is still in use by a SurfaceTexture

Additional information

Minimal reproduction app (you can comment and uncomment parts to test)

It contains 2 assets: They contain same mesh and material but one of the files also contains armature. File without armature loads correctly inside bevy, but the one with armature loads only as a scene. I dont like using scenes much, so I tried extracting mesh and material manually but it causes game to crash

Similar to #10810.

[CrazyRoka]

Hi @PatrickChodowski, @BenjaminBrienen, @mockersf

I've been investigating this issue, and I believe I've identified the root cause of the wgpu crash when loading individual meshes from GLTF files containing skinning data.

Problem Summary

Bevy crashes with a wgpu validation error when loading an individual mesh from a GLTF that includes an armature/skin. This error indicates an incompatibility between the bind group layout used for rendering (mesh_layout) and the bind group layout expected by the pipeline (skinned_mesh_layout).

Caused by:
  In RenderPass::end
    In a draw command, kind: Draw
      The BindGroupLayout with 'mesh_layout' label of current set BindGroup with 'model_only_mesh_bind_group' label at index 1 is not compatible with the corresponding BindGroupLayout with 'skinned_mesh_layout' label of RenderPipeline with 'pbr_opaque_mesh_pipeline' label
        Expected entry with binding 1 not found in assigned bind group layout

Root Cause Analysis

My investigation (including debugging with gdb and analyzing the stack trace) revealed the following:

• Prepass pipeline marks this mesh as skinned: The setup_morph_and_skinning_defs function correctly determines that the mesh is skinned. This is because is_skinned only checks for the presence of JOINTS_0 and WEIGHTS_0 vertex attributes in the Mesh asset, which are present. Prepass decides to use skinned_mesh_layout. Code

• Render function believes this mesh is non skinned: The core issue is that the render function can't find skin associated with the mesh. skin_indices map is always empty. Render pass decides to use mesh_layout. Code

• Skin indices are not correctly populated: extract_skins function is supposed to properly populate skin indices, however the main loop is never running. That's because SkinnedMesh component is never added to the spawned entity. Code

• This issue only occurs when loading individual meshes. When loading and spawning an entire scene, the SkinnedMesh component is correctly added during scene processing, and extract_skins populates skin_indices as expected. Code. That is why some examples work, while others manually spawn SkinnedMesh component Code.

Proposed Solution

I believe the most robust solution is to modify the bevy_gltf loader to add the SkinnedMesh component to entities even when loading individual meshes, if those meshes are associated with skinning data in the original GLTF.

Challenges and Request for Guidance

I'm not deeply familiar with Bevy's asset loading system, particularly how to properly create and add SkinnedMesh component when loading individual meshes using AssetServer::load.

I would greatly appreciate guidance from the maintainers on the following:

• Is the proposed solution of modifying bevy_gltf to add SkinnedMesh components during individual mesh loading, when skinning data is present, the correct and most maintainable approach to fix this issue?
• How to return multiple components as part of asset loading?
• Will it impact existing applications and potentially crash them or fail to compile?
• Are there any performance implications or potential pitfalls to consider when adding SkinnedMesh in this way?

I'm happy to contribute to implementing the fix once we have a clear path forward. Thank you for your time and assistance!

References

• Older issue: bevy panics when spawning GLTF Mesh asset without associated bones/armature #10810
• Related PR: GLTF loader: handle warning NODE_SKINNED_MESH_WITHOUT_SKIN #9360

[greeble-dev]

@CrazyRoka, I had a poke around this and think there's an alternative solution. But I'm not familiar with the renderer so I'd wait for a second opinion before believing anything I say.

• I doubt that adding a SkinnedMesh component is the right solution.
  • This expects the world to have joint entities to drive the skinning, but there won't be any if the mesh is spawned directly.
• I'm guessing it's better to make setup_morph_and_skinning_defs treat the mesh as unskinned.
  • This might be similar to the morph target path that checks a pipeline key.

So maybe the solution is something like:

1. Add MeshPipelineKey::SKINNED and BaseMeshPipelineKey::SKINNED.
2. In bevy_render/src/mesh/mod.rs, prepare_asset, set the pipeline key if Res<SkinIndices> has a valid entry for that mesh's entity.

But I'm not sure how prepare_asset would get to SkinIndices, or if it runs before SkinIndices is filled.

[CrazyRoka]

@greeble-dev Thank you for taking a look at this!

I appreciate your suggestion to avoid adding SkinnedMesh and instead modify the pipeline key to treat the mesh as unskinned. I agree that adding SkinnedMesh might not be ideal when joint entities are not present in the world.

I tried implementing your proposed solution (adding MeshPipelineKey::SKINNED and BaseMeshPipelineKey::SKINNED, and setting the key in prepare_asset if SkinIndices has a valid entry), but I ran into a couple of challenges:

1. Dependency Issue with SkinIndices: The SkinIndices struct is defined in bevy_pbr, while prepare_asset for Mesh is in bevy_render. Due to the dependency structure (bevy_pbr depends on bevy_render, not the other way around), I cannot directly access SkinIndices from within prepare_asset.
   • Moving skin.rs (where SkinIndices is defined) might be an option, but it's a public module, and moving it could potentially break existing users who rely on SkinIndices. Also, it is used in pbr crate, so it would require significant changes.
2. Accessing Skinning Information for the Current Mesh: Even if I could access SkinIndices, I'm not sure how to determine if the currently processed mesh (within prepare_asset) is associated with a skin. The prepare_asset function only receives a &Mesh, &GPUImage and a &mut MeshVertexBufferLayouts as parameters. It doesn't have access to the Entity or any other context to identify the correct skin in SkinIndices. The existing API seems to rely on the "main entity," which is not available in this context.

While modifying the pipeline key to treat the mesh as unskinned could work, it feels like we're losing valuable skinning information that's present in the GLTF. Ideally, the loader should be able to handle individually loaded skinned meshes in a way that preserves this information, even if it's not used for animation in the traditional sense.

Do you have any alternative suggestions or further guidance on how to address these challenges? Is there a way to:

1. Access or pass skinning-related information to prepare_asset for Mesh assets, even when they are loaded individually?
2. Potentially restructure the loader/renderer interaction to make this kind of information more readily available?
3. Create a new bind group layout that will handle meshes with unused skinning data?

I'm still committed to finding a good solution for this issue and appreciate any further insights you can provide.

[greeble-dev]

@CrazyRoka, my bad, I should have realised that prepare_asset is the wrong approach. It's per mesh asset, but the "do we have valid joints" property is of course per mesh instance.

After looking some more I realised that RenderMeshInstanceFlags::HAS_PREVIOUS_SKIN is doing something similar to what I wanted. I hacked in a HAS_CURRENT_SKIN variant and that appears to work in a simple test case. Commit here: 8fb68d7. This feels a little fragile though as multiple functions have to modify the key - I could have easily missed one.

[greeble-dev]

While modifying the pipeline key to treat the mesh as unskinned could work, it feels like we're losing valuable skinning information that's present in the GLTF. Ideally, the loader should be able to handle individually loaded skinned meshes in a way that preserves this information, even if it's not used for animation in the traditional sense.

I'm not sure I understand this? The skinning data on the mesh is preserved. Or do you mean that the loader should be able to select a single skinned mesh and also create the joints? So the user has three options:

1. Spawn entire scene with all skinned meshes and joint entities.
2. Spawn one skinned mesh, render as unskinned.
3. New option: Spawn one skinned mesh and necessary joint entities.

[greeble-dev]

I have the fix working and cleaned up (main...greeble-dev:bevy:issue-16929-skinned-mesh-crash). But I'm struggling to keep it up to date with other changes to the renderer, so I'm going wait a week or two and see if things calm down before making a PR.