Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add morph targets #8158

Merged
merged 39 commits into from
Jun 22, 2023
Merged

Add morph targets #8158

merged 39 commits into from
Jun 22, 2023

Conversation

nicopap
Copy link
Contributor

@nicopap nicopap commented Mar 22, 2023

Objective

Morph targets (also known as shape interpolation, shape keys, or blend shapes) allow animating individual vertices with fine grained controls. This is typically used for facial expressions. By specifying multiple poses as vertex offset, and providing a set of weight of each pose, it is possible to define surprisingly realistic transitions between poses. Blending between multiple poses also allow composition. Morph targets are part of the gltf standard and are a feature of Unity and Unreal, and babylone.js, it is only natural to implement them in bevy.

Solution

This implementation of morph targets uses a 3d texture where each pixel is a component of an animated attribute. Each layer is a different target. We use a 2d texture for each target, because the number of attribute×components×animated vertices is expected to always exceed the maximum pixel row size limit of webGL2. It copies fairly closely the way skinning is implemented on the CPU side, while on the GPU side, the shader morph target implementation is a relatively trivial detail.

We add an optional morph_texture to the Mesh struct. The morph_texture is built through a method that accepts an iterator over attribute buffers.

The MorphWeights component, user-accessible, controls the blend of poses used by mesh instances (so that multiple copy of the same mesh may have different weights), all the weights are uploaded to a uniform buffer of 256 f32. We limit to 16 poses per mesh, and a total of 256 poses.

More literature:

bevy_morph_targets-2023-04-11.mp4
bevy_morph_targets-2023-05-10.mp4

Acknowledgements

  • Thanks to storytold for sponsoring the feature
  • Thanks to superdump and james7132 for guidance and help figuring out stuff

Future work

  • Handling of less and more attributes (eg: animated uv, animated arbitrary attributes)
  • Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses)
  • Better animation API, see AnimationPlayer can play unrelated AnimationClips  #8357

Changelog

  • Add morph targets to bevy meshes
    • Support up to 64 poses per mesh of individually up to 116508 vertices, animation currently strictly limited to the position, normal and tangent attributes.
    • Load a morph target using Mesh::set_morph_targets
  • Add VisitMorphTargets and VisitMorphAttributes traits to bevy_render, this allows defining morph targets (a fairly complex and nested data structure) through iterators (ie: single copy instead of passing around buffers), see documentation of those traits for details
  • Add MorphWeights component exported by bevy_render
    • MorphWeights control mesh's morph target weights, blending between various poses defined as morph targets.
    • MorphWeights are directly inherited by direct children (single level of hierarchy) of an entity. This allows controlling several mesh primitives through a unique entity as per GLTF spec.
  • Add MorphTargetNames component, naming each indices of loaded morph targets.
  • Load morph targets weights and buffers in bevy_gltf
  • handle morph targets animations in bevy_animation (previously, it was a warn! log)
  • Add the MorphStressTest.gltf asset for morph targets testing, taken from the glTF samples repo, CC0.
  • Add morph target manipulation to scene_viewer
  • Separate the animation code in scene_viewer from the rest of the code, reducing #[cfg(feature)] noise
  • Add the morph_targets.rs example to show off how to manipulate morph targets, loading MorpStressTest.gltf

Migration Guide

  • (very specialized, unlikely to be touched by 3rd parties)
    • MeshPipeline now has a single mesh_layouts field rather than separate mesh_layout and skinned_mesh_layout fields. You should handle all possible mesh bind group layouts in your implementation
    • You should also handle properly the new MORPH_TARGETS shader def and mesh pipeline key. A new function is exposed to make this easier: setup_moprh_and_skinning_defs
    • The MeshBindGroup is now MeshBindGroups, cached bind groups are now accessed through the get method.

@nicopap nicopap added C-Feature A new feature, making something new possible A-Rendering Drawing game state to the screen A-Animation Make things move and change over time D-Complex Quite challenging from either a design or technical perspective. Ask for help! labels Mar 22, 2023
@james7132 james7132 modified the milestones: 0.10.1, 0.11 Mar 23, 2023
@nicopap nicopap force-pushed the morph-target branch 2 times, most recently from d657f73 to 64b5322 Compare April 10, 2023 16:39
@nicopap
Copy link
Contributor Author

nicopap commented Apr 12, 2023

Todo when merged: open issue detailing how paused AnimationPlayers overwrite user-set weights.

@nicopap nicopap marked this pull request as ready for review April 12, 2023 08:35
@nicopap
Copy link
Contributor Author

nicopap commented Apr 12, 2023

Ready for review. @james7132 and @willstott101 might be interested in this.

Comment on lines 38 to 56
fn morph_vertex(vertex: Vertex) -> Vertex {
var vertex = vertex;
let weight_count = layer_count();
for (var i: u32 = 0u; i < weight_count; i ++) {
let weight = weight_at(i);
if weight == 0.0 {
continue;
}
vertex.position += weight * morph(vertex.index, position_offset, i);
#ifdef VERTEX_NORMALS
vertex.normal += weight * morph(vertex.index, normal_offset, i);
#endif
#ifdef VERTEX_TANGENTS
vertex.tangent += vec4(weight * morph(vertex.index, tangent_offset, i), 0.0);
#endif
}
return vertex;
}
#endif
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that this is duplicated in prepass.wgsl because I didn't want to move the struct Vertex definition to mesh_types.wgsl (which probably should be the place where Vertex is defined)

Comment on lines 215 to 200
morph_nodes: Query<(&Children, &MorphWeights), (Without<Handle<Mesh>>, Changed<MorphWeights>)>,
mut morph_primitives: Query<&mut MorphWeights, With<Handle<Mesh>>>,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suspect splitting this in two different components is preferable. Otherwise it might become a bit awkward for the user to Query MorphWeights for specific mesh (as glTF mesh) or mesh (as bevy mesh)

@mockersf
Copy link
Member

could you add an example dedicated to morphing, out of the scene viewer?

@nicopap
Copy link
Contributor Author

nicopap commented Apr 13, 2023

@mockersf I can add a standalone example. Additionally, do you have any tips for avoiding code duplication between the example and the scene viewer?

@mockersf
Copy link
Member

Additionally, do you have any tips for avoiding code duplication between the example and the scene viewer?

No, example scene viewer is duplicating code from other examples... If something is useful enough you can put it in bevy_animation

@cart
Copy link
Member

cart commented Jun 20, 2023

Just pushed another "proposal commit": I moved morph target names to Mesh, which resolves my comment here: #8158 (comment)

examples/animation/morph_targets.rs Outdated Show resolved Hide resolved
crates/bevy_render/src/mesh/morph.rs Show resolved Hide resolved
@cart
Copy link
Member

cart commented Jun 20, 2023

My latest change broke MorphViewerPlugin, but I'd like to discuss #8158 (comment) before fixing it because the code will be informed by that choice.

@nicopap
Copy link
Contributor Author

nicopap commented Jun 20, 2023

I think morph target names are orthogonal to Mesh. It is not ideal to store them in the Mesh asset. Using a Arc<str> in a MorphTargetNames component seems to me a good way to avoid too many allocations.

@cart
Copy link
Member

cart commented Jun 20, 2023

I think morph target names are orthogonal to Mesh. It is not ideal to store them in the Mesh asset. Using a Arc in a MorphTargetNames component seems to me a good way to avoid too many allocations.

I don't follow this logic. Can you explain how it is orthogonal? The Mesh is the canonical owner of the actual morph target data (by effectively owning the morph target image with the morph target data packed into it). Storing additional data that describes the morph targets (their names) makes perfect sense to me. The Mesh is "the thing" that is being morphed by morph targets. The GLTF data model nests the targetNames field right inside of the GltfMesh type. Why shouldn't our Mesh type own the names of the targets?

We do the same thing for "mesh vertex attribute data" by correlating their "mesh vertex attribute names". This feels conceptually almost identical.

@nicopap
Copy link
Contributor Author

nicopap commented Jun 20, 2023

Hmm. Your reasoning is sensible. After all, the morph target image is stored on the Mesh.

Looks good.

@nicopap
Copy link
Contributor Author

nicopap commented Jun 20, 2023

You can use the old test model here to try your changes. It has several meshes with morph targets, including Morph-Primitive Test:

It has more edge cases than MorphStressTest.gltf. See reasoning as to why it was replaced: #8158 (comment)

@cart
Copy link
Member

cart commented Jun 21, 2023

I just removed the "nested propagated MorphWeights" in favor of duplicating curves within a clip. I think it keeps the mental model simpler by being truer to the current Bevy Mesh api. I also simplified the morph_targets example by making it play a specific clip (which is what actually games are going to do generally).

@nicopap
Copy link
Contributor Author

nicopap commented Jun 21, 2023

Sure. I can finish work on your design to speed things up.

@nicopap
Copy link
Contributor Author

nicopap commented Jun 21, 2023

I updated the scene_viewer/morph_viewer_plugin.rs to reflect the change to non-propagated morph weights.

Specifically pay attention to the add_targets function. Notice how as a user, I have to manually store the list of entities for a given instance of morph targets. Then use that list to iterate over all meshes related to a single instance of morph targets.

Manipulating individual morph targets programmatically, ie: not through a loaded glTF animation. Is a common usecase (dark souls character editor is an example). Similarly to how one would programmatically move bone transforms in a skinned mesh to handle IKs and ragdolls.

Try the scene_viewer with the Morph-Primitive-Test.glb I linked earlier. You'll notice the difference between managing morph weights across primitives of a single mesh and not managing them.

It's possible to write correct code. But if the user doesn't know about the subtle difference between a bevy mesh and a glTF mesh, they are going to be surprised when they update a MorphWeights component and see that only the fleshy part of the face moves, while the other part of the face with a different material stay fixed.

Note that it's fairly poor as a out-of-the-box experience. But it's possible to write a 3rd party plugin that adds a ManagedMorphWeights component that does what I've been describing.

@cart
Copy link
Member

cart commented Jun 21, 2023

Hmmm ok I do see your point. I've also played around with morph targets a bit in Godot (which has the sync behavior). There are definitely arguments in favor of both approaches, but your "hierarchical sync" approach does have the benefits of "less user code for manual morph targets" and "more aligned with GLTF and Godot user expectations for morph target behavior". I guess either of those should outweigh "conceptual purity / Bevy alignment". This is especially true in light of the fact that Godot, Unity, and Unreal all have "mesh surface / primitives" concepts. Which means we will likely also want them, just to accommodate existing workflows. And then we'd have "bevy aligned / conceptual purity" and "behavior aligned with user expectations".

In short: you were right and sorry for the back and forth!

I'll revert things back to hierarchical sync (while keeping the other changes I made). I'll also do the "MorphWeights component split" to resolve some of the UX concerns with the hierarchy approach.

@JMS55
Copy link
Contributor

JMS55 commented Jun 21, 2023

Small non-blocking consideration: in order to write proper motion vectors, morph targets will need to be able to calculate their vertex position last frame during the current frame, which will involve knowing last frame's weights.

@cart
Copy link
Member

cart commented Jun 21, 2023

I pushed the changes mentioned above. I also added MorphWeights::first_mesh() -> Option<&Handle<Mesh>> to make it easier to look up morph metadata stored in Mesh (such as the morph names). Still a bit of a compromise as we don't have a central "multi-surface Mesh" equivalent, but I still think this is preferable to storing names on individual entities.

@cart cart added this pull request to the merge queue Jun 22, 2023
Merged via the queue into bevyengine:main with commit c6170d4 Jun 22, 2023
@Selene-Amanita Selene-Amanita added the M-Needs-Migration-Guide A breaking change to Bevy's public API that needs to be noted in a migration guide label Jul 10, 2023
@nicopap nicopap mentioned this pull request Aug 3, 2023
@nicopap nicopap deleted the morph-target branch August 30, 2023 13:46
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Animation Make things move and change over time A-Rendering Drawing game state to the screen C-Feature A new feature, making something new possible M-Needs-Migration-Guide A breaking change to Bevy's public API that needs to be noted in a migration guide
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Cannot play animation correctly. GLTF Morph Targets
7 participants