Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot play animation correctly. #6814

Closed
AllenDang opened this issue Dec 1, 2022 · 5 comments · Fixed by #8158
Closed

Cannot play animation correctly. #6814

AllenDang opened this issue Dec 1, 2022 · 5 comments · Fixed by #8158
Labels
A-Animation Make things move and change over time A-Assets Load files from disk to use for things like images, models, and sounds C-Bug An unexpected or incorrect behavior

Comments

@AllenDang
Copy link

Bevy version

0.9.1

What you did

View a model with animation from GLTF via scene_viewer

What went wrong

The animation is incorrect.

Here is the result from scene_viewer.

ScreenFlow.mp4

And here is the result from official GLTF viewer.

ScreenFlow_official.mp4

Here is the GLTF file.

dragon_flying.glb.zip

Additional information

Other information that can be used to further reproduce or isolate the problem.
This commonly includes:

  • screenshots
  • logs
  • theories about what might be going wrong
  • workarounds that you used
  • links to related bugs, PRs or discussions
@AllenDang AllenDang added C-Bug An unexpected or incorrect behavior S-Needs-Triage This issue needs to be labelled labels Dec 1, 2022
@james7132
Copy link
Member

A quick glance makes it seem like it's binding the animations to the wrong bones or the wrong mesh. This could be an issue with the way we're extracting the hierarchy or the animations from the GLTF file.

@james7132 james7132 added A-Assets Load files from disk to use for things like images, models, and sounds A-Animation Make things move and change over time and removed S-Needs-Triage This issue needs to be labelled labels Dec 1, 2022
@mockersf
Copy link
Member

mockersf commented Dec 1, 2022

it looks like the model is using morph animation which are not supported in Bevy yet

@nicopap
Copy link
Contributor

nicopap commented Apr 12, 2023

Can confirm it's because it's using morph targets. I tested it with #8158 and the dragon gloriously flaps. This requires support for 50 targets (#8158 supports 64), which would be unreasonable for a game asset, the dragon asset is about 50 times larger than if it was animated with skeletal animations. Caveat: the rider doesn't bob in sync with the dragon, I assume this is because they are two different animations you are supposed to run together.

nicopap added a commit to nicopap/bevy that referenced this issue Apr 17, 2023
Objective
---------

- Add morph targets to `bevy_pbr` (closes bevyengine#5756) & load them from glTF
- Supersedes bevyengine#3722
- Fixes bevyengine#6814

[Morph targets][1] (also known as shape interpolation, shape keys, or
blend shapes) allow animating individual vertices with fine grained
controls. This is typically used for facial expressions. By
specifying multiple poses as vertex offset, and providing a set of
weight of each pose, it is possible to define surprisingly realistic
transitions between poses. Blending between multiple poses also allow
composition. Morph targets are part of the [gltf standard][2] and are
a feature of Unity and Unreal, and babylone.js, it is only natural to
implement them in bevy.

Solution
--------

This implementation of morph targets uses a 3d storage texture where
each pixel is a component of an animated attribute. Each layer is a
different target. We use a 2d texture for each target, because the
number of attribute×components×animated vertices is expected to
always exceed the maximum pixel row size limit of webGL2. It copies
fairly closely the way skinning is implemented on the CPU side, while
on the GPU side, the shader morph target implementation is a
relatively trivial detail.

We add an optional `morph_texture` to the `Mesh` struct. The
`morph_texture` is built through a method that accepts an iterator
over attribute buffers.

The `MorphWeights` component, user-accessible, controls the blend of
poses used by mesh instances (so that multiple copy of the same mesh
may have different weights), all the weights are uploaded to a
uniform buffer of 256 `f32`. We limit to 16 poses per mesh, and a
total of 256 poses.

More literature:
* Old babylone.js implementation (vertex attribute-based): https://www.eternalcoding.com/dev-log-1-morph-targets/
* Babylone.js implementation (similar to ours): https://www.youtube.com/watch?v=LBPRmGgU0PE
* GPU gems 3: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits
* Development discord thread https://discord.com/channels/691052431525675048/1083325980615114772

https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4

Acknowledgements
---------------

* Thanks to @storytold for sponsoring the feature
* Thanks to @superdump  and @james7132 for guidance and help figuring out stuff

Future work
-----------

- Handling of less and more attributes (eg: animated uv, animated arbitrary attributes)
- Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses)
- Better animation API, see bevyengine#8357

----

Changelog
---------

- Add morph targets to bevy meshes
	- Support up to 64 poses per mesh of individually up to
          116508 vertices, animation currently strictly limited to the
          position, normal and tangent attributes.
	- Load a morph target using `Mesh::set_morph_targets`
- Add `VisitMorphTargets` and `VisitMorphAttributes` traits to
  `bevy_render`, this allows defining morph targets (a fairly complex
  and nested data structure) through iterators (ie: single copy instead
  of passing around buffers), see documentation of those traits for
  details
- Add `MorphWeights` component exported by `bevy_render`
	- `MorphWeights` control mesh's morph target weights,
          blending between various poses defined as morph targets.
	- `MorphWeights` are directly inherited by direct children
          (single level of hierarchy) of an entity. This allows
          controlling several mesh primitives through a unique entity
          _as per GLTF spec_.
- Add `GltfMeshExtras` component to query gltf extras specific to
  meshes of a given node
	- This allows reading morph weight names, see the new
          `scene_viewer` `morph_viewer_plugin.rs` module for details.
- Load morph targets weights and buffers in `bevy_gltf`
- handle morph targets animations in `bevy_animation` (previously, it
  was a `warn!` log)
- Add the `multiple_morph_target_meshes.gltf` asset for morph targets
  testing. Load it with the scene viewer.

Migration Guide
---------------

- (very specialized, unlikely to be touched by 3rd parties)
	- `MeshPipeline` now has a single `mesh_layouts` field rather
          than separate `mesh_layout` and `skinned_mesh_layout` fields.
          You should handle all possible mesh bind group layouts in
          your implementation.
	- You should also handle properly the new `MORPH_TARGETS`
          shader def and mesh pipeline key. A new function is exposed
          to make this easier: `set_mesh_binding_defs` .
	- The `MeshBindGroup` resource doesn't exist anymore. Mesh
          bind groups are computed in `queue_mesh_bind_group` system
          and are available as the field `GpuMesh::bind_group` .

[1]: https://en.wikipedia.org/wiki/Morph_target_animation
[2]: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets
nicopap added a commit to nicopap/bevy that referenced this issue Apr 17, 2023
Objective
---------

- Add morph targets to `bevy_pbr` (closes bevyengine#5756) & load them from glTF
- Supersedes bevyengine#3722
- Fixes bevyengine#6814

[Morph targets][1] (also known as shape interpolation, shape keys, or
blend shapes) allow animating individual vertices with fine grained
controls. This is typically used for facial expressions. By
specifying multiple poses as vertex offset, and providing a set of
weight of each pose, it is possible to define surprisingly realistic
transitions between poses. Blending between multiple poses also allow
composition. Morph targets are part of the [gltf standard][2] and are
a feature of Unity and Unreal, and babylone.js, it is only natural to
implement them in bevy.

Solution
--------

This implementation of morph targets uses a 3d storage texture where
each pixel is a component of an animated attribute. Each layer is a
different target. We use a 2d texture for each target, because the
number of attribute×components×animated vertices is expected to
always exceed the maximum pixel row size limit of webGL2. It copies
fairly closely the way skinning is implemented on the CPU side, while
on the GPU side, the shader morph target implementation is a
relatively trivial detail.

We add an optional `morph_texture` to the `Mesh` struct. The
`morph_texture` is built through a method that accepts an iterator
over attribute buffers.

The `MorphWeights` component, user-accessible, controls the blend of
poses used by mesh instances (so that multiple copy of the same mesh
may have different weights), all the weights are uploaded to a
uniform buffer of 256 `f32`. We limit to 16 poses per mesh, and a
total of 256 poses.

More literature:
* Old babylone.js implementation (vertex attribute-based): https://www.eternalcoding.com/dev-log-1-morph-targets/
* Babylone.js implementation (similar to ours): https://www.youtube.com/watch?v=LBPRmGgU0PE
* GPU gems 3: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits
* Development discord thread https://discord.com/channels/691052431525675048/1083325980615114772

https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4

Acknowledgements
---------------

* Thanks to @storytold for sponsoring the feature
* Thanks to @superdump  and @james7132 for guidance and help figuring out stuff

Future work
-----------

- Handling of less and more attributes (eg: animated uv, animated arbitrary attributes)
- Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses)
- Better animation API, see bevyengine#8357

----

Changelog
---------

- Add morph targets to bevy meshes
	- Support up to 64 poses per mesh of individually up to
          116508 vertices, animation currently strictly limited to the
          position, normal and tangent attributes.
	- Load a morph target using `Mesh::set_morph_targets`
- Add `VisitMorphTargets` and `VisitMorphAttributes` traits to
  `bevy_render`, this allows defining morph targets (a fairly complex
  and nested data structure) through iterators (ie: single copy instead
  of passing around buffers), see documentation of those traits for
  details
- Add `MorphWeights` component exported by `bevy_render`
	- `MorphWeights` control mesh's morph target weights,
          blending between various poses defined as morph targets.
	- `MorphWeights` are directly inherited by direct children
          (single level of hierarchy) of an entity. This allows
          controlling several mesh primitives through a unique entity
          _as per GLTF spec_.
- Add `GltfMeshExtras` component to query gltf extras specific to
  meshes of a given node
	- This allows reading morph weight names, see the new
          `scene_viewer` `morph_viewer_plugin.rs` module for details.
- Load morph targets weights and buffers in `bevy_gltf`
- handle morph targets animations in `bevy_animation` (previously, it
  was a `warn!` log)
- Add the `multiple_morph_target_meshes.gltf` asset for morph targets
  testing. Load it with the scene viewer.

Migration Guide
---------------

- (very specialized, unlikely to be touched by 3rd parties)
	- `MeshPipeline` now has a single `mesh_layouts` field rather
          than separate `mesh_layout` and `skinned_mesh_layout` fields.
          You should handle all possible mesh bind group layouts in
          your implementation.
	- You should also handle properly the new `MORPH_TARGETS`
          shader def and mesh pipeline key. A new function is exposed
          to make this easier: `set_mesh_binding_defs` .
	- The `MeshBindGroup` resource doesn't exist anymore. Mesh
          bind groups are computed in `queue_mesh_bind_group` system
          and are available as the field `GpuMesh::bind_group` .

[1]: https://en.wikipedia.org/wiki/Morph_target_animation
[2]: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets
nicopap added a commit to nicopap/bevy that referenced this issue Apr 18, 2023
Objective
---------

- Add morph targets to `bevy_pbr` (closes bevyengine#5756) & load them from glTF
- Supersedes bevyengine#3722
- Fixes bevyengine#6814

[Morph targets][1] (also known as shape interpolation, shape keys, or
blend shapes) allow animating individual vertices with fine grained
controls. This is typically used for facial expressions. By
specifying multiple poses as vertex offset, and providing a set of
weight of each pose, it is possible to define surprisingly realistic
transitions between poses. Blending between multiple poses also allow
composition. Morph targets are part of the [gltf standard][2] and are
a feature of Unity and Unreal, and babylone.js, it is only natural to
implement them in bevy.

Solution
--------

This implementation of morph targets uses a 3d storage texture where
each pixel is a component of an animated attribute. Each layer is a
different target. We use a 2d texture for each target, because the
number of attribute×components×animated vertices is expected to
always exceed the maximum pixel row size limit of webGL2. It copies
fairly closely the way skinning is implemented on the CPU side, while
on the GPU side, the shader morph target implementation is a
relatively trivial detail.

We add an optional `morph_texture` to the `Mesh` struct. The
`morph_texture` is built through a method that accepts an iterator
over attribute buffers.

The `MorphWeights` component, user-accessible, controls the blend of
poses used by mesh instances (so that multiple copy of the same mesh
may have different weights), all the weights are uploaded to a
uniform buffer of 256 `f32`. We limit to 16 poses per mesh, and a
total of 256 poses.

More literature:
* Old babylone.js implementation (vertex attribute-based): https://www.eternalcoding.com/dev-log-1-morph-targets/
* Babylone.js implementation (similar to ours): https://www.youtube.com/watch?v=LBPRmGgU0PE
* GPU gems 3: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits
* Development discord thread https://discord.com/channels/691052431525675048/1083325980615114772

https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4

Acknowledgements
---------------

* Thanks to @storytold for sponsoring the feature
* Thanks to @superdump  and @james7132 for guidance and help figuring out stuff

Future work
-----------

- Handling of less and more attributes (eg: animated uv, animated arbitrary attributes)
- Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses)
- Better animation API, see bevyengine#8357

----

Changelog
---------

- Add morph targets to bevy meshes
	- Support up to 64 poses per mesh of individually up to
          116508 vertices, animation currently strictly limited to the
          position, normal and tangent attributes.
	- Load a morph target using `Mesh::set_morph_targets`
- Add `VisitMorphTargets` and `VisitMorphAttributes` traits to
  `bevy_render`, this allows defining morph targets (a fairly complex
  and nested data structure) through iterators (ie: single copy instead
  of passing around buffers), see documentation of those traits for
  details
- Add `MorphWeights` component exported by `bevy_render`
	- `MorphWeights` control mesh's morph target weights,
          blending between various poses defined as morph targets.
	- `MorphWeights` are directly inherited by direct children
          (single level of hierarchy) of an entity. This allows
          controlling several mesh primitives through a unique entity
          _as per GLTF spec_.
- Add `GltfMeshExtras` component to query gltf extras specific to
  meshes of a given node
	- This allows reading morph weight names, see the new
          `scene_viewer` `morph_viewer_plugin.rs` module for details.
- Load morph targets weights and buffers in `bevy_gltf`
- handle morph targets animations in `bevy_animation` (previously, it
  was a `warn!` log)
- Add the `multiple_morph_target_meshes.gltf` asset for morph targets
  testing. Load it with the scene viewer.

Migration Guide
---------------

- (very specialized, unlikely to be touched by 3rd parties)
	- `MeshPipeline` now has a single `mesh_layouts` field rather
          than separate `mesh_layout` and `skinned_mesh_layout` fields.
          You should handle all possible mesh bind group layouts in
          your implementation.
	- You should also handle properly the new `MORPH_TARGETS`
          shader def and mesh pipeline key. A new function is exposed
          to make this easier: `set_mesh_binding_defs` .
	- The `MeshBindGroup` resource doesn't exist anymore. Mesh
          bind groups are computed in `queue_mesh_bind_group` system
          and are available as the field `GpuMesh::bind_group` .

[1]: https://en.wikipedia.org/wiki/Morph_target_animation
[2]: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets
nicopap added a commit to nicopap/bevy that referenced this issue Apr 20, 2023
Objective
---------

- Add morph targets to `bevy_pbr` (closes bevyengine#5756) & load them from glTF
- Supersedes bevyengine#3722
- Fixes bevyengine#6814

[Morph targets][1] (also known as shape interpolation, shape keys, or
blend shapes) allow animating individual vertices with fine grained
controls. This is typically used for facial expressions. By
specifying multiple poses as vertex offset, and providing a set of
weight of each pose, it is possible to define surprisingly realistic
transitions between poses. Blending between multiple poses also allow
composition. Morph targets are part of the [gltf standard][2] and are
a feature of Unity and Unreal, and babylone.js, it is only natural to
implement them in bevy.

Solution
--------

This implementation of morph targets uses a 3d storage texture where
each pixel is a component of an animated attribute. Each layer is a
different target. We use a 2d texture for each target, because the
number of attribute×components×animated vertices is expected to
always exceed the maximum pixel row size limit of webGL2. It copies
fairly closely the way skinning is implemented on the CPU side, while
on the GPU side, the shader morph target implementation is a
relatively trivial detail.

We add an optional `morph_texture` to the `Mesh` struct. The
`morph_texture` is built through a method that accepts an iterator
over attribute buffers.

The `MorphWeights` component, user-accessible, controls the blend of
poses used by mesh instances (so that multiple copy of the same mesh
may have different weights), all the weights are uploaded to a
uniform buffer of 256 `f32`. We limit to 16 poses per mesh, and a
total of 256 poses.

More literature:
* Old babylone.js implementation (vertex attribute-based): https://www.eternalcoding.com/dev-log-1-morph-targets/
* Babylone.js implementation (similar to ours): https://www.youtube.com/watch?v=LBPRmGgU0PE
* GPU gems 3: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits
* Development discord thread https://discord.com/channels/691052431525675048/1083325980615114772

https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4

Acknowledgements
---------------

* Thanks to @storytold for sponsoring the feature
* Thanks to @superdump  and @james7132 for guidance and help figuring out stuff

Future work
-----------

- Handling of less and more attributes (eg: animated uv, animated arbitrary attributes)
- Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses)
- Better animation API, see bevyengine#8357

----

Changelog
---------

- Add morph targets to bevy meshes
	- Support up to 64 poses per mesh of individually up to
          116508 vertices, animation currently strictly limited to the
          position, normal and tangent attributes.
	- Load a morph target using `Mesh::set_morph_targets`
- Add `VisitMorphTargets` and `VisitMorphAttributes` traits to
  `bevy_render`, this allows defining morph targets (a fairly complex
  and nested data structure) through iterators (ie: single copy instead
  of passing around buffers), see documentation of those traits for
  details
- Add `MorphWeights` component exported by `bevy_render`
	- `MorphWeights` control mesh's morph target weights,
          blending between various poses defined as morph targets.
	- `MorphWeights` are directly inherited by direct children
          (single level of hierarchy) of an entity. This allows
          controlling several mesh primitives through a unique entity
          _as per GLTF spec_.
- Add `GltfMeshExtras` component to query gltf extras specific to
  meshes of a given node
	- This allows reading morph weight names, see the new
          `scene_viewer` `morph_viewer_plugin.rs` module for details.
- Load morph targets weights and buffers in `bevy_gltf`
- handle morph targets animations in `bevy_animation` (previously, it
  was a `warn!` log)
- Add the `multiple_morph_target_meshes.gltf` asset for morph targets
  testing. Load it with the scene viewer.

Migration Guide
---------------

- (very specialized, unlikely to be touched by 3rd parties)
	- `MeshPipeline` now has a single `mesh_layouts` field rather
          than separate `mesh_layout` and `skinned_mesh_layout` fields.
          You should handle all possible mesh bind group layouts in
          your implementation.
	- You should also handle properly the new `MORPH_TARGETS`
          shader def and mesh pipeline key. A new function is exposed
          to make this easier: `set_mesh_binding_defs` .
	- The `MeshBindGroup` resource doesn't exist anymore. Mesh
          bind groups are computed in `queue_mesh_bind_group` system
          and are available as the field `GpuMesh::bind_group` .

[1]: https://en.wikipedia.org/wiki/Morph_target_animation
[2]: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets
nicopap added a commit to nicopap/bevy that referenced this issue Apr 22, 2023
Objective
---------

- Add morph targets to `bevy_pbr` (closes bevyengine#5756) & load them from glTF
- Supersedes bevyengine#3722
- Fixes bevyengine#6814

[Morph targets][1] (also known as shape interpolation, shape keys, or
blend shapes) allow animating individual vertices with fine grained
controls. This is typically used for facial expressions. By
specifying multiple poses as vertex offset, and providing a set of
weight of each pose, it is possible to define surprisingly realistic
transitions between poses. Blending between multiple poses also allow
composition. Morph targets are part of the [gltf standard][2] and are
a feature of Unity and Unreal, and babylone.js, it is only natural to
implement them in bevy.

Solution
--------

This implementation of morph targets uses a 3d storage texture where
each pixel is a component of an animated attribute. Each layer is a
different target. We use a 2d texture for each target, because the
number of attribute×components×animated vertices is expected to
always exceed the maximum pixel row size limit of webGL2. It copies
fairly closely the way skinning is implemented on the CPU side, while
on the GPU side, the shader morph target implementation is a
relatively trivial detail.

We add an optional `morph_texture` to the `Mesh` struct. The
`morph_texture` is built through a method that accepts an iterator
over attribute buffers.

The `MorphWeights` component, user-accessible, controls the blend of
poses used by mesh instances (so that multiple copy of the same mesh
may have different weights), all the weights are uploaded to a
uniform buffer of 256 `f32`. We limit to 16 poses per mesh, and a
total of 256 poses.

More literature:
* Old babylone.js implementation (vertex attribute-based): https://www.eternalcoding.com/dev-log-1-morph-targets/
* Babylone.js implementation (similar to ours): https://www.youtube.com/watch?v=LBPRmGgU0PE
* GPU gems 3: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits
* Development discord thread https://discord.com/channels/691052431525675048/1083325980615114772

https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4

Acknowledgements
---------------

* Thanks to @storytold for sponsoring the feature
* Thanks to @superdump  and @james7132 for guidance and help figuring out stuff

Future work
-----------

- Handling of less and more attributes (eg: animated uv, animated arbitrary attributes)
- Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses)
- Better animation API, see bevyengine#8357

----

Changelog
---------

- Add morph targets to bevy meshes
	- Support up to 64 poses per mesh of individually up to
          116508 vertices, animation currently strictly limited to the
          position, normal and tangent attributes.
	- Load a morph target using `Mesh::set_morph_targets`
- Add `VisitMorphTargets` and `VisitMorphAttributes` traits to
  `bevy_render`, this allows defining morph targets (a fairly complex
  and nested data structure) through iterators (ie: single copy instead
  of passing around buffers), see documentation of those traits for
  details
- Add `MorphWeights` component exported by `bevy_render`
	- `MorphWeights` control mesh's morph target weights,
          blending between various poses defined as morph targets.
	- `MorphWeights` are directly inherited by direct children
          (single level of hierarchy) of an entity. This allows
          controlling several mesh primitives through a unique entity
          _as per GLTF spec_.
- Add `GltfMeshExtras` component to query gltf extras specific to
  meshes of a given node
	- This allows reading morph weight names, see the new
          `scene_viewer` `morph_viewer_plugin.rs` module for details.
- Load morph targets weights and buffers in `bevy_gltf`
- handle morph targets animations in `bevy_animation` (previously, it
  was a `warn!` log)
- Add the `multiple_morph_target_meshes.gltf` asset for morph targets
  testing. Load it with the scene viewer.

Migration Guide
---------------

- (very specialized, unlikely to be touched by 3rd parties)
	- `MeshPipeline` now has a single `mesh_layouts` field rather
          than separate `mesh_layout` and `skinned_mesh_layout` fields.
          You should handle all possible mesh bind group layouts in
          your implementation.
	- You should also handle properly the new `MORPH_TARGETS`
          shader def and mesh pipeline key. A new function is exposed
          to make this easier: `set_mesh_binding_defs` .
	- The `MeshBindGroup` resource doesn't exist anymore. Mesh
          bind groups are computed in `queue_mesh_bind_group` system
          and are available as the field `GpuMesh::bind_group` .

[1]: https://en.wikipedia.org/wiki/Morph_target_animation
[2]: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets
nicopap added a commit to nicopap/bevy that referenced this issue Apr 24, 2023
Objective
---------

- Add morph targets to `bevy_pbr` (closes bevyengine#5756) & load them from glTF
- Supersedes bevyengine#3722
- Fixes bevyengine#6814

[Morph targets][1] (also known as shape interpolation, shape keys, or
blend shapes) allow animating individual vertices with fine grained
controls. This is typically used for facial expressions. By
specifying multiple poses as vertex offset, and providing a set of
weight of each pose, it is possible to define surprisingly realistic
transitions between poses. Blending between multiple poses also allow
composition. Morph targets are part of the [gltf standard][2] and are
a feature of Unity and Unreal, and babylone.js, it is only natural to
implement them in bevy.

Solution
--------

This implementation of morph targets uses a 3d storage texture where
each pixel is a component of an animated attribute. Each layer is a
different target. We use a 2d texture for each target, because the
number of attribute×components×animated vertices is expected to
always exceed the maximum pixel row size limit of webGL2. It copies
fairly closely the way skinning is implemented on the CPU side, while
on the GPU side, the shader morph target implementation is a
relatively trivial detail.

We add an optional `morph_texture` to the `Mesh` struct. The
`morph_texture` is built through a method that accepts an iterator
over attribute buffers.

The `MorphWeights` component, user-accessible, controls the blend of
poses used by mesh instances (so that multiple copy of the same mesh
may have different weights), all the weights are uploaded to a
uniform buffer of 256 `f32`. We limit to 16 poses per mesh, and a
total of 256 poses.

More literature:
* Old babylone.js implementation (vertex attribute-based): https://www.eternalcoding.com/dev-log-1-morph-targets/
* Babylone.js implementation (similar to ours): https://www.youtube.com/watch?v=LBPRmGgU0PE
* GPU gems 3: https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits
* Development discord thread https://discord.com/channels/691052431525675048/1083325980615114772

https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4

Acknowledgements
---------------

* Thanks to @storytold for sponsoring the feature
* Thanks to @superdump  and @james7132 for guidance and help figuring out stuff

Future work
-----------

- Handling of less and more attributes (eg: animated uv, animated arbitrary attributes)
- Dynamic pose allocation (so that zero-weighted poses aren't uploaded to GPU for example, enables much more total poses)
- Better animation API, see bevyengine#8357

----

Changelog
---------

- Add morph targets to bevy meshes
	- Support up to 64 poses per mesh of individually up to
          116508 vertices, animation currently strictly limited to the
          position, normal and tangent attributes.
	- Load a morph target using `Mesh::set_morph_targets`
- Add `VisitMorphTargets` and `VisitMorphAttributes` traits to
  `bevy_render`, this allows defining morph targets (a fairly complex
  and nested data structure) through iterators (ie: single copy instead
  of passing around buffers), see documentation of those traits for
  details
- Add `MorphWeights` component exported by `bevy_render`
	- `MorphWeights` control mesh's morph target weights,
          blending between various poses defined as morph targets.
	- `MorphWeights` are directly inherited by direct children
          (single level of hierarchy) of an entity. This allows
          controlling several mesh primitives through a unique entity
          _as per GLTF spec_.
- Add `GltfMeshExtras` component to query gltf extras specific to
  meshes of a given node
	- This allows reading morph weight names, see the new
          `scene_viewer` `morph_viewer_plugin.rs` module for details.
- Load morph targets weights and buffers in `bevy_gltf`
- handle morph targets animations in `bevy_animation` (previously, it
  was a `warn!` log)
- Add the `multiple_morph_target_meshes.gltf` asset for morph targets
  testing. Load it with the scene viewer.

Migration Guide
---------------

- (very specialized, unlikely to be touched by 3rd parties)
	- `MeshPipeline` now has a single `mesh_layouts` field rather
          than separate `mesh_layout` and `skinned_mesh_layout` fields.
          You should handle all possible mesh bind group layouts in
          your implementation.
	- You should also handle properly the new `MORPH_TARGETS`
          shader def and mesh pipeline key. A new function is exposed
          to make this easier: `set_mesh_binding_defs` .
	- The `MeshBindGroup` resource doesn't exist anymore. Mesh
          bind groups are computed in `queue_mesh_bind_group` system
          and are available as the field `GpuMesh::bind_group` .

[1]: https://en.wikipedia.org/wiki/Morph_target_animation
[2]: https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets
@nicopap
Copy link
Contributor

nicopap commented May 10, 2023

How this looks with #8158

bevy_morph_targets-2023-05-10.mp4

@AllenDang
Copy link
Author

@nicopap perfect

github-merge-queue bot pushed a commit that referenced this issue Jun 22, 2023
# Objective

- Add morph targets to `bevy_pbr` (closes #5756) & load them from glTF
- Supersedes #3722
- Fixes #6814

[Morph targets][1] (also known as shape interpolation, shape keys, or
blend shapes) allow animating individual vertices with fine grained
controls. This is typically used for facial expressions. By specifying
multiple poses as vertex offset, and providing a set of weight of each
pose, it is possible to define surprisingly realistic transitions
between poses. Blending between multiple poses also allow composition.
Morph targets are part of the [gltf standard][2] and are a feature of
Unity and Unreal, and babylone.js, it is only natural to implement them
in bevy.

## Solution

This implementation of morph targets uses a 3d texture where each pixel
is a component of an animated attribute. Each layer is a different
target. We use a 2d texture for each target, because the number of
attribute×components×animated vertices is expected to always exceed the
maximum pixel row size limit of webGL2. It copies fairly closely the way
skinning is implemented on the CPU side, while on the GPU side, the
shader morph target implementation is a relatively trivial detail.

We add an optional `morph_texture` to the `Mesh` struct. The
`morph_texture` is built through a method that accepts an iterator over
attribute buffers.

The `MorphWeights` component, user-accessible, controls the blend of
poses used by mesh instances (so that multiple copy of the same mesh may
have different weights), all the weights are uploaded to a uniform
buffer of 256 `f32`. We limit to 16 poses per mesh, and a total of 256
poses.

More literature:
* Old babylone.js implementation (vertex attribute-based):
https://www.eternalcoding.com/dev-log-1-morph-targets/
* Babylone.js implementation (similar to ours):
https://www.youtube.com/watch?v=LBPRmGgU0PE
* GPU gems 3:
https://developer.nvidia.com/gpugems/gpugems3/part-i-geometry/chapter-3-directx-10-blend-shapes-breaking-limits
* Development discord thread
https://discord.com/channels/691052431525675048/1083325980615114772


https://user-images.githubusercontent.com/26321040/231181046-3bca2ab2-d4d9-472e-8098-639f1871ce2e.mp4


https://github.com/bevyengine/bevy/assets/26321040/d2a0c544-0ef8-45cf-9f99-8c3792f5a258

## Acknowledgements

* Thanks to `storytold` for sponsoring the feature
* Thanks to `superdump` and `james7132` for guidance and help figuring
out stuff

## Future work

- Handling of less and more attributes (eg: animated uv, animated
arbitrary attributes)
- Dynamic pose allocation (so that zero-weighted poses aren't uploaded
to GPU for example, enables much more total poses)
- Better animation API, see #8357

----

## Changelog

- Add morph targets to bevy meshes
- Support up to 64 poses per mesh of individually up to 116508 vertices,
animation currently strictly limited to the position, normal and tangent
attributes.
	- Load a morph target using `Mesh::set_morph_targets` 
- Add `VisitMorphTargets` and `VisitMorphAttributes` traits to
`bevy_render`, this allows defining morph targets (a fairly complex and
nested data structure) through iterators (ie: single copy instead of
passing around buffers), see documentation of those traits for details
- Add `MorphWeights` component exported by `bevy_render`
- `MorphWeights` control mesh's morph target weights, blending between
various poses defined as morph targets.
- `MorphWeights` are directly inherited by direct children (single level
of hierarchy) of an entity. This allows controlling several mesh
primitives through a unique entity _as per GLTF spec_.
- Add `MorphTargetNames` component, naming each indices of loaded morph
targets.
- Load morph targets weights and buffers in `bevy_gltf` 
- handle morph targets animations in `bevy_animation` (previously, it
was a `warn!` log)
- Add the `MorphStressTest.gltf` asset for morph targets testing, taken
from the glTF samples repo, CC0.
- Add morph target manipulation to `scene_viewer`
- Separate the animation code in `scene_viewer` from the rest of the
code, reducing `#[cfg(feature)]` noise
- Add the `morph_targets.rs` example to show off how to manipulate morph
targets, loading `MorpStressTest.gltf`

## Migration Guide

- (very specialized, unlikely to be touched by 3rd parties)
- `MeshPipeline` now has a single `mesh_layouts` field rather than
separate `mesh_layout` and `skinned_mesh_layout` fields. You should
handle all possible mesh bind group layouts in your implementation
- You should also handle properly the new `MORPH_TARGETS` shader def and
mesh pipeline key. A new function is exposed to make this easier:
`setup_moprh_and_skinning_defs`
- The `MeshBindGroup` is now `MeshBindGroups`, cached bind groups are
now accessed through the `get` method.

[1]: https://en.wikipedia.org/wiki/Morph_target_animation
[2]:
https://registry.khronos.org/glTF/specs/2.0/glTF-2.0.html#morph-targets

---------

Co-authored-by: François <mockersf@gmail.com>
Co-authored-by: Carter Anderson <mcanders1@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-Animation Make things move and change over time A-Assets Load files from disk to use for things like images, models, and sounds C-Bug An unexpected or incorrect behavior
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants