Change of the blending Skeleton3D bone rotations from 3.x makes confuse

[macryc]

Godot version

4.0 alpha 5

System information

Win10, GLES3, RTX 2070 mobile

Issue description

When adding animations in animation blend tree (via Add2/Add3 node) some bones rotate incorrectly and end up in weird positions after the animation is fully added.
Watch this vid. I am adding a back crawl animation to a forward crawl. In both, the legs should be trailing behind the torso, yet the right leg is twisting oddly as if it was either unaffected by the Add operation, or incorrectly affected.
I was able to replicate this issue with different animation Adds and a different bone is affected by this issue each time

bone_Add2.mp4

.

Steps to reproduce

Start a 3d scene
Import a Mixamo (or other) skleleton with at least 2 animations
add a AnimationTree
create new AnimationNodeBlendTree
add a Add2 and connect the 2 animations
slide the add amount to see smooth adding

Minimal reproduction project

No response

[TokageItLab]

NodeAdd had been working strangely since 3.x to begin with and made confusion #37661. Now that the blending algorithm has changed in 4.0, I think it needs to be fixed correctly.

[TokageItLab]

@macryc By the way, do you have a comparison video with 3.x? To begin with, 4.0 includes rest in the pose, so additive animation must be extracted by some means. (Ref: godotengine/godot-proposals#2700)
In other words, the 4.0 NodeAdd will apply rest twice if the same animation as in 3.x is used without modification. So the above video looks like a bug but may be the correct behavior...

[macryc]

@macryc By the way, do you have a comparison video with 3.x? To begin with, 4.0 includes rest in the pose, so additive animation must be extracted by some means. (Ref: godotengine/godot-proposals#2700) In other words, the 4.0 NodeAdd will apply rest twice if the same animation as in 3.x is used without modification. So the above video looks like a bug but may be the correct behavior...

This issue was introduced in Godot 4 alpha 5. It didn't exist in alpha 4. So it's not a Godot 3.x vs Godot 4 thing. Rather, it looks like it could have been introduced unintentionally while fixing the root bone bug?

[macryc]

For comparison, this is the exact same scene in Godot 4 alpha 4. Works fine here.
https://user-images.githubusercontent.com/82995330/160231497-f72fb17d-bc3f-4ab0-bdb9-aefb73ef71a4.mp4

[TokageItLab]

@macryc The blend method changed in alpha 5 should probably add animations with NodeAdd correctly. As we have discussed in the past with #37661 (comment) and #34134 (comment), they will add correctly if the inconsistencies in the blend order are removed.
In other words, in the previous inconsistent blends, NodeAdd and NodeBlend behaved almost identically.

Here are the points sequentially:

1. If we do not remove the blending inconsistencies, NodeAdd will not add animations correctly
2. The 4.0 rest behavior changes will make difference between "NodeAdd with inconsistencies removed in Godot 3.x" and "NodeAdd with inconsistencies removed in Godot 4"

[macryc]

You can see from the vids that the issue is affecting one bone. It affects different bones with each Add - it's difficult to see a pattern here. It's inconsistent and results in unpredictable behavior which makes Add operations quite unusable, at least for skeletons.

[TokageItLab]

It is obvious. I say again, in order to perform correct addition in Godot 4, at first, one must at least extract the additive animation godotengine/godot-proposals#2700. However, the ability to extract additive animation has not yet been implemented, so after it is added, the behavior of NodeAdd should be fixed. (In Godot 3, all animations were implicitly imported as Additive Animations.)

Sorry, I correct my misunderstand above, for Additive Animation it was already extracted by init_lot/rot/scl in BlendTree. However, I just tested it and it was adding correctly in my environment.

2022-03-26.19.50.10.mov

@macryc Can you send a sample project of a broken case?

[macryc]

Alright, thanks.
I think the issue is obvious by now but just in case, here's another occurrence. This one shows the issue more clearly, I think. Adding (blending, in this case, but the root cause is the same) an aiming animation to a walking_with_rifle animation. The left forearm bone rotates at the root as opposed to simply blend into the aiming animation:
https://user-images.githubusercontent.com/82995330/160235996-054096c4-73b3-4445-9a6a-c3efa5a2982d.mp4

[macryc]

It is obvious. I say again, in order to perform correct addition in Godot 4, at first, one must at least extract the additive animation godotengine/godot-proposals#2700. However, the ability to extract additive animation has not yet been implemented, so after it is added, the behavior of NodeAdd should be fixed. (In Godot 3, all animations were implicitly imported as Additive Animations.)

Sorry, I correct my misunderstand above, for Additive Animation it was extracted by init_lot/rot/scl in BlendTree. However, I just tested it and it was adding correctly in my environment.

2022-03-26.19.50.10.mov
@macryc Can you send a sample project of a broken case?

I will post it in a bit. But clearly, in your vid at 0:08 the hands are gone completely awry when adding the running anim...

[TokageItLab]

@macryc Please make sure it is not due to the optimizer, just to be sure. If one of the animation tracks is missing, the animation will be blended with the rest.

[macryc]

4_alfa4_FPS - add issue.zip
here's the project. Go to 'crawler' scene, open the animation tree and slide the Add2 node called 'problem'.

Please loop the long crawl anim to see it better. It looks like the left arm and leg are adding correctly while the right arm and leg are experiencing an issue. In the video above (aiming) we seem to have the same thing, like this problem is affecting the skeleton unilaterally or something...

[TokageItLab]

@macryc I tested the sample project and the animations definitely seems to be added correctly.
The behavior of NodeAdd is like rotating an elbow 90deg and then rotating the elbow more 90deg (so elbow will rotate 180deg). When large animations are added to each other, the bones may break is the correct behavior. As explained above, before the incompatibility of blending was removed (3.x and earlier than 4.0 alpha 4), NodeAdd do blending instead adding. There is no doubt about this.

[macryc]

I understand it's behaving as intended, but my point is that the intention is incorrect for many reasons:

1. you can't expect users to only be blending/adding simple animations
2. sequential adding/blending as you described will always result in unwanted/unpredicted bone positions, like in my aiming example where the end state seems to be correct but on they way to that end state there is unwanted elbow rotation occurring (and this is a very simple blend)

So what are the plans? Is this going to be addressed?

[TokageItLab]

@macryc The problem with your uploaded arms video seems to be simply a quaternion interpolation problem. Is that not complemented by the shortest distance? Can you send a sample project?

[macryc]

@macryc The problem with your uploaded arms video seems to be simply a quaternion interpolation problem. Is that not complemented by the shortest distance? Can you send a sample project?

OK but how do I solve this 'quaternion interpolation problem'? As you can see in the vid I'm simply blending 2 animations.
I will try and extract this but the project (and my animation tree) is massive...

[TokageItLab]

@macryc AnimationBlend is not allowed to rotate axis more than ±180 degrees with reference to rest. Please make sure that there is no unreasonable rotation of the elbow, or the rest is not broken. Ref: #57675 (comment)

This is my guess. It may be difficult to know that since there is no skin in your model elbow, but it appears that the roll of the elbow has been rotated 180 degrees from rest. If the model has a normal skin, the animation should not be blended to straddle 180 degrees because the Skin will be screwed up.

you can't expect users to only be blending/adding simple animations

Need to explain actual use cases for approving. Animation blending and addition are based on pure mathematics and nothing more. There are plans for retargeting and IK improvements in the meantime, but AnimationTree will have little to no further role in that.

[macryc]

Here's the player scene. It's obviously not built for sharing so you will see the anim tree is a bit messy.
Find the Blendnode called 'gun righclick aiminig' and scroll the slider to see how the left elbow rotates all over the place before it reaches the correct position. This should be a smooth blend.
ThePlayer_extract.zip
.

[TokageItLab]

@macryc I can't open it and determine the cause since it only contains tscn. But I think probably there is a problem with the T-pose (rest) or the actual bones that are being rotated. As I said:

This is my guess. It may be difficult to know that since there is no skin in your model elbow, but it appears that the roll of the elbow has been rotated 180 degrees from rest. If the model has a normal skin, the animation should not be blended to straddle 180 degrees because the Skin will be screwed up.

Or check to see if there is a small parent bone that is canceling the elbow rotation.

[macryc]

ISSUE DEMO.zip
Sorry forgot to add the animations. Try this.

Btw, why would the Tpose suddenly be a problem? It's a Mixamo import.

Here is a side-by-side comparison. Correct blending in alpha 4 (left) and incorrect blending in alpha 5 (right)

_.ThePlayer_extract.tscn.-.Issue.Demo.Blend.Add.-.Godot.Engine.2022-03-26.13-59-14.mp4

[macryc]

Need to explain actual use cases for approving. Animation blending and addition are based on pure mathematics and nothing more. There are plans for retargeting and IK improvements in the meantime, but AnimationTree will have little to no further role in that.

On this. One huge use case is mocap animations like Mixamo. Most 3d games made in Godot use them. If we can't add/blend those imported anims in Godot then that will be a deal breaker for the community. So the math behind add/blend will definitely need to be improved.

[TokageItLab]

@macryc Your model's rest is not T-pose.

Disable AnimationTree, play the T-pose and click "apply all poses to rest," then clear the AnimationTree cache (restarting Godot is a sure way to do this) and try blending again.

2022-03-26.23.01.08.mov

ISSUE DEMO_fixed.zip

As I have said many times before, the judging rotation direction of the blend is based on rest, so the model needs to be imported in a rest that is as close as possible to the middle of the range of motion for all joints #57675 (comment) and #57675 (comment). This is similar to the reason Unity Humanoid Muscle does bike poses. This is not a reason why motion capture data cannot be used, as it only sets the correct model rest. And that is the role of the importer, not BlendTree's role.

[Zireael07]

@TokageItLab: I think I would run into a similar problem! Can Godot somehow detect rest pose not being T/X/Y and warn like it should soon do on scaled bones?

[TokageItLab]

@Zireael07 The animation team plans to implement a T-poser as a post-process importer. However, I agree with you that we need to detect the pose closest to the T-pose/Apose before we can force the T-pose to be generated.
cc @fire @lyuma @SaracenOne

It may be a bit heavy processing to scan through all the animations the model has and detect which ones are T-poses. But, it is easy enough to just warn "not T-pose", so it might be right way is that let the user choose whether to manually select the T-pose animation or to force the T-pose with using T-poser.

[macryc]

In other words, the 4.0 NodeAdd will apply rest twice if the same animation as in 3.x is used without modification. So the above video looks like a bug but may be the correct behavior...

@TokageItLab can you explain this a little more? Does this mean that 4.0 additive animation adds its rotations on top of the base animation's current rotation? Trying to understand what's initial vs. rest pose (i believe this is the bone's pose in Tpose) and pose (actual current pose vs. rest pose). And what's actually being added and to what...?

[TokageItLab]

@macryc I was misunderstood about that and have already corrected it here. #59536 (comment)

However, if your model does not have a rest (a model where all bones meet at the origin with InitPose), the base pose will be added twice. In short, the difference from rest(InitPose) will be added.

[macryc]

@TokageItLab What I'm trying to understand is the difference between a bone's initial pose and rest pose.
I thought that initpose was simply a pose without rotations (quat(0,0,0,1)) and rest pose was the pose that is in the model's T-pose (which is almost always rotated vs. initpose).
Is this correct?

[TokageItLab]

@macryc I mean InitPose is one of the functions that can be selected from the viewport options when the Skeleton is selected. (I say Transform3D(Quaternion(0,0,0,1),Vector3(0,0,0)) an InitialTransform, not InitPose.)

Using it, all bones of the model will be rest pose, and NodeAdd will add the differences from the rest pose to each other.

The animation posted in #59536 (comment) is adding same animations with NodeAdd. You can see that each transformation is correctly amplified. Previously NodeAdd had an internal lerp(), so it actually behaved almost the same as NodeBlend and broken, but fixed it now.