Add animation retargeting for skeletons (baked and runtime)

[smix8]

I think we should probably add some retargeting node, like SkeletonRetarget, which works similar to an existing skeleton but allows to retarget the animations...

Originally posted by @reduz in godotengine/godot#20733 (comment)
Also mentioned and talked about in other issues and proposals under different terms and related problems over the past years like #188 and here

I already created a module for my own selfish use case. This proposal is to discuss if there is an interest to implement Animation Retargeting in Godot in an official manner for everyone.

If the discussion yields that this feature is wanted for Godot I created a C++ module for both Godot 3.3 and Godot 4.x that could serve as a headstart (see videolinks below). For now it is still in development and not public but I don't mind sharing and also maintaining it later. Currently my tool only supports 3D skeletons but I expect it would be useful for Skeleton2D users as well.

Describe the project you are working on

Editor/Engine and different 3D game projects that require a stable and efficient 3D animation pipeline.

Describe the problem or limitation you are having in your project

• In Godot Skeleton Animations can only be shared between two skeletons that are exactly the same and even small differences in bone rest poses and proportions are enough to result in noticeable wrong animations.
• Animation transform tracks have path requirements that require the same child structure and naming but Animationtrackpaths can not be reparented and the source node cannot be renamed without manual track path and text file editing.
• Fixes to skeleton animations require constant corrections inside DCC and reimport to Godot through a slow and multiple step import pipeline that discards all previous work on custom tracks and callbacks inside animations.
• No support for free-floating bones. Skeleton bones are all forced to connect on import placing them in unwanted positions and creating constant issues with physics bone objects behavior due to wrong bone rest angles and forward vectors.
• It is impossible to adjust animations at runtime to character changes in a performance-friendly way, e.g. character customization, by recalculating and baking animation data. No, IK is never a solution for this as IK is a costly calculation on each frame.

All those issues funnel to the same issue ... skeleton animations in Godot can't be reused properly and are stuck with import errors cause a critical part of the animation pipeline for video games is missing from the game engine.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The Animation Retargeting tool helps to transfer animation data to other skeletons and to correct animation errors in animations that are so stacked with keyframes and bone tracks that a manual correction by hand is out of the question.

An example use for retargeting animations would be an RPG game where animations were originally authored for a human skeleton but should now be reused for a smaller gnome, a stocky dwarf, a slim elf or a hulking oger character.

Runtime retargeting also solves file size issue for projects with many animations and many character variants as not all animation variants need to be included in the exported game as a file.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Every game engine deals with Animation Retargeting in a different way but it is often only the user frontend that is different.

Simplified steps of Animation Retargeting:

• Compares two Skeletons in bone hierarchy and rest poses
• Calculates transform offsets and saves them in a mapping
• Adjusts animation bone transform track keyframes with the calculated offset values
• bakes and overwrites existing animations or animationsplayers at runtime or exports new animations to file
• optional bone mapping for foreign or more complicated skeleton rigs that need to remap or ignore certain bones
• optional additional corrections, e.g. match hip/feet with ground level, adjust arm spacing, ...

For Godot I think the best solution is a new Animation Retargeting Node that works for both 2D and 3D.

Editor Inspector Frontend for all Users:

• Nodepath for Source Skeleton
• Nodepath for Source AnimationPlayer
• Nodepath for Retarget Skeleton
• Nodepath for Retarget AnimationPlayer
• Export folder, export format, replace existing animations options
• Bake Button / Functions for Runtime Retargeting

This solves common usage of retargeting similar skeleton animations.

Additional Editor Inspector Frontend for Advanced Users:

• Optional bone mappings and ignores for complex skeletons, e.g. skeletons with twist joints.
• Optional bone remappings for foreign skeletons, e.g. motion capture skeleton.
• Optional corrections for more advanced use case, e.g. hip corrections, arm spacing, posture, ...

This should solve more complicated usage from advanced users. They still need to do the mappings but all the options exist.

As a best effort some mappings could be included for skeletons that are used often but never changed by the skeleton creator, e.g. the typical default skeletons from some DCC or large stores that are all kept for years to keep compatibility with newer assets.

Reasoning behind the new Node:

• A Node with Nodepaths and settings works well in both editor and at runtime ingame with the same setup.
• Animation Retargeting is used by all user levels from beginner to advanced and a Node is far more user-friendly than some abstract class hidden somewhere in the engine or on a server.
• Animation Retargeting is an important engine feature that needs visibility and one of the first actions of new Godot users is to browse through the Add Node dialog and see what options are available.
• Godot has no default engine skeleton as a reference point, a retargeting tool needs a generic and simple interface to support all kind of custom skeletons from users.
• For runtime retargeting each Target Skeleton requires one AnimationRetargeting Node but they all can share the same source Skeleton node. Skeleton nodes are heavy (they constantly lag the editor on more complex rigs). For this reason alone I think Animation Retargeting should be a lightweight node and not a replacement for Skeleton2D/Skeleton3D like suggested in the starting quote.
• Transition from Unity/Unreal would be easy. Users that are familiar with animation retargeting from Unity and Unreal could recreate their known workflow and place their "reference" skeleton in an autoload and name it RetargetingManager.

Implementation work already done:
As mentioned, I am working on a C++ implementation for both Godot 3.3 and Godot 4.x that I plan to make public in the near future.

The current tool can be seen in action in this devlog videos:
https://www.youtube.com/watch?v=DeRkU6D8t4E Devlog#2 First workable GDScript prototype
https://www.youtube.com/watch?v=7uf0NVnMcb4 Devlog#3 Animation Retargeting between skeletons, now in C++
https://www.youtube.com/watch?v=PRJbesKeBDY Devlog#4 Bone Corrections to fix errors and create animation variants
https://www.youtube.com/watch?v=EJCHrdZrhOI Devlog#5 Module backported to Godot 3.x with more export options

The C++ implementation could use a final touchup by more experienced C++ users. I never wrote anything in C++ before and while I tried to copy from existing Godot source examples I guess the code is below standard but it "just works".

If this enhancement will not be used often, can it be worked around with a few lines of script?

No, not possible in a few lines of code. Beginners have no chance to achieve animation retargeting on their own and even more advanced developers constantly struggle with it. A script implementation is only a theoretical option that I already tried but the script performance is to atrocious to be used at runtime.

Is there a reason why this should be core and not an add-on in the asset library?

Animation, AnimationPlayer, AnimationTree, Skeleton3D are all core. The entire animation system and all animation editors are core (plugins) in Godot. Some core classes may need some additions for both performance reasons and runtime retargeting.

(I am now paraphrasing the points raised in the discussions)

Animation Retargeting is a feature standard for many years in other game engines to avoid the time consuming animation creation process in DCC software for different character variants and the slow reimport through the asset pipeline. Professional game developers expect Animation Retargeting in a game engine. It was a core complaint and discussion point over the past months when talking with professionals about Godot's potential 3D game capabilities (interestingly enough barely anyone cared about raw render performance). If Godot wants to be used by more professional companies Animation Retargeting is a must-have core feature. Animations are one of the most expensive parts of 3D game development that require a rock-solid pipeline. Addons are seen as unreliable as a replacement for core features for professional 3D game projects that can take multiple years of development. Every time a decision has to be made for a new 3D project all game engines that have bad animation retargeting support or animation support in general are just ignored for budget reasons and to minimize risk of project failure due to breaking pipelines.

[jeffrey-cochran]

Interesting. Animation retargeting sounds a lot like an image registration problem to me. I'll be curious to see if they go about solving it in the same way

[smix8]

The mentioned module development versions for Godot 4.x and Godot 3.x can be found here:
https://github.com/smix8/GodotAnimationRetargeting

[smix8]

Added a feature-reduced GDScript addon version for Godot 3.x due to popular demand.
https://github.com/smix8/GodotAnimationRetargeting/tree/godot_3.x_gdscript

While not the ideal version many users on the beginner level were deterred by the Godot compile from source process but wanted AnimationRetargeting for their smaller 3D projects so they could fix skeleton animations inside the editor.

[GeorgeS2019]

@smix8 can U please join our discussion. We are brainstorming the "ultimate" pipeline for creating human animation with realistic skin, facial expression, state-of-art skeletal animation for in-game character generation.

Once we have sufficient input, we will attempt to draft a few godot proposals, consolidate what have been written that are relevant.

[Uradamus]

For what it's worth, stuff like Blender handle this stuff by just checking the names of action channels against bone names in the armature, and pairing up whatever matches. So as long as you get your naming lined up, you can share actions(animations) with as many rigs as you want, the rigs technically don't even need to be the same, as it gracefully skips over stuff that lacks matches.

Like right now I am in the process of taking some old ported models with limited skeletons and adding in new bones so more advanced animations can be added down the line, but the old limited animations still play just fine as long as I keep the old bones the same names as they were before. I'd like to make it so all of the characters for this project use the same base skeleton scene with the animationplayer that has all the shared animations then just attach a different meshinstance for each character I need, they would all already have vertex groups with the proper names that match the bone names, so the same sort of matching can be done here as well.

Another alternative, would be to allow animationplayer nodes to be passed around and do the same sort of name matching. Then I can just export the models with a skeleton that has no animations and then instance the animationplayer to it. Either would be fine by me if this approach of name matching is considered as a viable solution.

EDIT: Thinking about it some more, the latter would be better with the ability to instance the animationplayer, as I do sometimes have to adjust bone positions to match characters with different builds, all the animation data is basically just bone rotations and some bobbing for the hips. I make up for differences in height by making all the characters have the same hip height, then I can scale the skeleton or parent node to match their intended height, and then the animations work at any scale, as they are doing their thing in their local space.

[GeorgeS2019]

@Uradamus Thx for valuable insight!

[Uradamus]

Right now the way I've been handling it is to have a base skeleton in Blender, that I copy, adjust as needed and add each new character mesh to, then hide everything other than that character, and it's copy of the rig, to export. But as soon as I modify any of the animations or add new ones, I then have to go back through every character in that master scene, manually sync up their NLA stack with the original skeleton, then re-export every single character one by one, every single time.

It isn't so bad right now, as I've only done 4 characters, but this project has many dozens more that still need to be worked in, and quite a few of them bring additional animations that have to be added to the base collection and synced with all the characters and then I have that animation collection duplicated a bunch of times in Godot unnecessarily, so the lack of an easy way to share animations in Godot means many, if not hundreds of, extra hours of work in Blender to make up for it in the long run with just the one project alone. Just to give this issue some real world perspective, heh.

I've been stalling on this aspect of the project for months now because I really don't want to add all that overhead to the project, hoping that some day a more sensible solution would find it's way into Godot to handle this sort of stuff. While the OPs solution looks tempting, I am hesitant to add in external dependencies, especially ones that add in new nodes that add a whole bunch of extra work to clear out if/when the core team decide on a different vanilla solution.

[Calinou]

Superseded by #3379.