Complete animation support

[mosra]

This is one thing I wanted to do since the very beginning. I'm slowly implementing various bits here and there. Making this public so we can track the (non)progress better and to have a list of requirements in case someone is willing to help.

• Interpolation
  • N-dimensional linear interpolation (Math::lerp())
  • N-dimensional Beziers with interpolation and subdivision (do we need more features?)
  • Quaternion linear/spherical interpolation (Math::lerp()/Math::slerp())
  • Dual quaternion linear/spherical interpolation (original paper)
    • Implement dual rotors to have equivalent underlying principles in 2D and 3D? http://marctenbosch.com/quaternions/
  • Screw interpolation (Dual Quternion Screw Linear Interpolation #114)
  • Cubic Hermite Spline interpolation, conversion from/to Bezier (Cubic Hermite splines #267)
    • slerp() / slerpShortestPath() overload for cubic hermite splines -- 523c167
    • Catmull-Rom conversion
    • TCB support (OpenGEX has that) conversion
  • bilerp for quaternions
  • slerp many quaternions together
  • more obscure interpolation types (logarithmic)? can that be done with easings?
  • no, I don't want to have rotation matrix interpolation with renormaliazation (for more information reread the above paper)
• Class for storing animation clips -- list of keyframes (time/value pairs) and animation duration (Keyframe-based animation support #191)
  • Templated on value type (interpolating 1D, 2D, 3D vectors, quaternions etc.)
  • Ability to quickly calculate the value at given time using given interpolator, ability to extrapolate
  • Preferably stateless
• Ability to specify duration for a track instead of having it implicit (glTF does that using the min/max properties which are currently ignored)
• Animation/Player class for storing animation state for given clip (Keyframe-based animation support #191)
  • Templated on clip value type
  • Something similar to what's in SceneGraph now, but without the scene graph and virtual calls
  • Start/stop/pause, play count etc.
• Animation::Player::stop(time), Animation::Player::pause(time) to schedule stop/pause at given time (the animation continues until then) ... then I'd need also Animation::Player::state(time), right?
• Non-linear time interpolation (OpenGEX needs that, glTF lists that as unsupported)
• Caching unpacked keyframes in the Player class for better memory locality
• "Connected" player making use of Corrade::Interconnect to report keyframe advances and state changes via signals (need to avoid hard dependency on Interconnect tho, maybe header-only?)
• Ability to do a weighted merge of animations from multiple players (make it possible to create some second-order Player?)
• Ease-in/out curves based on tweeny.js, https://easings.net/, http://robertpenner.com/easing/ -- c98a936, docs
• Ability to easily combine an existing interpolator with an easing curve, ability to combine an interpolator with a unpacker function -- c98a936
• Benchmark Player with multiple tracks and compare that to handling multiple tracks manually, as Keyframe-based animation support #191 does that only for a single track and the perceived overhead is big
• Extend Animation::Extrapolation (?) with ability to use last keyframe as the "before" extrapolation (and vice versa) ... or in general, override the behavior when the animation doesn't play yet or after it was finished, as with the CSS animation-fill-mode property (currently finishing causes the last keyframe to be used) and not-yet-started (manually stopped) causes the first keyframe to be used
• Support in importers
  • Transformation animation API -- currently all transformations are combined together into one 4x4 matrix, need to access them also separately if some are animated (10b4c06)
  • Skinning data API (list of bones joints, bone joint weights in meshes, bone objects skeleton assignment) (depends largely on Mesh Data rework #371) -- e8b6454, d7a4f3b, f447e99
  • Morph data API
  • Transformation animation import in OpenGEX not planned, nobody uses this format anymore
  • Transformation animation import in glTF (Basic animation import in glTF magnum-plugins#46)
  • Spline interpolation support in glTF (mosra/magnum-plugins@0c55e31)
  • Skinning data import in OpenGEX not planned, nobody uses this format anymore
  • Skinning data import in Assimp and glTF -- mosra/magnum-plugins@31fd4f1, mosra/magnum-plugins@9bf2b2b, mosra/magnum-plugins@c5aeedc, mosra/magnum-plugins@cf9c300, mosra/magnum-plugins@19af6b8
  • Morph data import in OpenGEX not planned, nobody uses this format anymore
  • Morph data import in glTF
  • others (light/camera property, texture coordinate animation, color animation) later?
  • import from other formats? (no, I'm not touching COLLADA ever again)
• Support for integral frame time (Keyframe-based animation support #191)
  • interpolation functions still take a pair of keyframes and a float, but everything else is integer
  • or having Animation::Track with float frame time but being able to play it back in an integral Player
  • also making it possible to directly use C++11 time types
• option to use 64bit integer nanosecond for global time (instead of floats) https://twitter.com/McCloudStrife/status/952375840930582528
• Reverse / bounce playback (forward then backward then forward ..., as in the CSS animation-direction property)
  • have an at() / interpolate() overload (a single template base for both?) that takes a signed integer as hint -- it'll still always only increment the hint but take abs(hint) as the frame index, so if passing negative, it'll go backwards.
  • what about wraparound? currently if out-of-bounds it resets it to zero, now it would need to reset to -size to go from the other side, can't really make that generic...
• Ability to replace/delete/modify tracks added to Player
  • so we are able to use one player for multiple clips (think DOD-optimized player for all rotations in a scene)
  • reschedule a track to play later, for example
  • return some handle type from add() (Containers::Handle with index + generation for this?)
  • deleting frees a slot to avoid invalidating other handles, reusing a slot increases handle generation
• use (integer?) ticks instead of an absolute time value and then have some scaling? (assimp does that: http://assimp.sourceforge.net/lib_html/data.html#anims)
• Constexpr TrackView (needs some rework to make this happen because currently TrackViewStorage does a function pointer cast)

Example opportunities:

• Basic (transformation) animation example -- preferably a scene imported from a file (or not? in case of more obscure usages?) -- e.g. http://the12principles.tumblr.com/, https://twitter.com/DavitMasia/status/1023862760789225472
• Skeletal animation example -- also imported from a file
• something like the Animation Periodic Table http://foxcodex.html.xdomain.jp/index.html or https://twitter.com/Yaesachi_64/status/1061934275128307712
• More advanced version of the Viewer example that is able to play back loaded animations (standalone project, see Magnum Player magnum-extras#6)
• Animation support in Magnum::Ui

More ideas:

• read through https://nfrechette.github.io/2016/10/21/anim_compression_toc/
• distill ideas from http://guillaumeblanc.github.io/ozz-animation/documentation/animation_runtime/ and http://guillaumeblanc.github.io/ozz-animation/documentation/references/:
  • sorting by time then track in a single array as opposed to having separate tracks sorted by time to have better memory locality
  • explicitly creating keyframes for start/stop time as an optimization step
  • another optimization step: storing animation data compressed (packed quats, half floats e.g.), uncompressing the nearest N (2) for every track and storing them in a cache (currently every track has a "last" cursor to speed up lookups, but that's still scattered around in memory)
• how to make this parallelizable when every track can be of a completely different type with different interpolator? having some "joint interpolators" that take a bundle of tracks and interpolate that together? or interpolating a bunch of different animations at the same time?

[Squareys]

Very nice, I'd really like to see this in Magnum, too!

Question -- should I give this more priority than Vulkan support (#91) or not?

As long as Vulkan is not really released yet, obviously not, but I definitely agree with you that Vulkan support would be a huge attraction to possible new users.
And with these new users, there might be someone who has time and motivation to focus on implementing this TODO list. Whereas implementing Vulkan takes alot of wide-spread knowledge and insight of the engine, which makes it hard task for anyone other than yourself. (Also, I believe you would have more fun implementing Vulkan support than animation support. )

[mosra]

@Squareys: I might actually have more fun with animations, Vulkan will require a lot of boring iterations in the beginning to get into some useful shape first (and lots of boring reading/learning). But yeah, you're right, implementing Vulkan support would be very hard for anyone that doesn't have deep insight into design of this engine, animations not so much.

[Squareys]

In addition to the paper mentioned in "Dual quaternion linear/spherical interpolation" (which discusses dual quaternion linear (and iterative) blending and screw interpolation (ScLERP)), this paper also suggests a method to compute screw parameters for screw interpolation.

[wivlaro]

I would vote for Animations over Vulkan. Animations would benefit people using the library and make it more appealing. Vulkan is good, sure, but I don't think it would make such a huge difference as having more animation support.

[VinnyVicious]

One thing that would be really amazing is a small interpolation example along the lines of Overgrowth's techniques: http://www.gamasutra.com/view/news/216973/Video_An_indie_approach_to_procedural_animation.php

Blending between only two keyframes at a specific time, producing a realtime animation.

[wivlaro]

👍 Great presentation!

[mosra]

@VinnyVicious to be frank, I thought it's always done like this :) having two, maybe three keyframes defined for each animation "clip" with some specific interpolation curve in between and then (automatically) combining all those interpolated poses together .. or am I missing something?

[VinnyVicious]

@mosra, usually, i just do the entire animation in Max and export the FBX containing it. And then play it in the engine. This leads to issues known as players skating, etc.

Does magnum have a builtin way to interpolate stuff? A small example with different curves? 👯

[wivlaro]

Yeah. It's much more common practise to just have animators do all the work and then probably use a bit of blending in the engine. Some games go further, but you'll often find that artists want more control than the engineer has time. 💃
... but not that this is the way it SHOULD be... :)

[VinnyVicious]

I wanted to get rid of the skating issues and also handle edge cases, like the jump technique. Everything is much more fluid using this Overgrowth technique. I can do more poses in Max and just ask the engine to work on the cue points based on player input.

Since i'm not very good when it comes to 3D math, i'm looking some examples on the subject. 🏄

[klaussilveira]

A good place to start learning more about IK is this CCD study: http://graphics.ucsd.edu/courses/cse169_w04/welman.pdf. This is also good: http://www.cs.cmu.edu/~./15464-s13/lectures/lecture6/IK.pdf

[mosra]

I think this could define the scope of initial "ready to use" animation support pretty well: http://the12principles.tumblr.com/

[mosra]

Forgot to mention that Beziers were added in 6b70974 (#165).

[mosra]

And I guess I could also publish my Animation braindump that I did a few months ago: https://gist.github.com/mosra/0ff8443d623cc0b09efb7a7299a0f692

[mosra]

More to add to the Disney 12 principles: https://twitter.com/jrcanest/status/788441769864073216

[mosra]

Stuff's being baked in #191.

[mosra]

More animation examples: https://twitter.com/DavitMasia/status/1023862760789225472

[VinnyVicious]

Amazing work!

[mosra]

Initial animation support merged in #191 and mosra/magnum-plugins#46.

[mosra]

Spline interpolation, Cubic Hermite, Catmull-Rom, TCB and other splines are being done in #267.

[mosra]

#267 with basic Cubic Hermite splines support needed for glTF import is merged, Cat-Rom and TCB conversion together with forgotten slerp() overloads is postponed to later.

[mosra]

Lots of great examples here: http://foxcodex.html.xdomain.jp/index.html

[mosra]

Easing is implemented in c98a936 (docs).

[pixelblender]

any plan to support ozz-animation (https://github.com/guillaumeblanc/ozz-animation) for skeletal animation maybe via integration? also seems like magnum-examples doesn't have skeletal animation example yet?

[Squareys]

@pixelblender I just recently got skeletal animation working with Magnum, all the pieces are there, but there's no example, no, and you have to write your own Skinning shaders.

[mosra]

Just for the record: some pieces for skeletal animation were recently integrated in e8b6454, d7a4f3b, 8c24ea1 and mosra/magnum-plugins@31fd4f1. Initial bits for shader support are in #444 and #441, I still need to finish and merge those (and then there will be an example as well).

[VinnyVicious]

That's awesome! Do you think an example with a character could be added so it's easier to implement?

[PierceLBrooks]

Is there any intention to eventually support automatic vertex group weight assignment algorithms such as "Bone Glow"?

[mosra]

Oh, that's cool -- didn't know such algorithms even were a thing. Yeah, why not, that'd be something to go into MeshTools I think.

Unfortunately I don't have time to implement that myself anytime soon, so contributions welcome :)

[PierceLBrooks]

The two closest things I've found on GitHub so far for skinning work are https://github.com/meshonline/Surface-Heat-Diffuse-Skinning & https://github.com/JSandusky/Urho3DProcGeom