Add support for animation streaming (animation compression already implemented in 4.0)

[reduz]

Describe the project you are working on

Godot

Describe the problem or limitation you are having in your project

Animations in Godot are relative large and cache inefficient, as each track has uncompressed data and sits in its own array. Modern imported animations (specially 3D) can have hundreds of tracks, leading to very slow processing times during playback.

Describe the feature / enhancement and how it helps to overcome the problem or limitation

The idea is to implement an animation compression format for imported animations, to make them small, streamable and cache efficient. It should be relatively simple and easy to implement.

Describe how your proposal will work, with code, pseudo-code, mock-ups, and/or diagrams

Animation storage and compression

Animations would be imported using a new format that allows for better compression. The first goal is that translation, rotation and scale are stored separated (unlike current transform tracks where they are joined), each with their own keyframes. This makes it easier to import, and also more efficient cache wise.

• TRACK_TYPE_COMPRESSED_3D_TRANSLATION
• TRACK_TYPE_COMPRESSED_3D_ROTATION
• TRACK_TYPE_COMPRESSED_3D_SCALE

These types of tracks don't contain their own memory, but load from a collection of "meta blocks" in memory.

Each metablock takes up around 4kb and has a begining time in the global timeline (represented as a frame #). Before reading any track, the animation player must check the proper metablock it will read the animation from. this means that keyframes must be inserted at the beginning and the end of the metablock for all tracks (and just at the beginning if there is no change).

Uncompressed Packet

Compressed format is lossy, so in case it is required, a track within a meta block can be stored uncompressed. This is the format without compression:

//64 bits
uint32_t frame;
float x;
float y;
float z;

Rotation is stored as follows: x/y is an octahedral normal storing axis, while z is the rotation. Converting from this to quaternion is extremely efficient.

Compressed Format

The compression format is based on "packets". Packets highly cache efficient and bit-compressed structures. There are two types of packets: Time and Data (Rotation/Translation/Scale).

Time/Offset Packets

A time offset packet is as follows (could be adjusted, just to give an idea)

//low word
Bits 0-15: frame within the metablock (max 2^16)
Bits 16-19: Time bit-width (2-16)
Bits 20-23: X Bit Width (2-16)
Bits 24-27: Y Bit Width (2-16)
Bits 28-31: Z Bit Width (2-16)
//high word
Bits 0-23: Byte offset to first packet (*4)
Bits 24-31: Amount of packets that follow

Data (Translation/Rotation/Scale) Packet

For translation and scale, the track contains the minimum and maximum values for XYZ (in float), so the 16 bit values contain a normalization value between them.

bits 0-15: Base X value of the packet
bits 16-31 Base Y value of the packet
bits 32-47: Base Z value of the packet

The following packets are bit-compressed and contain
[Time bit width] frame offset (unsigned)
[X bit width] X offset (signed)
[Y bit width] Y offset (signed)
[Z bit width] Z offset (signed)

Packets are always 4 byte aligned.

Streaming

Streaming allows playback of very large animations without having to load them entirely on disk at once. Metablocks can be streamed-in on demand. Streaming is optional, useful for games with large amount of animations or quick loading of cutscenes.

Q&A

Q: Why a format based on bitpacking and not curve fitting?
A: Bitpacking is very efficient for large amount of tracks, and can better compress high frequency information (mocap or IK sampling). It also compresses much faster.
Q: Again why the metablocks?
A: The idea of metablocks is to only load a single page from memory at once when processing an animation, hence making animation processing very cache efficient.

If this enhancement will not be used often, can it be worked around with a few lines of script?

Animation playback is core.

Is there a reason why this should be core and not an add-on in the asset library?

Animation playback is core.

[follower]

...only load a single page from memory at much when processing an animation...

Is "much" a typo for "once" here?

i.e. "...memory at once when..."

Or ..."at a time...", I guess...

[reduz]

@follower yes, fixed thanks!

[fire]

The bit packing approach makes sense given my failure trying this with curve fitting.

The call to split TRS was also what I checked and used in my curve fitting prototype.

Gordon mentioned caching packets (not in these words) but a few months ago.

For compression. What are the thoughts on stuff like run length encoding or ZSTD? I know that we have two different goals of size on disk is different than cpu cost.

I think we can defer generic compression and use the designed encoding. The current defaults of using Godot PackedScene resource (either binary, text or ZSTD) should suffice.

[TokageItLab]

Would this apply to Transform3DTrack? Also, does this affect the caching of blends in the AnimationTree?

I've worked on resolving cache incompatibilities between PropertyTrack and BezierTrack before (), and I think I'll need to resolve the incompatibility for Transform3DTrack at some point.

godotengine/godot#49411

Perhaps, as anyone mentioned in contributors chat, ideally, the three tracks should be merged into one BezierTrack. So it would be nice to have this streaming cache in a form that allows for integration at some point.

---

Also, I'm currently working on the implementation of orthogonal-mode and the non-orthogonal animation.

In the process, I realized that the Transform3DTrack needs two modes: orthogonal animation (slerp as Quat) and non-orthogonal animation (lerp as Matrix). Since currently that the Transform3DTrack is independent, we can force the Transform3DTrack exclusively for orthogonal-animation. However, when we eventually integrate everything into BezierTrack, we will need the option to choose how to animate Transform3D.

[reduz]

@TokageItLab This is kind of a replacement for Transform3DTrack, but it's baked (you will not be able to edit it) because it is compressed. I am still unsure whether it should use a single track or multiple ones, but it is possible it could just use a single one and be kind of API compatible with transform 3D track.

Regaring AnimationPlayer and AnimationTree, I will do separate proposal to suggest solutions for this.

This, by the way, is not Bezier and it is entirely optimized for streaming, I don't think this will be possible (or as efficient) with other track types to do this.

[WolfgangSenff]

How would this affect procedural animations that still use the AnimationPlayer (for instance with the style where the tracks are manipulated at runtime)? Would this be able to just be turned off and go back to the old way?

[reduz]

@WolfgangSenff sure, this is an optional mode for importing animations

[fire]

Is this fixed by godotengine/godot#54050?