Resting Bounds for skinned meshes are calculated without taking initial bind pose into account

[hybridherbst]

Description

Seems the resting bounds (that are used to calculate initial camera distance, shadow plane size, ...) don't take initial bone bind poses into account. This causes issues for e.g. characters that are posed, as they will often exceed the bounds of their mesh considerably (e.g. imagine a rest pose model that has arms hanging down, but the character has hands stretched out).

This is the case even for non-animated models - I think calculating the initial bounds should take skinning into account here.

Live Demo

• download attached zip and unpack: BentCylinder.zip
• drop the models into modelviewer.dev/editor
• note that they have different bounds but the result is identical meshes (only the position of the non-skinned mesh is different)
• one of them shows the shadow plane being too small
• the other shows off-centered viewing and camera distance clipping

Shadow plane too small

Clipping out of camera view

Version

• model-viewer: v1.10.1

Somewhat related, but here without any animation #934

[elalish]

Some guidance on what would be best here would definitely be helpful. We already have a problem with animated models, where we can only (peformantly) calculate bounds in one state, which the animation will obviously leave. I expand the shadow plane (by 2x) to try to help, but there's not really a great way to fix it besides authoring some microscopic geometry at the corners of where you want the bounds to be based.

Your example doesn't have the expanded shadow plane, so is it not animated? If that's true, why is it skinned? I need a bit more detail about the artist flows and use cases so I can try to make our behavior better. @mrdoob in case I'm not aware of a handy three.js function for this.

[hybridherbst]

Typical case for meshes that are skinned but not animated is models that are "posed", as in this case - the base model has a rig but usually an artist won't "bake" a new mesh after posing, it's all just the bone transforms. The actual skinned mesh stays the same, which is nice because you can then apply separate poses/animations to the same mesh or even different meshes that share the same bone hierarchy.

Usually (e.g. in Unity) there's a way to "bake" the mesh in place (and the engine does that out of the box when calculating bounds). In this case that wouldn't need to be every frame, instead just on the first one. There's also an option to provide one manually so the engine can skip the calculations, so artists would simply set one large enough; that doesn't work in glTF unfortunately.

Just found these pieces of code that apply to the same problem: https://discourse.threejs.org/t/object-bounds-not-updated-with-animation/3749/12 (which leads to https://jsfiddle.net/qosuLyhf/ and https://jsfiddle.net/q4sjbLuk/).
Not sure if they ever made it into three's core though - and I agree that calculating this for the entire animation might be too expensive for some models, but I think calculating for the initial pose (or maybe sample in like 10 places?) would improve the behaviour here.

[elalish]

Okay, that's good to know; I think using the initial pose could be reasonable, though can you have different initial poses for different animations? I've also thought about an extension to put manual bounds into a glTF; how does Unity represent the data? AABB, sphere, points?

[hybridherbst]

I can think of multiple "initial poses" depending on what the node has:

• no skinning, no animation: simple mesh
• skinning, no animation: bones applied to bind pose
• no skinning, per animation: transform at first frame
• skinning, per animation: bones applied to bind pose at first frame

Often (but not always) the first frame of the animation is close to the pose without animation (e.g. imagine a posed character that also has animations), so "calculate non-animated initial pose" would be a good start. The general case with animations is more difficult. I think there the "tiny corner objects" approach is ok as a workaround, but it will be required in far less cases if the initial pose is right.

Unity handles this as AABB and it's calculated automatically but can be changed on a per-instance base (so not for the mesh but for the node).
Personally I don't think a glTF extension should be needed since the data can be computed (and in the past extensions that ventured into renderer territory wheren't accepted IIRC).

Also I think this un-animated case is closer to a bug than the animated case (where one can argue that it's too expensive to calculate all places where the model will go).

[elalish]

Okay, we don't change framing when changing animations, so I think all we need here is to use the bind pose for a skinned mesh and that will hopefully be close enough to the animations to work alright.

An AABB isn't great for framing since it tends to leave a lot of margin when fit inside of the camera's pyramid-shaped frustum. The idea of an extension would be to make it quicker to calculate, rather than digging though a zillion verts in JS. I think a set of points is probably the only thing generic enough to work for most use cases.

[hybridherbst]

Agreed that AABBs don't make sense for framing :) and yes, I think framing the initial pose (aka applying the bones to the skin verts without taking animation into account) would be sufficient for many cases together with the existing 2-3x expansion of the floor.

[mrdoob]

Hmm... I think I came up with an efficient way of updating the bounding box per frame for SkinnedMesh.

The idea is... At init time, for every vertex in the geometry check the distance to the bones that have influence on it and store the longest distances to each bone. These distances would then become the bone's bounding sphere.

So in order to compute the SkinnedMesh bounding box, it should be a matter of going through all the bones and expanding the bounding box with the the transformed bounding sphere.

What do you guy think? I bet this is how other engines do it too.

[elalish]

That sounds pretty good; it would probably allow us to use three's normal frustum culling, which would be nice. Is this in any way required to calculate the bounds once based on the bind pose? I was hoping there was a function to do this already, but perhaps just wasn't fast enough to run per frame (which would be fine for now).

[mrdoob]

I think we have some code to raycast a SkinnedMesh but not to compute the bounding box.

[hybridherbst]

Would be interesting how the above algorithm fares with the files I attached to this issue (basically the mesh is offset differently but the bones are in the same place). I guess if the distance is from initial bound pose then the results should be identical; distance from unbound mesh would be different.

Regarding an extension, @elalish I just randomly stumbled upon https://github.com/KhronosGroup/glTF/tree/main/extensions/2.0/Vendor/FB_geometry_metadata which contains scene bounds (and geometric complexity)

[hybridherbst]

Found another case where this breaks, unfortunately this can happen with export defaults in Blender:
BendyCylinder-ScaledArmature.zip

Looks like this in model-viewer:

[jcowles]

This seems to only be fixed for "ModelScene", but still isn't handled when recomputing bounds for an arbitrary skinned mesh, no?

[elalish]

@jcowles I'm not sure what you mean. Perhaps you can file a new issue with specific repro steps and a testing URL?

[makc]

The idea is... At init time, for every vertex in the geometry check the distance to the bones that have influence on it and store the longest distances to each bone. These distances would then become the bone's bounding sphere.

@mrdoob what if another bone pulls the vertex even farther away?

[mrdoob]

Hmm, true...

[OndrejSpanel]

Vertices around bones do not create a rigid body when using weighted skinning. They can be stretched between multiple bones, which I think is something no rigid transformation of bounding boxes can cover. It might be interesting to check how other libraries are really handling this.

[mrdoob]

I think this may be fixed in r158: mrdoob/three.js#26919