validating inverseBindMatrix: rename and tighten

[andreasplesch]

The InverseBindMatrix of a skin is supposed

to bring coordinates being skinned into the same space as each joint

https://github.com/KhronosGroup/glTF/blob/master/specification/2.0/README.md#skins

In fact, that is how the spec. functionally defines inverseBindMatrix.

As such, its value will be the

the inverse of the global transform of the respective joint, in its initial configuration.

as the gltf tutorial points out: https://github.com/KhronosGroup/glTF-Tutorials/blob/master/gltfTutorial/gltfTutorial_020_Skins.md#the-skin

So it could be optional (#461) as it can be computed by the viewer, or alternatively preapplied, and cannot be animated.

It would be a convenience/performance feature to help with fast loading.

A validator/loader could check if the supplied inverseBindMatrix is actually the inverse of the global (or perhaps relative to root) transform.

However, for some, perhaps most, glTF scenes this is not the case. For example, https://github.com/KhronosGroup/glTF-Asset-Generator/blob/master/Output/Animation_Skin/Animation_Skin_01.gltf has inverseBindMatrices which are unrelated to their joint transforms.

This then means that these matrices actually do not bring coordinates being skinned into the same space as each joint. While the spec. seemingly is at odds with that usage, it is a result of the actual dual purpose of the matrix. The second purpose is to optionally serve as bind shape matrix, bringing the mesh into the initial pose expected by the animation of the joints. This is pointed out by the implementation note which seems to contradict the intended functional definition of the matrix.

Just from the terse spec. language and without further information from tutorials, examples or other implementations it would be neigh impossible to divine how inverseBindMatrix needs to be taken into account during skinning.

What about renaming the matrix by getting rid of 'inverse' and call it more generally jointBindMatrix ? And then redefining its function to be more general as the first and unmutable transform which needs to be applied to a skinned vertex, perhaps augmented by its intended use as a combined inverse bind and shape matrix.

At least this is how its current usage comes across. If it is too late for renaming in a future version, clarifying its functional definition may still be possible.

[lexaknyazev]

@donmccurdy Could you please add this issue to the list of all open skinning questions?

[donmccurdy]

Do you mean #1403? I didn't intend for that to be a list of all skinning questions, but rather a list of specific usage the spec currently allows, which need (a) examples, (b) clarification, or (c) to be removed if they aren't correct. So everything in that list has a next step of "test the implementations."

This seems like a spec issue to be clarified or fixed. I'm not immediately sure what's correct here, or why (some?) sample models are different.

[andreasplesch]

Models use the inverseBindMatrix in different ways for different purposes. I believe this is how the matrix is intended to be used although the spec. seems to be a bit misleading or self-contradicting. #1403 does not quite fit, I agree, but perhaps can be expanded to also include spec. issues. Referring to #1403 here may suffice ?

[javagl]

Following the skinning spec discussion after the previous call, I created this image, which is supposed to go (non-normatively) into an appendix:

---

---

Admittedly, I wasn't entirely sure about the global transform part. The spec about Skins currently says:

Implementation Note: Client implementations should apply only the transform of the skeleton root node to the skinned mesh while ignoring the transform of the skinned mesh node. In the example below, ...

Of course, at some point, the transform of the node has to be taken into account - or does this statement exactly refer to "cancelling out" the global transform for the skinning computations via the G^-1 in the computation of the Joint Matrix?

[andreasplesch]

There is a typo: Mutliplying => Multiplying
Not sure about 'final' since I believe the modelViewProjection matrix in glOverview is needed for the final position and contains the global transform of the skinned node.

[javagl]

There is a typo: Mutliplying => Multiplying

The usual test, to see whether people are reading it ;-) Thanks, I fixed that locally.

The point about the 'final' is also true. Do you think that rephrasing it t something like '...the world coordinate of the skinned vertex' could be better?

In any case, this is also related to the question about the "global transform" that I mentioned. Iff I understood this correctly, then the quoted implementation note from the spec refers to the fact that in the final computations of an actual implementation, there is something like

G^-1  * skinningStuff * G

(apologies for the informal style here), where the first G^-1 is the inverse of the global transform of the actual node of the skin, and the final G is the actual global transform - although it is not directly "visible" in the skinning formulas, because it comes in via the modelViewProjection matrix.

I'm still not perfectly sure about the role of the skeleton root. Actually, there's still a task on my TODO list, namely to lift the old https://github.com/javagl/gltfTutorialModels/tree/2.0/SimpleSkin example to become a proper glTFSampleModel. Maybe I can then also create different variants with different skeleton roots and different skinned-mesh-node-transforms, so that one can immediately see whether the implementation is right - and ideally, what is wrong otherwise: The skinning in general causes quite some trouble for implementors...

[javagl]

An open question here referred to the statement from the spec

Implementation Note: Client implementations should apply only the transform of the skeleton root node to the skinned mesh while ignoring the transform of the skinned mesh node. In the example below, ...

I think my current implementation does not do this. Instead, it uses the transform of the node which actually references the mesh. I think that this might be critical, because it's not so easy to understand why it has to be done like that. In doubt, I'd add another example (in addition to the SimpleSkin example that I just PR'ed ) that shows the difference.

---

More specifically, referring to the image that is shown above (supposed to go into the appendix): Should the first bullet point

• Let G be the global transform of the node that refers to the skin

be changed to

• Let G be the global transform of the root node of the skeleton to which the skin is attached?

?

[andreasplesch]

I seem to remember a discussion in an Issue on why the transform of the skinned mesh node should be ignored. That might help to determine if the multiplications of joints by G^-1 actually is equivalent to that, or if that was just a convention which is widely expected.
But I could not immediately find the issue.

[lexaknyazev]

Now that the spec is clear that the global transform of a node that contains a skinned mesh is ignored, the only remaining use of the inverseBindMatrices data is to be the

first and unmutable transform which needs to be applied to a skinned vertex

The updated skinning section mentions when these transformations happen.

Although I agree with the property name being suboptimal, we cannot change it at this point.

[andreasplesch]

Sounds good. Thanks.