Are skin weights normalized?

[donmccurdy]

Is it [ optional | best practice | required ] that skin weights should be normalized in glTF? If a vertex's weights are[0, 0, 0, 0], is that invalid, or is some behavior expected?

See:

• glTF Loader animation bug mrdoob/three.js#10754 (comment)
• https://twitter.com/hugoamnov/status/953645111736401923
• https://knowledge.autodesk.com/support/maya/learn-explore/caas/CloudHelp/cloudhelp/2016/ENU/Maya/files/GUID-97C10B22-F128-4464-A91C-CD798680365D-htm.html

I'm inclined to say normalization should at least be considered best practice with an implementation note.

[javagl]

Just to be nitpicking clear: By "normalized" mean that they sum to 1.0, right? (Roughly corresponding to be a Convex Combination )

[donmccurdy]

That’s what i’d meant, perhaps we should define that too :)

From Maya, on skinning:

The process of scaling numbers so that they all add up to one is called normalization.

[najadojo]

At least two of the skinned models in the gltf samples have non-normalized weights that were causing trouble for our engine. I've ended up renormalizing the weight attribute as I read them in now.

[takahirox]

IMO normalized skin weights is just common 3d modeling/animation tips, and it's beyond glTF specification.

Normalized skin weights makes skinning animation natural/smooth and I'm sure most of all models have normalized weights. But weights doesn't have to be normalized. Weights just indicates how strongly the bone(joint) should influence the vertex.

So shouldn't be required but could be best practice if we want?

[donmccurdy]

For an example, in mrdoob/three.js#10754 (comment) the "bug" was that some vertices in the model had weights of 0 for all of the bones. Blender interpreted that as, "this vertex never moves". Three.js was (I think) moving those vertices with the skeleton root instead. Babylon and Qtek viewers failed to load the file. Validation showed no related issues, although it does check to see that other values like quaternions are normalized. (here's the file)

Is there a case where an artist would prefer non-normalized skin weights? If the best practice exists because many engines fail if weights aren't normalized, that seems like a good argument for putting strong-ish language in the spec.

[takahirox]

IMO normalized weights shouldn't be required. I don't think file format doesn't need to have such a limitation. But could be best practice as common 3d modeling/animation tips. Because

• From the perspective of the basic skinning idea, skin weights doesn't have to be normalized. It just indicates how strongly the bone(joint) should influence the vertex. Indeed I know some tools can set non-normalized weights. (No idea about the case where non-normalized weights is preferred tho)
• But normalized weights is recommended in general for natural/smooth skinning animation as I mentioned. I know some tools expect normalized weights.
• I think the fact that skinning behavior of non-normalized weights can be different in existing viewers is beyond file format specification.

Each tool can have such a limitation but file format doesn't need to have.

(BTW skinned vertex position will be (0,0,0) with all zeros weights in Three.js if I'm right)

[takahirox]

I found this sentence in "Skinning Definitions" of COLLADA spec v1.5 https://www.khronos.org/files/collada_spec_1_5.pdf

A vertex is typically weighted to one or more joints, where the sum of the weights equals 1

According to https://github.com/KhronosGroup/glTF/blob/master/specification/2.0/figures/gltfOverview-2.0.0a.png The vertex skinning in COLLADA is similar to that in glTF.

I think it'd be good to mention normalized weights in the spec. Now, what we have to decide is whether it should be required or not.

Update: I'm very wavering now tho I said it shouldn't be required. I want more many people's comments.

[donmccurdy]

+1 for more comments before deciding whether to require normalization.

At minimum though, I think warnings from the validator would be worthwhile, since it is best practice and some engines may not handle unnormalized weights well.

[donmccurdy]

@lexaknyazev does validating that skinning weights sum to 1 sound reasonable?

[lexaknyazev]

Just linear sum or should it be squared?

[lexaknyazev]

Also, could weights be negative? (I assume no, but maybe there're some rare use cases.)

[donmccurdy]

Just linear sum. I'm not sure if weights can be negative... 🤔 would be good to get an opinion on that from a technical artist, along with whether all-zero weights are a thing.

[takahirox]

@donmccurdy 's comment in #1195 (comment)

Unlike exchange formats, glTF deliberately restricts some features that certain engines or modeling tools allow, because they can't be widely supported. That choice is critical for glTF models to be portable and reliable.

From the perspective of this view, I came to think making normalized weights (and non-negative weights?) required would be reasonable unless there's a strong demand for non-normalized weights and negative weights.

[javagl]

I'm not an artist (well, not that kind of artist), so cannot say whether there may be a reason to have weights that do not sum to 1.0. I could imagine it, maybe for "exaggerating" some movement or influence? But until this is sorted out: Maybe one option would be to just disallow the weights to be all zeros...?

[deltakosh]

We (babylonjs team) are fixing it when loading data from .babylonjs file: https://github.com/BabylonJS/Babylon.js/blob/master/src/Mesh/babylon.geometry.ts#L1068

So it is definitely a +1 for normalized weights

[PatrickRyanMS]

I am a tech artist who works with @bghgary and he asked me to weigh in on this topic... pun intended. I can give you a few examples that may help you land on how to approach this topic. I am going to speak mostly with Maya in mind as I don't want to make absolute statements because DCC tools all have their own quirks and methods. However, this should give you a sense of what to expect from meshes.

First, Maya will always normalize skinning weights. It will not allow me to have negative values (clamped to 0) or weights that don't add up to one. If I edit the weight on a vertex that is skinned to four bones and set one of them to zero, Maya will distribute the removed weight to the other three bones. From a practical standpoint, I don't know why I would want to assign negative weights to a vertex so that it moves opposite of the bone. What I would actually do is to create a second bone to skin those vertices to and apply a relationship between those bones that moves them in opposite directions. An animator or tech artist will want a more exposed method of tweaking a rig than digging into the weights of the vertices as you would have to then have to redistribute weights on the vertices to normalize again. However, changing the relationship of the bones to one another would be a quicker alteration.

As far as zero weight [0, 0, 0, 0] on a vertex, I can think of times when we would want this. A quick example would be to imagine a cannon with a stack of cannon balls next to it. I want a skinned animation on the cannon when it fires, but the cannon balls will stay still. For optimization I combine the meshes for the cannon balls and the cannon into one mesh. In essence, the cannon balls have zero weight on their vertices to the skeleton and will stay in place when the animation plays.

I would expect the zero-weight vertices to move in relation to their game object parent (not the skeleton root bone). So if I have a parent node to the mesh and skeleton that moves, I would expect the unweighted vertices to also move, but not if the skeleton root moves in isolation to the parent.

Let me know if any of this is unclear or if you want me to provide examples.

[donmccurdy]

@PatrickRyanMS Thanks! That helps clarify a lot.

As far as zero weight [0, 0, 0, 0] on a vertex, I can think of times when we would want this.

Your example for when you might want this makes sense to me. But, do you know of any workflows where that succeeds in practice? From your description, Maya won't allow it, and based on this thread 3DS Max doesn't either. Blender does, but on export it was not behaving in three.js in the same way it did in Blender, and the model failed to load in other renderers.

I think the decision for all-zero weights should be a combination of two factors: (1) is it useful, and (2) is it likely to work reliably enough. If either answer is no, we probably want to at least warn in validation tooling about this case.

[lexaknyazev]

Given that some workflows require more than 4 influences per bone, should we say that linear sum over all weights (e.g., over 8 values in WEIGHTS_0 and WEIGHTS_1 combined) must be 1.0 (to some precision)?

Also, given that DCC tools don't export negative values, we should warn on them.

[donmccurdy]

@bghgary I'd thought that the glTF file in this thread was coming out of Blender with all-zero weights on the vertices that had been configured that way in Blender. Looking more closely, it seems that Blender was assigning them all to the root bone the same way that Maya and 3DS Max would. So, I don't actually know how three.js would have handled that.

If it's reasonably easy to create an example asset (e.g. in glTF-Asset-Generator) then it would be nice to have, but given that we now have 0 DCC tools capable of writing them, I'd be OK going with @lexaknyazev's suggestion.

[bghgary]

If it's reasonably easy to create an example asset (e.g. in glTF-Asset-Generator) then it would be nice to have

We don't have support for skins yet in the asset generator. I was going to see if we can hack in support specifically for this case, but it sounds like we don't need it right now. We can revisit this when we get to skins in the asset generator.

[donmccurdy]

Sounds good. Let's do the implementation note and validation, and create sample assets later on as possible.

[PatrickRyanMS]

I did do some experimentation with zero weights in Maya. Here was the scenario:

1. Create a sphere and bind it with a smooth skin to a skeleton consisting of two bones (root and end)
2. Create a cylinder and combine the meshes of the sphere and cylinder while the sphere is still bound to the skeleton

The results were as I expected. Moving the either of the joints will deform the sphere but nothing would happen to the cylinder. In essence, the weights on the vertices in the combined mesh did not all add up to one as the root joint did not affect the cylinder portion of the mesh.

There was also one part of the interaction that was unexpected. I could now move the mesh away from the skeleton by simply selecting the mesh and editing the transform. The sphere portion of the mesh still deforms based on the movement of the skeleton joints but also takes into account the transform of the mesh when determining final position for the vertices.

I could also no longer get into the component inspector and view the weights of the individual vertices any longer as Maya is keeping the history of the object intact to understand how to translate the portion of the mesh that is bound as well as adding an offset from the mesh node transform.

I can save this file as a maya file and all of the history remains in intact and when I load back into Maya all of the behavior above remains. I next tried to get the file out of Maya and into Unity to see how an engine might interpret Maya's history. When exporting to FBX, the mesh was deemed invalid and did not export, though the rest of the nodes such as the joints exported. I also tried importing the Maya ascii file straight into Unity assuming the FBX wasn't going to support that. Unity was able to parse the joints in the file, but the mesh was excluded again. My guess is that the zero weights on the cylinder was making the mesh invalid.

Lastly, I went back into the Maya file to determine if there was any way to get the mesh to export. Deleting history on the mesh will clean it up as expected, but also deletes the skinning information which is also expected. Deleting non-deformer history is the way to delete the history and retain the skin, but a strange thing happened. The cylinder portion of the combined mesh had weights assigned to the vertices in a somewhat random way. Most of them seemed to be evenly-ish split between the joints but a few on the top of the cylinder had different weights than the rest causing them to deform at rates different from the others.

However the strangest portion of this was that upon deleting the non-deformer history, the original cylinder mesh node reappeared in the position where its bind pose would have been and did not have a shader applied. It was still part of the combined mesh so now the mesh was comprised of a sphere and two cylinders, one without a shader. Kind of a ghost of mesh past. I still cannot get into the components to see the skinning weights, so this feels like the only way to get a valid mesh is to delete history and skin the combined mesh fresh, but that you would need a separate joint either as a parent of the sphere root joint or as a sibling of the sphere root joint both under another parent joint to skin the cylinder to.

I approached this initial scenario expecting that this might be an optimization that might need to be done late in the process and how could there be as little lost work as possible with skinned meshes. Now that I am digging into exactly what Maya is doing, this seems like a hack in the workflow. It may be ok for pre-render where your final engine in Maya, but exporting into a format to be interpreted by another engine breaks the example.

So I would conclude that while the optimization case is still valid, a zero-weight solution may not be the way to achieve it, but rather an operation that modifies the skeleton and re-skins the combined meshes. @bghgary and I discussed ways to automate that, but there are a lot of variables to actually achieving that such as applying weights to the original skinned vertices once the combined mesh is created in the case that UV islands in the two new meshes overlap. This is an issue because Maya exports an image for each joint that carries a value in UV space for each vert in the mesh when you export weights. That is a different problem to solve, however.

In the end, I don't know that we have any paths to create a zero weight vertex in a skinned mesh, so I also agree with the solution from @lexaknyazev that the linear sum of all bone weights on a vertex should equal 1. How we enforce that could be a couple of paths. Either an exporter could fail exporting a mesh where it sees non-normalized weights like the FBX did, or the weights could be normalized with zero weight vertices receiving a weight of 1.0 to the root bone automatically. Personally I would rather the mesh fail to export with a message explaining why instead of not knowing why my asset behaves unexpectedly after export.

[lexaknyazev]

One more slightly related question. Could a vertex have the same bone index used more than once, e.g.:

JOINTS_0:
0, 1, 2, 2

WEIGHTS_0:
0.33, 0.00, 0.34, 0.33

If such case is correct, we must ensure that engines behave consistently (i.e., provide models); otherwise the spec must be updated.

[donmccurdy]

Thanks for the detailed investigation @PatrickRyanMS!

In the end, I don't know that we have any paths to create a zero weight vertex in a skinned mesh, so I also agree with the solution from @lexaknyazev that the linear sum of all bone weights on a vertex should equal 1.

Sounds good to me. We can consider loosening this if someone comes up with a use case and viable workflow. I'd likewise prefer an error from exporters or validators, as opposed to re-weighting arbitrarily.

@lexaknyazev the same bone index could be listed more than once, if the weights are 0 for other uses. E.g. the Blender exporter reuses the index 0 when a vertex needs fewer than four bones, with 0 weights on all but one instance. Here's a quick example:
BoneWeightTest.zip

But in your example you mention non-zero weights on the same bone twice, and I don't think that's correct... We could update the spec to disallow multiple uses of the same bone index with non-zero weights I believe.

[bghgary]

should we say that linear sum over all weights (e.g., over 8 values in WEIGHTS_0 and WEIGHTS_1 combined) must be 1.0 (to some precision)

We are running into some issues with https://github.com/KhronosGroup/glTF-Sample-Models/tree/master/2.0/CesiumMan where the bones in that model do not sum up to 1 see referenced issue. It would be good to get this resolve for the spec and fix the relevant sample models.

[emackey]

I guess the validator should flag this as an error, then?

[bghgary]

I guess the validator should flag this as an error, then?

I vote yes.

[donmccurdy]

Proposed changes: #1352.