Skinning use cases and examples

[donmccurdy]

We currently do not have example assets for many of the less-common situations involving skinning, which has prompted questions about whether they are valid and how to implement them.

Have examples:

• Joint hierarchy is intermixed with non-joint nodes. (Skeleton is not displayed with models exported from Maya2glTF donmccurdy/three-gltf-viewer#101)
  • EXAMPLE
  • NOTES: Renders in 3/3 web players. Known use cases.
  • ACTION: None.
• Animation transforming parents of joints (works) or parents of skinned mesh (no effect). (Add implementation notes about which transforms apply to skinned meshes #1195)
  • EXAMPLE
  • NOTES: Renders in 3/3 web players. Known use cases.
  • ACTIONS:
    • Perhaps warn in validator if mesh has ignored transform.
• Some or all joints are detached from the scene hierarchy ([Clarification] Should I animate nodes that are NOT part of the scene? #1383)
  • EXAMPLE
  • NOTES: Fails in multiple engines. Plausible use cases are connected to advanced situations, which are likely to require extensions or future work to enable anyway.
  • ACTIONS:
    • Update spec to disallow.
    • Add to asset generator as negative case.
    • Add check to validator.
• Joint hierarchy has no common root. ([Clarification] Skeleton root and joints hierarchy #1270)
  • As discussed, the skeleton property should probably be removed, and can already be ignored by implementations. In addition to that, we should have an example asset for which no common root is possible (e.g. bones form two subtrees starting from the scene root).
  • EXAMPLE
  • ACTION: If hierarchies without a common root can be implemented consistently, remove skeleton property. If not, consider adding restriction that joint hierarchy should have a common root, whether or not skeleton property is kept.
  • NOTES: Renders in web players. Very likely to break on import to DCC tools. Use cases are not very clear... leaning to disallow this use. Decision on skeleton TBD.
• Multiple meshes sharing a single skin. (Sharing skeletons between two skinned mesh #1285)
  • EXAMPLE
  • ACTIONS: None.
  • NOTES: Works in current engines, but doesn't really unlock certain use cases for shared skins (like crowds). It's valid, but an extension or spec update might be needed to enable those advanced use cases later.
• A subset of joints in one skin are reused in another skin. (Skinning: Can a node be in multiple skins? #1454)
  • EXAMPLE
  • ACTIONS: None.

TBD:

• Many (>4) joint vertex influences. (Skinning influenced by more than 4 joints #1381)
  • TODO — example
  • ACTIONS: None, spec states >4 influences are allowed.

In each case, if we find that an example cannot be consistently implemented we may tighten the spec accordingly.

[donmccurdy]

@bghgary would some of these be reasonably within scope for glTF-Asset-Generator to output? I can start making some one-off models in Blender if not.

[bghgary]

@donmccurdy Some of these are planned already. See https://github.com/bghgary/glTF-Asset-Generator/issues/437. I think the only ones that are not covered are the first one and last one.

[donmccurdy]

Added another edge case above, "Node references a skin but not a mesh.". See #1448.

[lexaknyazev]

@donmccurdy This isn't allowed by schema:

glTF/specification/2.0/schema/node.schema.json

Lines 85 to 88 in 6612de2

	"dependencies": {
	"weights": [ "mesh" ],
	"skin": [ "mesh" ]
	},

[donmccurdy]

Ah good to know. Removed the addition above.

[donmccurdy]

Added A subset of joints in one skin are reused in another skin. above, per #1454.

[donmccurdy]

Thanks to @kcoley, @stevk, and @bghgary, for adding new test cases in glTF-Asset-Generator, with more on the way. 🎉

From those that are available now, I've filled in three of the "example" links above:

Joint hierarchy is intermixed with non-joint nodes. (donmccurdy/three-gltf-viewer#101)

This works fine for all of the three implementations tested (BabylonJS, ThreeJS, PlayCanvas). We may want to test a native engine like Unreal or Unity as well, but it seems like this case can probably be allowed, and should possibly be mentioned as an implementation note. It is a known use case that engines may attach objects to bones after loading, and I see no reason to disallow it in the model itself.

Animation transforming parents of joints (works) or parents of skinned mesh (no effect). (#1195)

Works in all three, and the spec is already clear on this case.

Some or all joints are detached from the scene hierarchy (#1383)

Works only in BabylonJS. I'm not sure whether this should be allowed.

[zellski]

Pragmatic question -- for an exporter like FBX2glTF, do you think it's reasonable to cater to the loaders that have issues with your example #1 (joints form a strict subset of the relevant sub-scene-graph) by simply adding the inactive but tree-connecting nodes as joints in the skin? It definitely solves the issue I'm working on, but I'm not clever enough to figure out if it's likely to cause other issues on some esoteric models.

[bghgary]

I've update the first comment with the latest examples from asset generator v0.6. The only one that is missing is the >4 joints one because no engines support it right now.

[bghgary]

adding the inactive but tree-connecting nodes as joints in the skin?

Would you also add weights and inverse bind matrices for them? I think the down side is that it costs more space?

[zellski]

adding the inactive but tree-connecting nodes as joints in the skin?

Would you also add weights and inverse bind matrices for them? I think the down side is that it costs more space?

I would, and it does, which is irritating. Still, it's per-node waste rather than per-vertex waste, and on the models I just tested on, the size increase is a small fraction of a percent. I can live with that (and if others hate it it's easy to add command line flags). I'm more worried about unintended side effects I hadn't though of –– to say I'm weak on skinning is an understatement.

[bghgary]

I don't call myself an expert on skinning either, but I don't think there will be any side effects. BTW, which viewers don't work correctly without this?

[zellski]

BTW, which viewers don't work correctly without this?

The bug report I received mentioned Blender glTF import & Windows 3D Viewer.

I didn't test Blender, but the latter stumbles like this:

If we peek at the bones we see unnaturally many of them are rooted in the same spot on the lumbar spine. The misplaced limbs (head, left arm, right arm) all sit on nodes that don't don't have FbxSkin deformers in the original FBX, so they don't become joints. Maybe the loader assumes a tree, interprets this skeleton as four disjoint trees, and roots them all in the common shared ancestor node?

After the "fix" all is well (and animations run correctly too, so WEIGHTS_0 and JOINTS_0 appear to be undisturbed).

Yikes, that was a lot of large images. Sorry to spam.

[bghgary]

Yes, confirmed with 3D viewer. This will get fixed as we require that the app pass all asset generator tests.

[zen85]

I also have the problem that constraints on a mesh to another mesh or parenting does not really work. i get it to the point where the mesh gets positioned correctly once in the beginning but does not move along when iE the hand of the model moves.

i am relativly new to this: do i understand you correctly that this is a known issue of the exporter (in my case Blender) but you need testdata to solve that issue? because i would happily provide you with that if you tell me what you need and i can safe some of your precious time since it seems you are busy on so many forums answering our questions. can i contribute/support you at this somehow?

i just want to attach supersimple eyebrows on my robotface with morph-targets to get some kind of expression in that face. or is there another route to get that working? i want to achive something similar to that: https://threejs.org/examples/?q=morph#webgl_animation_skinning_morph

[donmccurdy]

@zen85 it sounds like you're describing a bug in a Blender exporter? If you haven't already, could you file a bug on the exporter you're using? If it's the one that ships with Blender, glTF-Blender-IO, we're tracking a related issue here: KhronosGroup/glTF-Blender-IO#75.

If you're able to share an example model and attach it to the issue above, that would be helpful. Thanks!