Discussion about 3D features: bones, IK, ragdolls

[slapin]

I'm sorry to bother by this things, but I need to make some things clear.
I have the following set of features I need from Godot andI wonder about plans and opinions on
implementing this. I just need comments on how hard to implemnt this and that, ideas on implementation, plans, details, anything. I hope I'm not alone interested in this, but any attempt
to start discussion failed so far, so I'd like to try this way.

1. More user friendly bone transform editing and animation. Should allow quaternion
   and XYZ angles, scale, transform, animation of the fields, from GUI. Being implemented for everything else, it is just natural to have this for bones too. That would allow to animate skeletons from Godot
   without having to calculate transformation matrices by hand
2. Bone constraints - we need to be able to limit bone rotation, this is requirement for 3.
   Can be done in a way similar to Blender, just rotational constraints with angles, on XYZ and quaternion
   coordinates, global and local.
3. Inverse kinematics. We have this feature for 2D, but 3D one is a bit more complex,
   requires 2 targets, the IK target and pole target to be sloved right. This can be implemented in GDScript now, but done this way is messy without having 0.1 and 1.
4. Ragdoll - this can be implemented in many different ways, including via 1. plus applied constraints, with forces applied to IK targets and bases. But better use some fast published solution, which
   are available a lot.

My opinion about this is that the start should be with 0. and 1. but I might be wrong here,
as I'm not professional in this field, but I'm really curious and interested, and have an attitude.
Any ideas?

[reduz]

All this needs to be implemented together, likely in the Skeleton class.
It's not that difficult IMO.
I would do something like this:

1. Per bone, add a constraint option, which can likely be hinge or cone
   twist. For Ragdoll you most likely want to define a radius per bone too
   (you'll be creating a capsule shape for it)
2. Converting to rag-doll should be relatively straightforward because the
   physics engine already provides these joints
3. For IK, this is applied per frame as a post process, IK as a technique
   is relatively easy to implement, you just:
   a) go through every "constraint"and apply it "softly", as in
   b) First find the perfect solution between the elements affected by the
   constraint
   c) then "apply it" as a small interpolation, say with a value 0.2
   (customizable) between the current state and the solution state, so you
   push it towards the desired state instead of solving it
   d) go back to a) and repeat a few times (Generally 7 or 8 times,
   customizable)
4. chains of bones (parent-child relationships) without set constraints
   should be considered solid pieces.

The gizmo plugin system is now documented, so you can also have a gizmo
that shows your skeleton as well as your constraints now, and edit them
visually.

Obviously you need to know some vector math for this, as you have to
decompose the transform of each bone into scale, rotation and offset,
rotation being a quaternion so you can interpolate it. Bones without non
orthogonal matrices will also obviously not work (but this is fine,
skeletons never have this).

If you need any help with this let me know.. also I think there is a good
example in a Unity plugin (FinalIK) that you can get, I've heard it has a
nice visual constraint editor.

On Fri, Mar 11, 2016 at 5:47 PM, Sergey Lapin notifications@github.com
wrote:

I'm sorry to bother by this things, but I need to make some things clear.
I have the following set of features I need from Godot andI wonder about
plans and opinions on
implementing this. I just need comments on how hard to implemnt this and
that, ideas on implementation, plans, details, anything. I hope I'm not
alone interested in this, but any attempt
to start discussion failed so far, so I'd like to try this way.

More user friendly bone transform editing and animation. Should allow
quaternion
and XYZ angles, scale, transform, animation of the fields, from GUI.
Being implemented for everything else, it is just natural to have this for
bones too. That would allow to animate skeletons from Godot
without having to calculate transformation matrices by hand
2.

Bone constraints - we need to be able to limit bone rotation, this is
requirement for 3.
Can be done in a way similar to Blender, just rotational constraints
with angles, on XYZ and quaternion
coordinates, global and local.
3.

Inverse kinematics. We have this feature for 2D, but 3D one is a bit
more complex,
requires 2 targets, the IK target and pole target to be sloved right.
This can be implemented in GDScript now, but done this way is messy without
having 0.1 and 1.
4.

Ragdoll - this can be implemented in many different ways, including
via 1. plus applied constraints, with forces applied to IK targets and
bases. But better use some fast published solution, which
are available a lot.

My opinion about this is that the start should be with 0. and 1. but I
might be wrong here,
as I'm not professional in this field, but I'm really curious and
interested, and have an attitude.
Any ideas?

—
Reply to this email directly or view it on GitHub
#4018.

[slapin]

• About IK - what I don't really understand is implementation details.

As it is known IK in 3D is underconstrained problem. To make it fully set
and have single solution for each bone joint added a pole target which sets
the direction to this IK chain.
The IK implementations I seen just did the following:

1. They looped through bones from leaf of chain to the base. Each bone was
   rotated to a small angular value in direction of IK target.
2. The whole chain was rotated by a small angle in direction of pole target.
   I'd like to see the actual implementation of this or more deeper
   explanation, as we have the folowing problems here:
   a) the process either will be very slow to get to the target or it will
   overshoot and undershoot to serious values.
   How can we avoid that? This IK algorithm came from robot control, probably
   there is a version more optimized for 3D graphics?
   b) how to calculate angle to which we need to rotate each bone in case 1.
   and 2. ?

About ragdolls - do you think we need a RigidBody per bone - won't this be
very slow, as it is now quite slow to have 5 active RigidBodies in scene.
Probably we can implement lighter version?

About constraints - in case of humanoid charcters, for example, we need 2
types of constraints - cone-type, which rotates around cone
(head, upperleg, shoulders) with limited axis rotations (as local bone
coordinates are often bound to axes)., hinge-type, which rotates around
defined axis
with angular limit around this axis (lower legs, lower arms, feet, etc.
Should we generate a Joint for each bone connection of chain?

When I experimented with ragdolls implemented using RigidBodies and Joints
I had serious stability problems - the construction was seriously explosive
and unpredictable. Probably we need some extra logic behind the scene to
support this...

About gizmo plugins - can I use it directly from editor in 3D viewport? I
could not implement any new viewport function,
any help is appreciated.

On Sat, Mar 12, 2016 at 1:10 AM, Juan Linietsky notifications@github.com
wrote:

All this needs to be implemented together, likely in the Skeleton class.
It's not that difficult IMO.
I would do something like this:

1. Per bone, add a constraint option, which can likely be hinge or cone
   twist. For Ragdoll you most likely want to define a radius per bone too
   (you'll be creating a capsule shape for it)
2. Converting to rag-doll should be relatively straightforward because the
   physics engine already provides these joints
3. For IK, this is applied per frame as a post process, IK as a technique
   is relatively easy to implement, you just:
   a) go through every "constraint"and apply it "softly", as in
   b) First find the perfect solution between the elements affected by the
   constraint
   c) then "apply it" as a small interpolation, say with a value 0.2
   (customizable) between the current state and the solution state, so you
   push it towards the desired state instead of solving it
   d) go back to a) and repeat a few times (Generally 7 or 8 times,
   customizable)
4. chains of bones (parent-child relationships) without set constraints
   should be considered solid pieces.

The gizmo plugin system is now documented, so you can also have a gizmo
that shows your skeleton as well as your constraints now, and edit them
visually.

Obviously you need to know some vector math for this, as you have to
decompose the transform of each bone into scale, rotation and offset,
rotation being a quaternion so you can interpolate it. Bones without non
orthogonal matrices will also obviously not work (but this is fine,
skeletons never have this).

If you need any help with this let me know.. also I think there is a good
example in a Unity plugin (FinalIK) that you can get, I've heard it has a
nice visual constraint editor.

On Fri, Mar 11, 2016 at 5:47 PM, Sergey Lapin notifications@github.com
wrote:

I'm sorry to bother by this things, but I need to make some things clear.
I have the following set of features I need from Godot andI wonder about
plans and opinions on
implementing this. I just need comments on how hard to implemnt this and
that, ideas on implementation, plans, details, anything. I hope I'm not
alone interested in this, but any attempt
to start discussion failed so far, so I'd like to try this way.

More user friendly bone transform editing and animation. Should allow
quaternion
and XYZ angles, scale, transform, animation of the fields, from GUI.
Being implemented for everything else, it is just natural to have this
for
bones too. That would allow to animate skeletons from Godot
without having to calculate transformation matrices by hand
2.

Bone constraints - we need to be able to limit bone rotation, this is
requirement for 3.
Can be done in a way similar to Blender, just rotational constraints
with angles, on XYZ and quaternion
coordinates, global and local.
3.

Inverse kinematics. We have this feature for 2D, but 3D one is a bit
more complex,
requires 2 targets, the IK target and pole target to be sloved right.
This can be implemented in GDScript now, but done this way is messy
without
having 0.1 and 1.
4.

Ragdoll - this can be implemented in many different ways, including
via 1. plus applied constraints, with forces applied to IK targets and
bases. But better use some fast published solution, which
are available a lot.

My opinion about this is that the start should be with 0. and 1. but I
might be wrong here,
as I'm not professional in this field, but I'm really curious and
interested, and have an attitude.
Any ideas?

—
Reply to this email directly or view it on GitHub
#4018.

—
Reply to this email directly or view it on GitHub
#4018 (comment).

[reduz]

1. They looped through bones from leaf of chain to the base. Each bone was
   rotated to a small angular value in direction of IK target.

I can't see this working properly IMO, Trying to simply satisfy the
constraints is definitely the way to go, then everything should work by
itself.
You have to also make sure that some bone chain endpoints can be market as
anchored (ie, foot, or hands, but not generally head) this is what would
make the algorithm be useful in the end.

1. The whole chain was rotated by a small angle in direction of pole target.
   I'd like to see the actual implementation of this or more deeper
   explanation, as we have the folowing problems here:

The problem is that "rotating in the direction of the pole target" is not
really is a way to solve an IK chain. Imagine you have a loop, what is the
direction of the pole target in every part of a loop?

a) the process either will be very slow to get to the target or it will
overshoot and undershoot to serious values.

There is zero chance of overshoot in an iterative incremental approach

How can we avoid that? This IK algorithm came from robot control, probably
there is a version more optimized for 3D graphics?
b) how to calculate angle to which we need to rotate each bone in case 1.
and 2. ?

You have to not only calculate angle but also positions of both bones at
each side of the angle. your constraints to solve will likely be

1. Keeping distance constraints between bones
2. Keeping the right torsion angular limits for hinges and cone twisting.

Basically you just have a list of constraints to keep, and then solve every
single of them iteratively.

For example imagine you don't have any angular constraints and it's just
distance:

what you will do is kind of like:

1. go thorugh all distance constraints
2. If the bone endpoints are further apart or lower apart than their
   distance constraint, pull them together, and use as midpoint to do this
   operation the side of the chain with the larger weight (ie, if only the arm
   is being used for IK, the shoulder will have infinite weight and the hand
   will have zero weight, thus the endpoint closer to the shoulder will be the
   midpoint)
3. Instead of setting both ends of the bone to that solved position you
   computed (that satisfied the distance) to put them together, you only put
   them together say, 20% (0.2) of the way there, then go to the next
   constraint and do the same.
4. when done, go to 1 and repeat a few times.

this is super extremely fast even if you iterate a few times, there is not
even any chance of it being slow.

On Fri, Mar 11, 2016 at 8:16 PM, Sergey Lapin notifications@github.com
wrote:

• About IK - what I don't really understand is implementation details.

As it is known IK in 3D is underconstrained problem. To make it fully set
and have single solution for each bone joint added a pole target which sets
the direction to this IK chain.
The IK implementations I seen just did the following:

1. They looped through bones from leaf of chain to the base. Each bone was
   rotated to a small angular value in direction of IK target.
2. The whole chain was rotated by a small angle in direction of pole
   target.
   I'd like to see the actual implementation of this or more deeper
   explanation, as we have the folowing problems here:
   a) the process either will be very slow to get to the target or it will
   overshoot and undershoot to serious values.
   How can we avoid that? This IK algorithm came from robot control, probably
   there is a version more optimized for 3D graphics?
   b) how to calculate angle to which we need to rotate each bone in case 1.
   and 2. ?

About ragdolls - do you think we need a RigidBody per bone - won't this be
very slow, as it is now quite slow to have 5 active RigidBodies in scene.
Probably we can implement lighter version?

About constraints - in case of humanoid charcters, for example, we need 2
types of constraints - cone-type, which rotates around cone
(head, upperleg, shoulders) with limited axis rotations (as local bone
coordinates are often bound to axes)., hinge-type, which rotates around
defined axis
with angular limit around this axis (lower legs, lower arms, feet, etc.
Should we generate a Joint for each bone connection of chain?

When I experimented with ragdolls implemented using RigidBodies and Joints
I had serious stability problems - the construction was seriously explosive
and unpredictable. Probably we need some extra logic behind the scene to
support this...

About gizmo plugins - can I use it directly from editor in 3D viewport? I
could not implement any new viewport function,
any help is appreciated.

On Sat, Mar 12, 2016 at 1:10 AM, Juan Linietsky notifications@github.com
wrote:

All this needs to be implemented together, likely in the Skeleton class.
It's not that difficult IMO.
I would do something like this:

1. Per bone, add a constraint option, which can likely be hinge or cone
   twist. For Ragdoll you most likely want to define a radius per bone too
   (you'll be creating a capsule shape for it)
2. Converting to rag-doll should be relatively straightforward because
   the
   physics engine already provides these joints
3. For IK, this is applied per frame as a post process, IK as a technique
   is relatively easy to implement, you just:
   a) go through every "constraint"and apply it "softly", as in
   b) First find the perfect solution between the elements affected by the
   constraint
   c) then "apply it" as a small interpolation, say with a value 0.2
   (customizable) between the current state and the solution state, so you
   push it towards the desired state instead of solving it
   d) go back to a) and repeat a few times (Generally 7 or 8 times,
   customizable)
4. chains of bones (parent-child relationships) without set constraints
   should be considered solid pieces.

The gizmo plugin system is now documented, so you can also have a gizmo
that shows your skeleton as well as your constraints now, and edit them
visually.

Obviously you need to know some vector math for this, as you have to
decompose the transform of each bone into scale, rotation and offset,
rotation being a quaternion so you can interpolate it. Bones without non
orthogonal matrices will also obviously not work (but this is fine,
skeletons never have this).

If you need any help with this let me know.. also I think there is a good
example in a Unity plugin (FinalIK) that you can get, I've heard it has a
nice visual constraint editor.

On Fri, Mar 11, 2016 at 5:47 PM, Sergey Lapin notifications@github.com
wrote:

I'm sorry to bother by this things, but I need to make some things
clear.
I have the following set of features I need from Godot andI wonder
about
plans and opinions on
implementing this. I just need comments on how hard to implemnt this
and
that, ideas on implementation, plans, details, anything. I hope I'm not
alone interested in this, but any attempt
to start discussion failed so far, so I'd like to try this way.

More user friendly bone transform editing and animation. Should allow
quaternion
and XYZ angles, scale, transform, animation of the fields, from GUI.
Being implemented for everything else, it is just natural to have this
for
bones too. That would allow to animate skeletons from Godot
without having to calculate transformation matrices by hand
2.

Bone constraints - we need to be able to limit bone rotation, this is
requirement for 3.
Can be done in a way similar to Blender, just rotational constraints
with angles, on XYZ and quaternion
coordinates, global and local.
3.

Inverse kinematics. We have this feature for 2D, but 3D one is a bit
more complex,
requires 2 targets, the IK target and pole target to be sloved right.
This can be implemented in GDScript now, but done this way is messy
without
having 0.1 and 1.
4.

Ragdoll - this can be implemented in many different ways, including
via 1. plus applied constraints, with forces applied to IK targets and
bases. But better use some fast published solution, which
are available a lot.

My opinion about this is that the start should be with 0. and 1. but I
might be wrong here,
as I'm not professional in this field, but I'm really curious and
interested, and have an attitude.
Any ideas?

—
Reply to this email directly or view it on GitHub
#4018.

—
Reply to this email directly or view it on GitHub
<#4018 (comment)
.

—
Reply to this email directly or view it on GitHub
#4018 (comment).

[slapin]

Well, there is still something I don't quite understand. I don't quite see how we fully solve
the IK constraint, as there are infinite number of possible solutions. I understand the interpolation part,
but the full solution part is beyond my understanding...

If we solve the constraint for each joint separately, it is possible (how do we handle pole target in this case?), but how to do it for the whole chain?

[reduz]

I'm not sure how to explain this properly, I wish I could..

you never really reach the full solve, you only approximate towards it
enough that it's indistinguishable from the real solution.

By approximating each constraint individually (interpolate the solution a %
of the way there) , and then iterating the whole solution pass a few times,
you will be very close tot he full solution.

On Sat, Mar 12, 2016 at 1:06 PM, Sergey Lapin notifications@github.com
wrote:

Well, there is still something I don't quite understand. I don't quite see
how we fully solve
the IK constraint, as there are infinite number of possible solutions. I
understand the interpolation part,
but the full solution part is beyond my understanding...

If we solve the constraint for each joint separately, it is possible (how
do we handle pole target in this case?), but how to do it for the whole
chain?

—
Reply to this email directly or view it on GitHub
#4018 (comment).

[slapin]

Well, I use the following as theory reference material:
http://freespace.virgin.net/hugo.elias/models/m_ik2.htm
and the like and they use quite different approach...
can you advice me some reading on the subject? The most interesting parts are
IK and constraint enforcement.

[slapin]

I implemented prototype code https://github.com/slapin/godot-3d-ik-playground
Now I need help making it into C++.
See demo: https://www.youtube.com/watch?v=jL3wgX-QBTk

[dodgyville]

Would be great to have some more user friendly bone transformation editing and animation in the GUI (eg animation bone rotations). I almost feel like in the GUI hierarchy the mesh instance should contain the skeleton, because then the skeleton could contain bones.

[pwab]

Hey @slapin as far as I know some of your listed points were somehow implemented since the last comment here or are being worked on in #39353 by @TwistedTwigleg.

Maybe you could reevaluate what still needs to be done and create a GIP in the proposals repository or add some points to an already existing one like godotengine/godot-proposals#750.

[Calinou]

Closing per @pwab's comment.