Add XR Face Tracking support

[Malcolmnixon]

This PR implements godotengine/godot-proposals#9078 by:

• Adding an XRFaceTracker Object (containing face-tracking blend shape weights)
• Adding XRFaceTracker3D Node (to drive a face mesh)

The XRFaceTracker type defines a standard Godot definition of face-tracking data based on the Unified Expressions standard. Different providers (OpenXR-Meta, OpenXR-HTC, VMC, MoCap, etc.) can map their custom data formats into this standard, and then consumers such as XRFaceTracker3D (or custom game logic) can act on this standardized data.

The following demonstration video shows a face being driven by an OpenXR XRFaceTracker.

2024-02-18.21-21-04.mp4

[dsnopek]

Thanks!

Overall, this looks really nice to me :-)

Does XRFaceTrackingProvider::update and XRFaceTrackingProvider::_update need to exist, or should the updating just be done by the process() of the OpenXR extension.

Whether or not we'll need the update() method depends on if we have any timing issues, ie will the code that updates the face tracking data run before or after we need to access it for rendering. If it runs after, that means face tracking data will be one frame delayed.

So, I'd probably lean towards keeping the update() method. It's possible we may not need it in the case of OpenXR (the process on the OpenXR extension may be sufficient), but since this is API agnostic and we may end up with WebXR or Apple Vision implementations at some point, I think it's safer to keep it around.

[BastiaanOlij]

Feedback from discussions on contributors chat.

• Suggest to rename the data object XRFaceTracker and drop the _update method. This brings it in line with XRPositionalTracker approach for controllers.
• Suggest to rename the node to XRFaceTracker3D to be in line with other 3D nodes and set it apart from the data object.
• Remove the 'Provider' suffix from the methods added to XRServer

[BastiaanOlij]

Looks great! Maybe update the original description of the PR to match the latest decisions.

[dsnopek]

Suggest to rename the data object XRFaceTracker and drop the _update method. This brings it in line with XRPositionalTracker approach for controllers.

Does this assume that face tracker data will be updated by XRInterface::process() or earlier? That's the assumption with positional tracker data.

We need some way (even if it's just convention) to know that we're updating the tracker data before it'll be used. For "remote" face tracking data, we could even register an XRInterface since the API does allow for multiple interfaces to be active as noted in the comment in XRServer::_process():

void XRServer::_process() {
	// called from our main game loop before we handle physics and game logic
	// note that we can have multiple interfaces active if we have interfaces that purely handle tracking

	// process all active interfaces
	for (int i = 0; i < interfaces.size(); i++) {
		if (!interfaces[i].is_valid()) {
			// ignore, not a valid reference
		} else if (interfaces[i]->is_initialized()) {
			interfaces.write[i]->process();
		};
	};
};

[Malcolmnixon]

I modified XRServer.get_face_tracker_paths() : PackedStringArray to instead be XRServer.get_face_trackers() : Dictionary. This aligns better with the existing XRServer.get_trackers() : Dictionary method used for positional trackers.

Similar to the other method, this one returns a dictionary of StringName to XRFaceTracker allowing the caller to check for existence, enumerate, and get the tracker instances all using the result of this function.

[Malcolmnixon]

I have a question about naming conventions. This PR introduces two classes:

• XRFaceTracker - a RefCounted data object containing tracked face blend shape weights
• XRFaceTracker3D - a Node used to drive a MeshInstance3D face from the blend shape weights

The important point is that XRFaceTracker3D is NOT a Node3D as it needs no spatial location. It has the 3D name because:

1. It drives a MeshInstance3D
2. Removing the 3D would cause a name-collision with the XRFaceTracker object

Is this acceptable, or is it a violation of a naming standard where the '3D' postfix of a node implies it inherits from Node3D?

[fire]

Is this the first time a '3D' postfix of a node does not inherits from Node3D? That suggest maybe a different name would be better.

[dsnopek]

Maybe:

• XRFaceTracker for the RefCounted data object, because it's tracking data
• XRFace for the Node, because it's not tracking, it's rendering the face?

[Mickeon]

May be more accurate but I did have to look up what it meant as a non-native English speaker.

Suggested change

	Constricts the right eye pupil.
	Contracts the right eye pupil.

[Malcolmnixon]

The names and descriptions come from the Unified Expressions standard. They actually put pictures showing the side-effect on meshes because facial distortions and the associated terminology are a specialized subject.

That website does have hover-over popups for each blend-shape with corresponding anatomical information and enhanced descriptions. Perhaps using the extended descriptions may be more useful.

[Mickeon]

Being more detailed here, since there are no images would definitely be useful. But I also do believe, wherever the description comes from, it can be "toned down" a bit for most readers.

[BastiaanOlij]

@Mickeon I'm kinda with Malcolm on this one. The descriptions come directly from the standard and we refer to the standard. I don't think we should be re-inventing the wheel here.

People who work on this stuff will be looking at the unified expression standard and documentation around this, or be already familiar with the standard. If we suddenly deviate that is only going to lead to extra confusion.

Getting people up and running who aren't familiar here, there is a lot more information to disseminate. including a lot of stuff around modeling and how to setup the blend shapes in the first place. That is more tutorial grounds and could use a nice step by step write down in the manual, or a good video tutorial.

[Malcolmnixon]

The standard is maintained in Github at https://github.com/VRCFaceTracking/docs and the author has requested feedback and PR improvements. Possibly feeding improvements through the standard might be the better approach as Godot is adopting this standard.

I am concerned about creating too many URL links to the standard as it would put us at the mercy of URL path changes; and I'm not sure we have the right to copy the images.

[Mickeon]

Suggested change

	Constricts the left eye pupil.
	Contracts the left eye pupil.

[Mickeon]

These do not feel explanatory enough. Not a VR expert and I'm not sure what they are referring to, indeed.

[Mickeon]

I believe "tilt" is a simpler yet equivalent word? "lean"?

Suggested change

	Right corner lip slants up.
	Right corner lip tilts up.

[Mickeon]

Suggested change

	Tongue arches down then up inside the mouth.
	Tongue bends down then up inside the mouth.

[Mickeon]

Ah so that's how it's called. This PR is turning into a human biology lesson for me.

Suggested change

	The Adam's apple visibly swallows.
	The neck's Adam's apple visibly swallows.

[Mickeon]

Suggested change

	Constricts both pupils.
	Contracts both pupils.

[Mickeon]

Unsure on what this means exactly at a glance.

[Malcolmnixon]

After discussing with @BastiaanOlij we decided to rename XRFaceTracker3D to XRFaceModifier3D to better align with the upcoming SkeletonModifier3D. Additionally it now extends from Node3D so developers can use this node in to hold the mesh if desired.

[akien-mga]

Thanks!