FBX mesh breakage morphing MTIs

[sandtreader]

Description

Hi,

I'm a Three.JS newbie, playing with facial animation on FBX and GLTF models based on MorphTargetInfluences (MTIs). I have a simple model loader which reads the model and providers sliders to move the various MTIs / blend shapes. This was all working until a recent update from three@0.158.0 to three@0.162.0 (it's still bad in 0.163.0).

The problem is only on FBX models, and the visual symptom is the mesh appears to break up like a death scene from Tron 1 as the MTIs are changed from 0. The bigger the intended change, the more vertices are affected, and they only seem to be affected on one side of the face, and the ones that should be changing don't move. It's as if the MTIs were being applied to the wrong vertices.

So the question is, is there any known recent change in the FBX morphing import, and any workround?

Thanks!

Paul

Reproduction steps

1. Load an FBX model from FBXLoader (sorry I can't share the model - but it comes from here: https://www.cgtrader.com/3d-models/character/woman/rigged-female-model-with-facial-blend-shapes. The morphing is excellent, but I can't get the Maya textures to load - a different issue)
2. Locate the right mesh in the model
3. Get MTIs from mesh.morphTargetDictionary and tweak values in the morphTargetInfluences array

Code

Working in R3F:

ModelPage.tsx:

import React, { useRef, useState, useEffect } from 'react';
import { Mesh, Scene } from 'three';
import { Canvas, useLoader } from '@react-three/fiber';
import { OrbitControls } from '@react-three/drei';
import { Stack, Slider, Grid, Typography } from '@mui/material';

type ModelPageProps = {
  loader: any,
  file: string,
  scale: number,
  position: number[]
};

const ModelPage: React.FC<ModelPageProps> = ({loader, file, scale, position}) => {
  let model = useLoader(loader, file);
  if (model.scene) model = model.scene;  // For GLTF

  const scene = useRef<Scene>(null);
  const [mesh, setMesh] = useState<Mesh | null>(null);
  const [mtis, setMtis] = useState<Array<number>>([]);

  // Find mesh in model on first load
  useEffect(() => {
    console.log("Initialising model");
    if (model)
    {
      model.traverse( (child: any) => {
        if (child instanceof Mesh)
        {
          console.log(`Found a mesh ${child.name}`);
          child.updateMorphTargets();
          if (child.morphTargetInfluences)
          {
            console.log(`Morph targets ${JSON.stringify(child.morphTargetDictionary)}`);
            if (child.morphTargetInfluences.length > 10)
            {
              setMesh(child);
              child.updateMorphTargets();
              console.log(`Mesh morph values: ${JSON.stringify(child.morphTargetInfluences)}`);
              setMtis(child.morphTargetInfluences);
            }
          }
        }
      });
    }
  }, [model]);

  // Update the MTIs from our own copy
  useEffect(() => {
    if (mesh?.morphTargetInfluences) mesh.morphTargetInfluences = mtis;
  }, [mtis]);

  return (
    <Stack direction="row" sx={{ width: '100%', height: '80vh', justifyContent: 'space-between' }} >
      <Grid container spacing={1} sx={{ overflowY: 'auto' }}>
        {
          mesh && mesh.morphTargetDictionary &&
          Object.entries(mesh.morphTargetDictionary).map(([key, index]) => {
            const name = key.replaceAll('_', ' ');
            return (
              <>
                <Grid item xs={6}>
                  <Typography align='right'>{name}</Typography>
                </Grid>
                <Grid item xs={5}>
                  <Slider value={mtis[index]} max={1} step={0.01} sx={{ marginLeft: 2, flex: 1 }}
                          onChange={(_, v) => {
                            console.log(`${name} = ${v}`);
                            if (typeof v === 'number')
                            {
                              setMtis(oldMtis => { const newMtis=[...oldMtis]; newMtis[index]=v; return newMtis; });
                            }
                          }}/>
                </Grid>
              </>
            )
          })
        }
      </Grid>
      <Canvas>
        <ambientLight intensity={1}/>
        <directionalLight position={[3, 3, 10]}/>
        <pointLight position={[1, 1, -3]} decay={0} />
        <primitive ref={scene} object={model} scale={scale} position={position} />
        <OrbitControls/>
      </Canvas>
    </Stack>
  );
};

export default ModelPage;

BodyModelPage.tsx:

import React, { Suspense } from 'react';
import { FBXLoader } from 'three/examples/jsm/loaders/FBXLoader';
import ModelPage from './ModelPage';

const BodyModelPage: React.FC = () => {
  return (
    <Suspense>
      <ModelPage loader={FBXLoader} file="models/halle/model.fbx"
                 scale={0.3} position={[0,-50,-5]} />
    </Suspense>
  );
};

export default BodyModelPage;

This is then used in a larger React app but that's not relevant, I don't think...

Live example

Sorry I don't have this publicly available (I could do with some effort)

Screenshots

Console output:

Found a mesh Halle_Base_Body
ModelPage.tsx:36 Morph targets {"BlendShapes_Halle_Base_Body.Open":0,"BlendShapes_Halle_Base_Body.Explosive":1,"BlendShapes_Halle_Base_Body.Dental_Lip":2,"BlendShapes_Halle_Base_Body.Tight_O":3,"BlendShapes_Halle_Base_Body.Tight":4,"BlendShapes_Halle_Base_Body.Wide":5,"BlendShapes_Halle_Base_Body.Affricate":6,"BlendShapes_Halle_Base_Body.Lip_Open":7,"BlendShapes_Halle_Base_Body.Brow_Raise_Inner_L":8,"BlendShapes_Halle_Base_Body.Brow_Raise_Inner_R":9,"BlendShapes_Halle_Base_Body.Brow_Raise_Outer_L":10,"BlendShapes_Halle_Base_Body.Brow_Raise_Outer_R":11,"BlendShapes_Halle_Base_Body.Brow_Drop_L":12,"BlendShapes_Halle_Base_Body.Brow_Drop_R":13,"BlendShapes_Halle_Base_Body.Brow_Raise_L":14,"BlendShapes_Halle_Base_Body.Brow_Raise_R":15,"BlendShapes_Halle_Base_Body.Eye_Blink":16,"BlendShapes_Halle_Base_Body.Eye_Blink_L":17,"BlendShapes_Halle_Base_Body.Eye_Blink_R":18,"BlendShapes_Halle_Base_Body.Eye_Wide_L":19,"BlendShapes_Halle_Base_Body.Eye_Wide_R":20,"BlendShapes_Halle_Base_Body.Eye_Squint_L":21,"BlendShapes_Halle_Base_Body.Eye_Squint_R":22,"BlendShapes_Halle_Base_Body.Nose_Scrunch":23,"BlendShapes_Halle_Base_Body.Nose_Flanks_Raise":24,"BlendShapes_Halle_Base_Body.Nose_Flank_Raise_L":25,"BlendShapes_Halle_Base_Body.Nose_Flank_Raise_R":26,"BlendShapes_Halle_Base_Body.Nose_Nostrils_Flare":27,"BlendShapes_Halle_Base_Body.Cheek_Raise_L":28,"BlendShapes_Halle_Base_Body.Cheek_Raise_R":29,"BlendShapes_Halle_Base_Body.Cheek_Suck":30,"BlendShapes_Halle_Base_Body.Cheek_Blow_L":31,"BlendShapes_Halle_Base_Body.Cheek_Blow_R":32,"BlendShapes_Halle_Base_Body.Mouth_Smile":33,"BlendShapes_Halle_Base_Body.Mouth_Smile_L":34,"BlendShapes_Halle_Base_Body.Mouth_Smile_R":35,"BlendShapes_Halle_Base_Body.Mouth_Frown":36,"BlendShapes_Halle_Base_Body.Mouth_Frown_L":37,"BlendShapes_Halle_Base_Body.Mouth_Frown_R":38,"BlendShapes_Halle_Base_Body.Mouth_Blow":39,"BlendShapes_Halle_Base_Body.Mouth_Pucker":40,"BlendShapes_Halle_Base_Body.Mouth_Pucker_Open":41,"BlendShapes_Halle_Base_Body.Mouth_Widen":42,"BlendShapes_Halle_Base_Body.Mouth_Widen_Sides":43,"BlendShapes_Halle_Base_Body.Mouth_Dimple_L":44,"BlendShapes_Halle_Base_Body.Mouth_Dimple_R":45,"BlendShapes_Halle_Base_Body.Mouth_Plosive":46,"BlendShapes_Halle_Base_Body.Mouth_Lips_Tight":47,"BlendShapes_Halle_Base_Body.Mouth_Lips_Tuck":48,"BlendShapes_Halle_Base_Body.Mouth_Lips_Open":49,"BlendShapes_Halle_Base_Body.Mouth_Lips_Part":50,"BlendShapes_Halle_Base_Body.Mouth_Bottom_Lip_Down":51,"BlendShapes_Halle_Base_Body.Mouth_Top_Lip_Up":52,"BlendShapes_Halle_Base_Body.Mouth_Top_Lip_Under":53,"BlendShapes_Halle_Base_Body.Mouth_Bottom_Lip_Under":54,"BlendShapes_Halle_Base_Body.Mouth_Snarl_Upper_L":55,"BlendShapes_Halle_Base_Body.Mouth_Snarl_Upper_R":56,"BlendShapes_Halle_Base_Body.Mouth_Snarl_Lower_L":57,"BlendShapes_Halle_Base_Body.Mouth_Snarl_Lower_R":58,"BlendShapes_Halle_Base_Body.Mouth_Bottom_Lip_Bite":59,"BlendShapes_Halle_Base_Body.Mouth_Down":60,"BlendShapes_Halle_Base_Body.Mouth_Up":61,"BlendShapes_Halle_Base_Body.Mouth_L":62,"BlendShapes_Halle_Base_Body.Mouth_R":63,"BlendShapes_Halle_Base_Body.Mouth_Lips_Jaw_Adjust":64,"BlendShapes_Halle_Base_Body.Mouth_Bottom_Lip_Trans":65,"BlendShapes_Halle_Base_Body.Mouth_Skewer":66,"BlendShapes_Halle_Base_Body.Mouth_Open":67}
ModelPage.tsx:41 Mesh morph values: [0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0]
...
BlendShapes Halle Base Body.Tight O = 0.01
ModelPage.tsx:70 BlendShapes Halle Base Body.Tight O = 0.05
ModelPage.tsx:70 BlendShapes Halle Base Body.Tight O = 0.27
ModelPage.tsx:70 BlendShapes Halle Base Body.Tight O = 0.28
ModelPage.tsx:70 BlendShapes Halle Base Body.Tight O = 0.37
ModelPage.tsx:70 BlendShapes Halle Base Body.Tight O = 0.46
ModelPage.tsx:70 BlendShapes Halle Base Body.Tight O = 0.5
ModelPage.tsx:70 BlendShapes Halle Base Body.Tight O = 0.53
ModelPage.tsx:70 BlendShapes Halle Base Body.Tight O = 0.57
ModelPage.tsx:70 BlendShapes Halle Base Body.Tight O = 0.58
ModelPage.tsx:70 BlendShapes Halle Base Body.Tight O = 0.61
ModelPage.tsx:70 BlendShapes Halle Base Body.Tight O = 0.66
... etc ...

Version

0.163.0

Device

Desktop

Browser

Chrome

OS

Linux

[Mugen87]

three@0.158.0 to three@0.162.0

Is it possible for you to pinpoint the exact release when the breakage occurs? That would make it easier to find the breaking change.

sorry I can't share the model

Is there maybe another asset that you can share for reproduction? Without a test asset it will be hard to investigate the issue.

[Mugen87]

Maybe this issue is fixed with #28397. Do you mind testing with latest dev?

https://github.com/mrdoob/three.js/blob/dev/examples/jsm/loaders/FBXLoader.js

[Mugen87]

Closing. Feel free to reopen the issue if you can still reproduce with at least r165.