Add support for glTF multi-receptacle assets

[aclegg3]

Motivation and Context

Current TriangleMeshReceptacle parsing supports only single mesh PLY files. This PR adds support for multi-mesh glTF files.

In the multi-mesh case, a unique TriangleMeshReceptacle instance is created for each sub-mesh.

For example, if 3 sub-meshes exist in the 1234.glb file referenced from receptacle metadata like:

"receptacle_mesh_1234": {
      "name": "receptacle_mesh_1234",
      "mesh_filepath": "my_rec_meshes/1234.glb"
  }

The parser will produce 3 TriangleMeshReceptacles named:

"receptacle_mesh_1234.0000"
"receptacle_mesh_1234.0001"
"receptacle_mesh_1234.0002"

NOTE some un-quiet-able logging is expected from receptacle loading and import initialization.
This will be fixed soon in a follow-up change to Magnum APIs introducing log quieting functionality.
E.g.:

Trade::GltfImporter::scene(): node 0 extras modelObb property is Utility::JsonToken::Type::Object, skipping
...
PluginManager::Manager: duplicate static plugin AnySceneImporter, ignoring

How Has This Been Tested

Locally with .glb and .ply files.
Should update test_assets in habitat-sim to include new .glb receptacles.

Types of changes

• [Development] A pull request that add new features to the habitat-lab task and environment codebase. Development Pull Requests must be small (less that 500 lines of code change), have unit testing, very extensive documentation and examples. These are typically new tasks, environments, sensors, etc... The review process for these Pull Request is longer because these changes will be maintained by our core team of developers, so make sure your changes are easy to understand!

Checklist

• My code follows the code style of this project.
• I have updated the documentation if required.
• I have read the CONTRIBUTING document.
• I have completed my CLA (see CONTRIBUTING)
• I have added tests to cover my changes if required.

[mosra]

With facebookresearch/habitat-sim#2066 this whole code could be replaced with something like this, assuming the goal is to get back a list of meshes that have their positions transformed according to the scene hierarchy. Basically it's about porting this example snippet to Python:

meshes = [importer.mesh(i) for i in range(importer.mesh_count)]

# Mesh referenced by mesh_assignments[i] has a corresponding transform in
# mesh_transformations[i]. Association to a particular node ID is stored in 
# scene.mapping(mn.trade.SceneField.MESH)[i], but it's not needed for anything
# here.
mesh_assignments: cr.containers.StridedArrayView1D = \
    scene.field(mn.trade.SceneField.MESH)
mesh_transformations: List[mn.Matrix4] = \
    mn.scenetools.flatten_transformation_hierarchy3d(scene, mn.trade.SceneField.MESH)
assert len(mesh_assignments) == len(mesh_transformations)

# A mesh can be referenced by multiple nodes, so this can't operate in-place.
flattened_meshes: List[mn.trade.MeshData] = []
for mesh_id, transformation in zip(mesh_assignments, mesh_transformations):
    flattened_meshes += [mn.meshtools.transform3d(mesh_id, transformation)]

The flattened_meshes is then very close to the mesh_data array here. You may however still want to store the MeshData instances or (references to) the position arrays from them instead of a List[mn.Vector3], as that's significantly more memory-efficient (as far as I can see, each mn.Vector3 instance stored in a list is 64 bytes, a lot more than the three floats per position when stored inside MeshData).

If you'd want to reduce the memory use even more, you could discard also all unused attributes except positions. I.e. do this in the mesh loading code:

mesh = ...
# This operation affects only the metadata -- the remaining attributes stay in 
# the vertex buffer but are no longer referenced and thus treated like padding
mesh = mn.meshtools.filter_only_attributes(mesh, [mn.trade.MeshAttribute.POSITION])
# This forces a repack of the mesh, throwing away all unused data (by default 
# it keeps the original layout including all padding if it's already interleaved)
mesh = mn.meshtools.interleave(mesh, mn.meshtools.InterleaveFlag.NONE)

(Wrote this off the top of my head, can't guarantee it compiles or works. But should be pretty close to a working state.)

[aclegg3]

Thanks for the API updates and suggestions here.
I refactored the code to use the new APIs and took your suggestion to store the MeshData directly instead of lists.

However, I ran into some issues with the memory improvement suggestions. I think these could be added in a later PR, but for reference:

1. mesh = mn.meshtools.filter_only_attributes(mesh, [mn.trade.MeshAttribute.POSITION])
   Seems to corrupt the indices:

len(self.mesh_data.indices) = 6
len(self.mesh_data.attribute(mn.trade.MeshAttribute.POSITION)) = 4
index = 0
self.mesh_data.indices[index] = 32349

2. mesh = mn.meshtools.interleave(mesh, mn.meshtools.InterleaveFlag.NONE) maybe need some additional plugin or compile flags?

AttributeError: module '_magnum.meshtools' has no attribute 'InterleaveFlag'

[aclegg3]

Also, I'm getting the following terminal output even with export MAGNUM_LOG=quiet:

PluginManager::Manager: duplicate static plugin AssimpImporter, ignoring
PluginManager::Manager: duplicate static plugin AnySceneImporter, ignoring
PluginManager::Manager: duplicate static plugin AnyImageImporter, ignoring
Trade::GltfImporter::scene(): node 0 extras modelNormal property is Utility::JsonToken::Type::Object, skipping
Trade::GltfImporter::scene(): node 0 extras modelObb property is Utility::JsonToken::Type::Object, skipping
Trade::GltfImporter::scene(): node 1 extras modelNormal property is Utility::JsonToken::Type::Object, skipping
Trade::GltfImporter::scene(): node 1 extras modelObb property is Utility::JsonToken::Type::Object, skipping
Trade::GltfImporter::scene(): node 2 extras modelNormal property is Utility::JsonToken::Type::Object, skipping
Trade::GltfImporter::scene(): node 2 extras modelObb property is Utility::JsonToken::Type::Object, skipping

[mosra]

export MAGNUM_LOG=quiet

Unlike with HABITAT_LOG, this affects only the startup log when creating a GL context, not everything. Implementing that would either mean getenv() on every print (which is very slow), or need some global "logging context" singleton which -- while completely fine in Habitat -- is a rather nasty thing to have in a library / middleware setting and would introduce more problems than it would solve.

From the application perspective, silencing the output is possible with scoped redirection, such as this in C++ (relevant docs):

{
    Mn::Warning redirectWarnings{nullptr};
    // all warnings coming from Magnum are silenced now
}

// all warnings coming from Magnum are printed again

Hm, so maybe I could expose something similar to Python as well. I'm thinking it could be doable with a context manager and a with statement, although I'd need to look up how to do that with pybind. Would this make sense?

import magnum as mn

with mn.redirect_warnings(None):
    # all warnings coming from Magnum are silenced now
    
# all warnings coming from Magnum are printed again

Another idea I had was to introduce Flag::Quiet on the importer interface, and make the plugins respect it. There's Flag::Verbose already. I'll add that, this seems good to have in any case.

Trade::GltfImporter::scene(): node 0 extras modelNormal property is Utility::JsonToken::Type::Object, skipping

I was asking about these on Slack a while ago: https://cvmlp.slack.com/archives/C048W359BFC/p1677178392534799?thread_ts=1677014872.077569&cid=C048W359BFC

My goal is to make them all easily available through Python, so you could for example get them as scene.field(importer.scene_field_for_name('modelNormal')), which also means having them implicitly preserved though all possible scene transformations and exported back when you save a file again. There's two options as I said in that thread -- either those get changed to not be in nested objects so the existing importer code can fetch them, or I update the glTF import/export code to handle them, somehow. I didn't get to the latter yet.

[mosra]

module '_magnum.meshtools' has no attribute 'InterleaveFlag'

It's InterleaveFlags, sorry. (I just typed that code snippet out from memory.)

Seems to corrupt the indices

Uh oh. Working on getting that fixed 😉 The root cause is that meshtools.filter_*_attributes() modify the metadata but reference the original mesh, and Python wasn't made aware of the original still being referenced. I'm making some additional changes to have these accidents caught automatically to prevent this from happening again.

I think this should work for now, I'll ping you once I have the overal robustness improved:

mesh = ...
# `mesh_filtered` now refers to data in `mesh`
# (`mesh_filtered.vertex_data_flags` isn't `OWNED`)
# until i make a proper fix, it's important to use a different variable
# which should tell Python's GC to not delete the original
mesh_filtered = mn.meshtools.filter_only_attributes(mesh, [mn.trade.MeshAttribute.POSITION])

# makes the mesh standalone / self-owned again
# (`mesh.vertex_data_flags` is `OWNED`)
mesh = mn.meshtools.interleave(mesh_filtered, mn.meshtools.InterleaveFlags.NONE)

[aclegg3]

Thanks, this appears to be working now.

[aclegg3]

Hm, so maybe I could expose something similar to Python as well. ...

with mn.redirect_warnings(None):
    # all warnings coming from Magnum are silenced now

...
Another idea I had was to introduce Flag::Quiet on the importer interface, and make the plugins respect it. ...

Either of these should functionally solve the problem from my perspective. In general a solution to quieting logs would be useful once we get to scaled training or running batched processes. 👍

Trade::GltfImporter::scene(): node 0 extras modelNormal property is Utility::JsonToken::Type::Object, skipping

I was asking about these on Slack a while ago: https://cvmlp.slack.com/archives/C048W359BFC/p1677178392534799?thread_ts=1677014872.077569&cid=C048W359BFC

I generally agree that accessing these nested objects would be valuable in the long run, but currently I just want to suppress the noise. 🙂

[vincentpierre]

LGTM, I would prefer if @mosra gave it an approval before merging.

[mosra]