Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Articulated object skinning #2076

Merged
merged 12 commits into from
Apr 24, 2023
Merged

Articulated object skinning #2076

merged 12 commits into from
Apr 24, 2023

Conversation

0mdc
Copy link
Contributor

@0mdc 0mdc commented Apr 18, 2023

Context

To enable training of collaborative agents, we need the ability to render humans in simulation.

This changeset introduces skinned mesh support to Habitat-Sim by enabling articulated objects to render a skinned mesh that follows along its articulations.

skinning.webm

How it works

Articulated objects are defined by URDF files, while skinned meshes are creating using glTF or FBX files.

Upon creating an articulated object, Habitat looks for a <asset name>.ao_config.json file, which may contain a render_asset field containing the skinned mesh path.

If a skinned mesh is found, the system instantiates both objects and string-matches glTF bone names with URDF link names. During this process, new child transform nodes are created for each articulated object link so that the skinned mesh can be driven.

For the system to work as desired, both URDF and glTF definitions need to share the same bone names, dimensions and unit conventions. Blender scripts that generates URDF files from rigged meshes will be made available for enabling quick creation of models that can satisfy these constraints.

ao_config.json

The following parameters are added in ao_config.json:

  • render_asset: Specifies the path to the GLTF model. Activates skinning and disables primitive rendering.
  • debug_render_primitivies: Flag that turns on primitive rendering on top of skinned mesh for debugging.

ao_skinning_params
Left: default, Middle: render_asset, Right: render_asset + debug_render_primitivies

Current limitations

  • PBR shading is not implemented.
    • This is a small change that will be done after the architecture is pinned down, i.e. after this PR is merged.
  • Gfx-replay not yet supported.
  • Batch rendering not yet supported.
  • Scaling of URDF file not yet supported.

How has this been tested

Tested locally on viewer.cpp and on Habitat-Lab prototype human interactive viewer.

Types of changes

  • Docs change / refactoring / dependency upgrade
  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist

  • My code follows the code style of this project.
  • My change requires a change to the documentation.
  • I have updated the documentation accordingly.
  • I have read the CONTRIBUTING document.
  • I have completed my CLA (see CONTRIBUTING)
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@facebook-github-bot facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label Apr 18, 2023
Copy link
Contributor

@eundersander eundersander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd love to see a unit test soon with sample data. I'm curious e.g. are you supposed to strip the per-link visual elements out of the URDF if you're also attaching a skinned model.

@@ -44,6 +49,7 @@ struct RenderAssetInstanceCreationInfo {
Corrade::Containers::Optional<Magnum::Vector3> scale;
Flags flags;
std::string lightSetupKey;
std::shared_ptr<physics::ArticulatedObject> rig;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

RenderAssetInstanceCreationInfo is meant to be a lightweight object that can be easily serialized (JsonEspTypes.h), so I'm a little confused to see this member here. How does it get serialized?

Copy link
Contributor Author

@0mdc 0mdc Apr 20, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that this is not ideal.

I put the rig there because it's a straightforward way to pass it down the resource manager with minimal changes. It does not get serialized like other fields.

I'm not sold on a solution. In the spirit of making this gfx-replayable, perhaps this could evolve into a rig ID that would be registered in the resource manager, and eventually matched with a rig representation in the batch renderer.

What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah. Let's just assume this will get revisited when you integrate skinning into gfx-replay. I'm happy for this to merge as-is.

Copy link
Contributor

@eundersander eundersander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a few more minor comments. I like that articulated objects generally don't know anything about skinned models, and skinned models don't know that much about articulated objects (just pointers to SceneNodes). Nice to keep these as decoupled as possible.

src/esp/gfx/GenericDrawable.cpp Outdated Show resolved Hide resolved
lastTransform = jointTransformations[i];
} else {
// Joint not found - use last transform.
jointTransformations[i] = lastTransform;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does this failed-find happen in practice?

Copy link
Contributor Author

@0mdc 0mdc Apr 19, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This happens if the URDF doesn't perfectly match the skinned model.

For example, skinned models can have a first "root" bone that goes from the origin to the pelvis. It's also somewhat common for skinned models to have extra leaf bones.

Our rig-to-URDF tool should minimize these cases.

@0mdc 0mdc force-pushed the articulated-object-skinning branch from 2b98b73 to e96d590 Compare April 19, 2023 18:17
@0mdc 0mdc requested review from aclegg3 and mosra April 20, 2023 19:41
@0mdc 0mdc force-pushed the articulated-object-skinning branch from ef1fb4e to f34c6fb Compare April 20, 2023 20:58
@0mdc 0mdc marked this pull request as ready for review April 20, 2023 22:21
@0mdc 0mdc force-pushed the articulated-object-skinning branch from 2f8a18f to 51abdd7 Compare April 21, 2023 14:45
Copy link
Contributor

@jturner65 jturner65 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM except for the minor things with the SkinData header. This is really cool!

src/esp/gfx/SkinData.h Outdated Show resolved Hide resolved
Copy link
Contributor

@eundersander eundersander left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left some more comments.

}
ao->setJointPositions(jointPos);

checkPinholeCameraRGBAObservation(*simulator, "SimTestSkinnedAOPose.png",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Awesome! Happy to see image-based tests for this feature.

src/esp/gfx/GenericDrawable.cpp Outdated Show resolved Hide resolved
src/esp/gfx/SkinData.h Outdated Show resolved Hide resolved
src/esp/assets/ResourceManager.cpp Outdated Show resolved Hide resolved
src/esp/gfx/GenericDrawable.cpp Outdated Show resolved Hide resolved
@0mdc 0mdc merged commit 8282d14 into main Apr 24, 2023
@0mdc 0mdc deleted the articulated-object-skinning branch April 24, 2023 21:46
ykarmesh pushed a commit that referenced this pull request May 16, 2023
* skip unsupported 3D primitives (#2054)

* bugfix for setArticulatedObjectModelFilename causing garbage log output (#2053)

* Gfx-replay polish: workaround for material-overrides; new keyframe getter (#2035)

* gfx-replay polish: workaround for material-overrides; new keyframe getter API needed for a python application

* --Have pre-commit use node version compatible with Ubuntu 18.04 (#2058)

* --use system node install if present
* --change to hook-specific node version specification
* --change to nearest official release version

* Fix viewer.py framebuffer size mismatch. (#2055)

* Fix viewer.py framebuffer size mismatch.

* Change int tuples to mn.Vector2.

* Minor simplifications.

* Replay renderer: add line-rendering and unproject() (#2057)

* integrate DebugLineRender into replay renderers
* add ClassicReplayRenderer.unproject
* stub implementation for BatchReplayRenderer.unproject
* lighting hack in ClassicReplayRenderer to get ReplicaCAD stages to render correctly

* bugfix: cylinder proimitive collsion shape did not use halflength (#2060)

* --have disabled renderer test only execute 1 time (#2065)

* --(Bugfix) Fix normal transformation calc and address backface culling in case of negative scaling/reflections (#2062)

* --don't use co-factor matrix for transforming pre-calculated normals
* --address backface winding in Generic and Phong drawables

TODO :  Still need to set this up in batch renderer.
* --revert change in batch renderer

The normal matrix derivation will be changed in magnum to be appropriate, so we will keep the original verbiage for clarity.  We keep the changes in the default and PBR renderer calls since we need the determinant to determine if winding order must be reversed for backface culling.

* --test negative scaling along 3 axis
* --make copy of observation buffer for ground truth observation

The buffer is shared, so without a copy subsequent observations would overwrite the gt observation buffer.

* Update Magnum submodules to latest. (#2066)

* Update magnum submodules. (#2073)

* Add runtime perf stats for troubleshooting perf problems (#2070)

* add ResourceManager::getDrawableCountAndNumVertices, Sim.get_runtime_perf_stat_values, and related helpers

* --Refactor Semantic Mesh loading/flattening to remove deprecated functionality and improve efficiency  (#2079)

* --refactor to remove deprecated functionality
* --reviewer suggestions;
* --fixed inappropriate alloc

* Articulated object skinning (#2076)

* Add rig property to RenderAssetInstanceCreationInfo.

* Add render_asset to ao_config.json.

* Create render asset from ao_config.json when creating an articulated object.

* Store bone names into MeshTransformNode.

* Add skinning asset loading and phong rendering.

* Assorted minor fixes.

* Add flag to render articulated object primitives while having a skinned mesh for debugging.

* Add skinned articulated object test.

* Fix MetadataMediatorTest

* Cache joint transformations in drawables, other minor fixes.

* Fix test asset path + other minor fixes.

* Fix skinned mesh caching. Make ao_config.json render_asset path relative to the file. Code clean-up.

* --Add access to Scene Instance-level user defined attributes. (#2081)

* --add access to user defined in scene instance configs.

* --If SceneDataset or SceneInstance do not exist, return nullptr.


---------

Co-authored-by: Alexander Clegg <alexanderwclegg@gmail.com>

* --Don't make needless copy of scene instance attributes when retrieving user defined values (#2082)

* --don't make needless copy of scene instance attributes
* --add a test

* gfx-replay: fix to remove reflection when converting node transform matrix to rotation/translation (#2085)

* Articulated object semantics (#2086)

* Parse semantic_id from ao_config.json.

* Propagate semantic_id to skinned mesh.

* Add skinned mesh semantics test.

* Propagate semantics for non-skinned articulated object.

* --Convert materials to use magnum materials. (#2083)

* --add Mn::MaterialTools; WIP
* --address test issue;
* --convert materials to magnum materials
* --address fallback material not having defaults
* --fix many attribute accessor bugs.
* --appropriately make new attribute name with lowercase letter
* --Flat/Phong and Semantic textures work. Have to fix building PBR layers
* --improve ptr attribute naming; organize custom attribute assignment
Still need to build texture pointer attribute array with layers if they exist in base material
* --use owning material constructors
* --cleanup; clang-tidy; fix flat vs phong ambient txtr map in phong shader
Flat now has ambient Texture mapped directly from source material, so  no need to check baseColorTexture anymore.
* --get layers working properly
* --support noneRoughnessMetallic texture; handle per-vert objectID for PBR drawables
* --cache all material quantities in drawable to speed up draw access
* --prepopulate normal texture

* --address inappropriate bitflag check; add vertexID flag to PBR shader (#2090)

* enable frustum-culling for classic replay renderer (#2096)

* enable recompute_navmesh when creating sim with create_renderer==false (#2097)

* --[BUGFIX] Reverse directional lights before sending to uniform; (#2094)

* --reverse directional lights before sending to uniform;
Also correct existing light setups
* --fix lighting tutorial lights
* --correct orientations of sample  and default light with names

* Update Magnum submodules. (#2100)

* --[BUGFIX] Single channel texture support (#2102)

* --Address uncompressed single and double channel textures by modifying swizzles to be rrr1 and rrrg respectively.

* add intertiaFromURDF option to URDF loading API (#2098)

* add additional magnum importer plugins to python build dependencies

* refactor to reduce redundancy

---------

Co-authored-by: Eric Undersander <eundersander@fb.com>
Co-authored-by: John Turner <7strbass@gmail.com>
Co-authored-by: Mikaël Dallaire Côté <110583667+0mdc@users.noreply.github.com>
Co-authored-by: Alexander Clegg <alexanderwclegg@gmail.com>
Co-authored-by: Vladimír Vondruš <mosra@centrum.cz>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CLA Signed Do not delete this pull request or issue due to inactivity.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants