[Ovmm] Merge main and add magnum plugins (#2106)

* skip unsupported 3D primitives (#2054)

* bugfix for setArticulatedObjectModelFilename causing garbage log output (#2053)

* Gfx-replay polish: workaround for material-overrides; new keyframe getter (#2035)

* gfx-replay polish: workaround for material-overrides; new keyframe getter API needed for a python application

* --Have pre-commit use node version compatible with Ubuntu 18.04 (#2058)

* --use system node install if present
* --change to hook-specific node version specification
* --change to nearest official release version

* Fix viewer.py framebuffer size mismatch. (#2055)

* Fix viewer.py framebuffer size mismatch.

* Change int tuples to mn.Vector2.

* Minor simplifications.

* Replay renderer: add line-rendering and unproject() (#2057)

* integrate DebugLineRender into replay renderers
* add ClassicReplayRenderer.unproject
* stub implementation for BatchReplayRenderer.unproject
* lighting hack in ClassicReplayRenderer to get ReplicaCAD stages to render correctly

* bugfix: cylinder proimitive collsion shape did not use halflength (#2060)

* --have disabled renderer test only execute 1 time (#2065)

* --(Bugfix) Fix normal transformation calc and address backface culling in case of negative scaling/reflections (#2062)

* --don't use co-factor matrix for transforming pre-calculated normals
* --address backface winding in Generic and Phong drawables

TODO :  Still need to set this up in batch renderer.
* --revert change in batch renderer

The normal matrix derivation will be changed in magnum to be appropriate, so we will keep the original verbiage for clarity.  We keep the changes in the default and PBR renderer calls since we need the determinant to determine if winding order must be reversed for backface culling.

* --test negative scaling along 3 axis
* --make copy of observation buffer for ground truth observation

The buffer is shared, so without a copy subsequent observations would overwrite the gt observation buffer.

* Update Magnum submodules to latest. (#2066)

* Update magnum submodules. (#2073)

* Add runtime perf stats for troubleshooting perf problems (#2070)

* add ResourceManager::getDrawableCountAndNumVertices, Sim.get_runtime_perf_stat_values, and related helpers

* --Refactor Semantic Mesh loading/flattening to remove deprecated functionality and improve efficiency  (#2079)

* --refactor to remove deprecated functionality
* --reviewer suggestions;
* --fixed inappropriate alloc

* Articulated object skinning (#2076)

* Add rig property to RenderAssetInstanceCreationInfo.

* Add render_asset to ao_config.json.

* Create render asset from ao_config.json when creating an articulated object.

* Store bone names into MeshTransformNode.

* Add skinning asset loading and phong rendering.

* Assorted minor fixes.

* Add flag to render articulated object primitives while having a skinned mesh for debugging.

* Add skinned articulated object test.

* Fix MetadataMediatorTest

* Cache joint transformations in drawables, other minor fixes.

* Fix test asset path + other minor fixes.

* Fix skinned mesh caching. Make ao_config.json render_asset path relative to the file. Code clean-up.

* --Add access to Scene Instance-level user defined attributes. (#2081)

* --add access to user defined in scene instance configs.

* --If SceneDataset or SceneInstance do not exist, return nullptr.


---------

Co-authored-by: Alexander Clegg <alexanderwclegg@gmail.com>

* --Don't make needless copy of scene instance attributes when retrieving user defined values (#2082)

* --don't make needless copy of scene instance attributes
* --add a test

* gfx-replay: fix to remove reflection when converting node transform matrix to rotation/translation (#2085)

* Articulated object semantics (#2086)

* Parse semantic_id from ao_config.json.

* Propagate semantic_id to skinned mesh.

* Add skinned mesh semantics test.

* Propagate semantics for non-skinned articulated object.

* --Convert materials to use magnum materials. (#2083)

* --add Mn::MaterialTools; WIP
* --address test issue;
* --convert materials to magnum materials
* --address fallback material not having defaults
* --fix many attribute accessor bugs.
* --appropriately make new attribute name with lowercase letter
* --Flat/Phong and Semantic textures work. Have to fix building PBR layers
* --improve ptr attribute naming; organize custom attribute assignment
Still need to build texture pointer attribute array with layers if they exist in base material
* --use owning material constructors
* --cleanup; clang-tidy; fix flat vs phong ambient txtr map in phong shader
Flat now has ambient Texture mapped directly from source material, so  no need to check baseColorTexture anymore.
* --get layers working properly
* --support noneRoughnessMetallic texture; handle per-vert objectID for PBR drawables
* --cache all material quantities in drawable to speed up draw access
* --prepopulate normal texture

* --address inappropriate bitflag check; add vertexID flag to PBR shader (#2090)

* enable frustum-culling for classic replay renderer (#2096)

* enable recompute_navmesh when creating sim with create_renderer==false (#2097)

* --[BUGFIX] Reverse directional lights before sending to uniform; (#2094)

* --reverse directional lights before sending to uniform;
Also correct existing light setups
* --fix lighting tutorial lights
* --correct orientations of sample  and default light with names

* Update Magnum submodules. (#2100)

* --[BUGFIX] Single channel texture support (#2102)

* --Address uncompressed single and double channel textures by modifying swizzles to be rrr1 and rrrg respectively.

* add intertiaFromURDF option to URDF loading API (#2098)

* add additional magnum importer plugins to python build dependencies

* refactor to reduce redundancy

---------

Co-authored-by: Eric Undersander <eundersander@fb.com>
Co-authored-by: John Turner <7strbass@gmail.com>
Co-authored-by: Mikaël Dallaire Côté <110583667+0mdc@users.noreply.github.com>
Co-authored-by: Alexander Clegg <alexanderwclegg@gmail.com>
Co-authored-by: Vladimír Vondruš <mosra@centrum.cz>

.pre-commit-config.yaml

@@ -136,6 +136,7 @@ repos:
     hooks:
     -   id: eslint
         args: [--fix, --ext .html,.js]
+        language_version: 14.21.3
         additional_dependencies:
         - eslint@6.4.0
         - eslint-config-prettier@6.3.0

data/test_assets/urdf/skinned_prism.ao_config.json

@@ -0,0 +1,5 @@
+{
+    "render_asset": "../objects/skinned_prism.glb",
+    "semantic_id": 100,
+    "debug_render_primitives": false
+}

data/test_assets/urdf/skinned_prism.urdf

@@ -0,0 +1,107 @@
+<?xml version="1.0" ?>
+<robot name="skinned_prism">
+  <material name="mat">
+    <color rgba="1.0 1.0 1.0 1.0"/>
+  </material>
+
+  <link name="A">
+    <visual>
+      <material name="mat"/>
+      <origin rpy="0 0 0" xyz="0 0 0"/>
+      <geometry>
+        <box size="0.05 0.05 0.05"/>
+      </geometry>
+    </visual>
+    <collision>
+      <origin rpy="0 0 0" xyz="0 0 0"/>
+      <geometry>
+        <box size="0.05 0.05 0.05"/>
+      </geometry>
+    </collision>
+  </link>
+
+  <link name="B">
+    <visual>
+      <material name="mat"/>
+      <origin rpy="0 0 0" xyz="0 0 0"/>
+      <geometry>
+        <box size="0.05 0.05 0.05"/>
+      </geometry>
+    </visual>
+    <collision>
+      <origin rpy="0 0 0" xyz="0 0 0"/>
+      <geometry>
+        <box size="0.05 0.05 0.05"/>
+      </geometry>
+    </collision>
+  </link>
+
+  <link name="C">
+    <visual>
+      <material name="mat"/>
+      <origin rpy="0 0 0" xyz="0 0 0"/>
+      <geometry>
+        <box size="0.05 0.05 0.05"/>
+      </geometry>
+    </visual>
+    <collision>
+      <origin rpy="0 0 0" xyz="0 0 0"/>
+      <geometry>
+        <box size="0.05 0.05 0.05"/>
+      </geometry>
+    </collision>
+  </link>
+
+  <link name="D">
+    <visual>
+      <material name="mat"/>
+      <origin rpy="0 0 0" xyz="0 0 0"/>
+      <geometry>
+        <box size="0.05 0.05 0.05"/>
+      </geometry>
+    </visual>
+    <collision>
+      <origin rpy="0 0 0" xyz="0 0 0"/>
+      <geometry>
+        <box size="0.05 0.05 0.05"/>
+      </geometry>
+    </collision>
+  </link>
+
+  <link name="E">
+    <visual>
+      <material name="mat"/>
+      <origin rpy="0 0 0" xyz="0 0 0"/>
+      <geometry>
+        <box size="0.05 0.05 0.05"/>
+      </geometry>
+    </visual>
+    <collision>
+      <origin rpy="0 0 0" xyz="0 0 0"/>
+      <geometry>
+        <box size="0.05 0.05 0.05"/>
+      </geometry>
+    </collision>
+  </link>
+
+  <joint name="B" type="spherical">
+    <parent link="A"/>
+    <child link="B"/>
+    <origin xyz="0 0 1.5" rpy="0 0 0"/>
+  </joint>
+  <joint name="C" type="spherical">
+    <parent link="B"/>
+    <child link="C"/>
+    <origin xyz="0 0 3" rpy="0 0 0"/>
+  </joint>
+  <joint name="D" type="spherical">
+    <parent link="C"/>
+    <child link="D"/>
+    <origin xyz="0 0 4.5" rpy="0 0 0"/>
+  </joint>
+  <joint name="E" type="spherical">
+    <parent link="D"/>
+    <child link="E"/>
+    <origin xyz="0 0 6" rpy="0 0 0"/>
+  </joint>
+</robot>

examples/tutorials/lighting_tutorial.py

@@ -149,7 +149,7 @@ def main(show_imgs=True, save_imgs=False):
 
     # create a custom light setup
     my_default_lighting = [
-        LightInfo(vector=[2.0, 2.0, 1.0, 0.0], model=LightPositionModel.Camera)
+        LightInfo(vector=[-2.0, -2.0, -1.0, 0.0], model=LightPositionModel.Camera)
     ]
     # overwrite the default DEFAULT_LIGHTING_KEY light setup
     sim.set_light_setup(my_default_lighting)
@@ -208,7 +208,7 @@ def main(show_imgs=True, save_imgs=False):
     # create a new setup with an additional light
     new_light_setup = existing_light_setup + [
         LightInfo(
-            vector=[0.0, 0.0, 1.0, 0.0],
+            vector=[0.0, 0.0, -1.0, 0.0],
             color=[1.6, 1.6, 1.4],
             model=LightPositionModel.Camera,
         )

examples/viewer.py

@@ -53,24 +53,25 @@ def __init__(self, sim_settings: Dict[str, Any]) -> None:
         )
 
         # Compute environment camera resolution based on the number of environments to render in the window.
-        surface_size: tuple(int, int) = (
+        window_size: mn.Vector2 = (
             self.sim_settings["window_width"],
             self.sim_settings["window_height"],
         )
-        grid_size: tuple(int, int) = ReplayRenderer.environment_grid_size(self.num_env)
-        camera_resolution: tuple(int, int) = (
-            surface_size[0] / grid_size[0],
-            surface_size[1] / grid_size[1],
-        )
-        self.sim_settings["width"] = camera_resolution[0]
-        self.sim_settings["height"] = camera_resolution[1]
 
         configuration = self.Configuration()
         configuration.title = "Habitat Sim Interactive Viewer"
-        configuration.size = surface_size
+        configuration.size = window_size
         Application.__init__(self, configuration)
         self.fps: float = 60.0
 
+        # Compute environment camera resolution based on the number of environments to render in the window.
+        grid_size: mn.Vector2i = ReplayRenderer.environment_grid_size(self.num_env)
+        camera_resolution: mn.Vector2 = mn.Vector2(self.framebuffer_size) / mn.Vector2(
+            grid_size
+        )
+        self.sim_settings["width"] = camera_resolution[0]
+        self.sim_settings["height"] = camera_resolution[1]
+
         # draw Bullet debug line visualizations (e.g. collision meshes)
         self.debug_bullet_draw = False
         # draw active contact point debug line visualizations
@@ -138,11 +139,12 @@ def __init__(self, sim_settings: Dict[str, Any]) -> None:
         # text object transform in window space is Projection matrix times Translation Matrix
         # put text in top left of window
         self.window_text_transform = mn.Matrix3.projection(
-            mn.Vector2(surface_size)
+            self.framebuffer_size
         ) @ mn.Matrix3.translation(
-            mn.Vector2(
-                surface_size[0] * -HabitatSimInteractiveViewer.TEXT_DELTA_FROM_CENTER,
-                surface_size[1] * HabitatSimInteractiveViewer.TEXT_DELTA_FROM_CENTER,
+            mn.Vector2(self.framebuffer_size)
+            * mn.Vector2(
+                -HabitatSimInteractiveViewer.TEXT_DELTA_FROM_CENTER,
+                HabitatSimInteractiveViewer.TEXT_DELTA_FROM_CENTER,
             )
         )
         self.shader = shaders.VectorGL2D()
@@ -558,13 +560,31 @@ def key_press_event(self, event: Application.KeyEvent) -> None:
                 self.cached_urdf = urdf_file_path
                 aom = self.sim.get_articulated_object_manager()
                 ao = aom.add_articulated_object_from_urdf(
-                    urdf_file_path, fixed_base, 1.0, 1.0, True
+                    urdf_file_path,
+                    fixed_base,
+                    1.0,
+                    1.0,
+                    True,
+                    maintain_link_order=False,
+                    intertia_from_urdf=False,
                 )
                 ao.translation = (
                     self.default_agent.scene_node.transformation.transform_point(
                         [0.0, 1.0, -1.5]
                     )
                 )
+                # check removal and auto-creation
+                joint_motor_settings = habitat_sim.physics.JointMotorSettings(
+                    position_target=0.0,
+                    position_gain=1.0,
+                    velocity_target=0.0,
+                    velocity_gain=1.0,
+                    max_impulse=1000.0,
+                )
+                existing_motor_ids = ao.existing_joint_motor_ids
+                for motor_id in existing_motor_ids:
+                    ao.remove_joint_motor(motor_id)
+                ao.create_all_motors(joint_motor_settings)
             else:
                 logger.warn("Load URDF: input file not found. Aborting.")
 

src/cmake/FindMagnum.cmake

@@ -128,7 +128,7 @@
 #
 # Features of found Magnum library are exposed in these variables:
 #
-#  MAGNUM_BUILD_DEPRECATED      - Defined if compiled with deprecated APIs
+#  MAGNUM_BUILD_DEPRECATED      - Defined if compiled with deprecated features
 #   included
 #  MAGNUM_BUILD_STATIC          - Defined if compiled as static libraries
 #  MAGNUM_BUILD_STATIC_UNIQUE_GLOBALS - Defined if static libraries keep the
@@ -910,7 +910,7 @@ foreach(_component ${Magnum_FIND_COMPONENTS})
 
         # SceneTools library
         elseif(_component STREQUAL SceneTools)
-            set(_MAGNUM_${_COMPONENT}_INCLUDE_PATH_NAMES FlattenMeshHierarchy.h)
+            set(_MAGNUM_${_COMPONENT}_INCLUDE_PATH_NAMES FlattenTransformationHierarchy.h)
 
         # ShaderTools library
         elseif(_component STREQUAL ShaderTools)

src/cmake/FindMagnumPlugins.cmake

@@ -39,6 +39,7 @@
 #  PngImporter                  - PNG importer
 #  PrimitiveImporter            - Primitive importer
 #  SpirvToolsShaderConverter    - SPIR-V Tools shader converter
+#  SpngImporter                 - PNG importer using libspng
 #  StanfordImporter             - Stanford PLY importer
 #  StanfordSceneConverter       - Stanford PLY converter
 #  StbDxtImageConverter         - BC1/BC3 image compressor using stb_dxt
@@ -168,10 +169,10 @@ set(_MAGNUMPLUGINS_PLUGIN_COMPONENTS
     JpegImporter KtxImageConverter KtxImporter MeshOptimizerSceneConverter
     MiniExrImageConverter OpenExrImageConverter OpenExrImporter
     OpenGexImporter PngImageConverter PngImporter PrimitiveImporter
-    SpirvToolsShaderConverter StanfordImporter StanfordSceneConverter
-    StbDxtImageConverter StbImageConverter StbImageImporter
-    StbResizeImageConverter StbTrueTypeFont StbVorbisAudioImporter StlImporter
-    UfbxImporter WebPImporter)
+    SpirvToolsShaderConverter SpngImporter StanfordImporter
+    StanfordSceneConverter StbDxtImageConverter StbImageConverter
+    StbImageImporter StbResizeImageConverter StbTrueTypeFont
+    StbVorbisAudioImporter StlImporter UfbxImporter WebPImporter)
 # Nothing is enabled by default right now
 set(_MAGNUMPLUGINS_IMPLICITLY_ENABLED_COMPONENTS )
 
@@ -471,6 +472,12 @@ foreach(_component ${MagnumPlugins_FIND_COMPONENTS})
             set_property(TARGET MagnumPlugins::${_component} APPEND PROPERTY
                 INTERFACE_LINK_LIBRARIES SpirvTools::SpirvTools SpirvTools::Opt)
 
+        # SpngImporter plugin dependencies
+        elseif(_component STREQUAL SpngImporter)
+            find_package(Spng REQUIRED)
+            set_property(TARGET MagnumPlugins::${_component} APPEND PROPERTY
+                INTERFACE_LINK_LIBRARIES Spng::Spng)
+
         # StanfordImporter has no dependencies
         # StanfordSceneConverter has no dependencies
         # StbDxtImageConverter has no dependencies

src/cmake/dependencies.cmake

@@ -199,7 +199,9 @@ if(NOT USE_SYSTEM_MAGNUM)
 
   # These are enabled by default but we don't need them for anything yet
   set(MAGNUM_WITH_SHADERTOOLS OFF CACHE BOOL "" FORCE)
-  set(MAGNUM_WITH_MATERIALTOOLS OFF CACHE BOOL "" FORCE)
+  # These used to be disabled here but now aren't, explicitly enable them to
+  # update options in existing builds
+  set(MAGNUM_WITH_MATERIALTOOLS ON CACHE BOOL "" FORCE)
 
   # These are enabled by default but we don't need them if not building GUI
   # viewers -- disabling for slightly faster builds. If you need any of these
@@ -347,16 +349,25 @@ if(NOT USE_SYSTEM_MAGNUM)
     # Make Magnum text rendering plugins (used by the native viewer) available
     # for Python as well; and reset that back to strange build procedures that
     # turn some features off again later can still work.
+    set(
+      common_plugins
+      Magnum::AnyImageConverter
+      Magnum::AnyImageImporter
+      Magnum::AnySceneImporter
+      MagnumPlugins::AssimpImporter
+      MagnumPlugins::BasisImporter
+      MagnumPlugins::GltfImporter
+      MagnumPlugins::StanfordImporter
+      MagnumPlugins::StbImageConverter
+      MagnumPlugins::StbImageImporter
+    )
     if(BUILD_GUI_VIEWERS)
-      set(MAGNUM_PYTHON_BINDINGS_STATIC_PLUGINS
-          MagnumPlugins::StbTrueTypeFont Magnum::AnySceneImporter
-          MagnumPlugins::AssimpImporter CACHE STRING "" FORCE
-      )
-    else()
-      set(MAGNUM_PYTHON_BINDINGS_STATIC_PLUGINS Magnum::AnySceneImporter
-                                                MagnumPlugins::AssimpImporter
+      set(MAGNUM_PYTHON_BINDINGS_STATIC_PLUGINS ${common_plugins}
+                                                MagnumPlugins::StbTrueTypeFont
           CACHE STRING "" FORCE
       )
+    else()
+      set(MAGNUM_PYTHON_BINDINGS_STATIC_PLUGINS ${common_plugins} CACHE STRING "" FORCE)
     endif()
     add_subdirectory("${DEPS_DIR}/magnum-bindings")
   endif()

src/esp/assets/CMakeLists.txt

@@ -31,6 +31,7 @@ find_package(
   AnyImageImporter
   AnySceneImporter
   GL
+  MaterialTools
   MeshTools
   SceneGraph
   SceneTools
@@ -82,7 +83,7 @@ target_link_libraries(
          MagnumPlugins::StanfordImporter
          MagnumPlugins::StbImageImporter
          MagnumPlugins::StbImageConverter
-  PRIVATE geo io
+  PRIVATE geo io Magnum::MaterialTools
 )
 
 if(BUILD_WITH_VHACD)

src/esp/assets/MeshMetaData.h

@@ -45,6 +45,9 @@ struct MeshTransformNode {
 
   /** @brief Default constructor. */
   MeshTransformNode() = default;
+
+  /** @brief Node name in the original file. */
+  std::string name{};
 };
 
 /**
@@ -74,6 +77,10 @@ struct MeshMetaData {
   std::pair<start, end> textureIndex =
       std::make_pair(ID_UNDEFINED, ID_UNDEFINED);
 
+  /** @brief Index range (inclusive) of skin data for the asset in the global
+   * asset datastructure. */
+  std::pair<start, end> skinIndex = std::make_pair(ID_UNDEFINED, ID_UNDEFINED);
+
   /** @brief The root of the mesh component transformation hierarchy tree which
    * stores the relationship between components of the asset.*/
   MeshTransformNode root;
@@ -116,6 +123,19 @@ struct MeshMetaData {
     textureIndex.second = textureEnd;
   }
 
+  /**
+   * @brief Sets the skin indices for the asset. See @ref
+   * ResourceManager::skins_.
+   * @param skinStart First index for asset skin data in the global
+   * skin datastructure.
+   * @param skinEnd Final index for asset skin data in the global skin
+   * datastructure.
+   */
+  void setSkinIndices(int skinStart, int skinEnd) {
+    skinIndex.first = skinStart;
+    skinIndex.second = skinEnd;
+  }
+
   /**
    * @brief Set the root frame orientation based on passed frame
    * @param frame target frame in world space