HITL: rearrange_v2 example app and other features/fixes for Feb 12 de…

…mo (#1810) * nicer instructions for installing Magnum on Mac * add MVP rearrange_v2 HITL app * add GuiPlacementHelper * add ClientHelper * changes to GuiPickHelper. * add config habitat_hitl.debug_line_width * changes to how connection_record/connection_params are handled * add config habitat_hitl.networking.wait_for_app_ready_signal * support for tracking client latency * add color and billboard-versus-up-facing options for highlight messages * minor bugfix to unity_dataset_processing.py * add hydra-core to requirements.txt
facebookresearch · Feb 20, 2024 · 03de61e · 03de61e
1 parent 5e2c0bb
commit 03de61e
Show file tree

Hide file tree

Showing 26 changed files with 1,283 additions and 249 deletions.
diff --git a/examples/hitl/pick_throw_vr/README.md b/examples/hitl/pick_throw_vr/README.md
@@ -19,7 +19,8 @@ This is an example HITL application that allows a user to interact with a scene,
   - [Launch Command](#launch-command)
   - [Unity Data Folder](#unity-data-folder)
     - [Dataset Processing](#dataset-processing)
-      - [Installation](#installation-1)
+      - [Magnum Installation for Mac](#magnum-installation-for-mac)
+      - [Magnum Installation for Linux](#magnum-installation-for-linux)
       - [Usage](#usage)
       - [Importing Data Into Unity](#importing-data-into-unity)
   - [Running Locally from Unity](#running-locally-from-unity)
@@ -101,13 +102,23 @@ Therefore, a script is provided so that you can process your datasets and add th
 
 ### Dataset Processing
 
-#### Installation
+The dataset processing script requires latest Magnum binaries, which should be installed separately from Habitat as described below.
 
-This step requires [Magnum](https://github.com/mosra/magnum) data processing tools. They are not built along with Habitat.
+#### Magnum Installation for Mac
+
+Magnum is easiest to install on Mac via Homebrew.
+
+1. Follow [magnum-bindings installation instructions](https://doc.magnum.graphics/python/building/#homebrew-formulas-for-macos).
+   * In addition to `corrade`, `magnum`, and `magnum-bindings`, you may need `magnum-plugins`.
+2. Test your install: `python -c "from magnum import math, meshtools, scenetools, trade"`.
+   * Beware homebrew installs python packages like magnum to its own Python location, not your current conda environment's Python.
+   * Depending on how Homebrew has installed Python, you may need to use `python3` instead of `python`.
+
+#### Magnum Installation for Linux
 
 It is recommended that you create a new `conda` environment so that it can be reused in the future without interfering with Habitat.
 
-1. Get the latest `magnum-tools`.
+1. Install magnum binaries for Linux.
    * Navigate to the [Magnum CI](https://github.com/mosra/magnum-ci/actions/workflows/magnum-tools.yml).
    * Select the latest green workflow run.
    * Scroll down to "Artifacts".
@@ -117,19 +128,19 @@ It is recommended that you create a new `conda` environment so that it can be re
 ```
 conda create --name magnum python=3.10
 ```
-3. Navigate to the `site-packages` of your new environment, e.g. `~/anaconda/envs/magnum/lib/python3.10/site-packages/`.
-4. Create a `magnum.pth` file in this directory.
-5. Add the absolute path to `magnum-tools`'s `python` folder to this file, e.g:
+1. Navigate to the `site-packages` of your new environment, e.g. `~/anaconda/envs/magnum/lib/python3.10/site-packages/`.
+2. Create a `magnum.pth` file in this directory.
+3. Add the absolute path to `magnum-tools`'s `python` folder to this file, e.g:
 ```
 /home/USER/Documents/magnum-tools/linux-x64/python/
 ```
-6. The Magnum libraries will now be included upon activating your `magnum` environment. You may validate by assessing that the following commands don't return errors:
+1. The Magnum libraries will now be included upon activating your `magnum` environment. You may validate by assessing that the following commands don't return errors:
    * `conda activate magnum`
    * `python -c "from magnum import math, meshtools, scenetools, trade"`
 
 #### Usage
 
-To process the dataset, activate your `magnum` conda environment and navigate to your `habitat-lab` root directory. Run the following command:
+To process the dataset, navigate to your `habitat-lab` root directory. Run the following command:
 
 ```
 python ./scripts/unity_dataset_processing/unity_dataset_processing.py \

diff --git a/examples/hitl/pick_throw_vr/pick_throw_vr.py b/examples/hitl/pick_throw_vr/pick_throw_vr.py
@@ -23,7 +23,6 @@
     GuiHumanoidController,
 )
 from habitat_hitl.environment.gui_navigation_helper import GuiNavigationHelper
-from habitat_hitl.environment.gui_pick_helper import GuiPickHelper
 from habitat_hitl.environment.gui_throw_helper import GuiThrowHelper
 from habitat_hitl.environment.hablab_utils import (
     get_agent_art_obj_transform,
@@ -81,11 +80,6 @@ def __init__(self, app_service: AppService):
         self._throw_helper = GuiThrowHelper(
             self._app_service, self.get_gui_controlled_agent_index()
         )
-        self._pick_helper = GuiPickHelper(
-            self._app_service,
-            self.get_gui_controlled_agent_index(),
-            self._get_gui_agent_feet_height(),
-        )
 
         self._avatar_switch_helper = AvatarSwitcher(
             self._app_service, self._gui_agent_ctrl
@@ -128,9 +122,6 @@ def on_environment_reset(self, episode_recorder_dict):
         self._target_obj_ids = sim._scene_obj_ids
 
         self._nav_helper.on_environment_reset()
-        self._pick_helper.on_environment_reset(
-            agent_feet_height=self._get_gui_agent_feet_height()
-        )
 
         self._camera_helper.update(self._get_camera_lookat_pos(), dt=0)
         self._count_tsteps_stop = 0
@@ -393,7 +384,7 @@ def _update_grasping_and_set_act_hints_local(self):
             # check for new grasp and call gui_agent_ctrl.set_act_hints
             if self._held_target_obj_idx is None:
                 assert not self._gui_agent_ctrl.is_grasped
-                translation = self._get_gui_agent_translation()
+                translation = self._gui_agent_ctrl.get_base_translation()
 
                 min_dist = self._can_grasp_place_threshold
                 min_i = None
@@ -542,19 +533,13 @@ def _viz_objects(self):
     def get_gui_controlled_agent_index(self):
         return self._gui_agent_ctrl._agent_idx
 
-    def _get_gui_agent_translation(self):
-        assert isinstance(self._gui_agent_ctrl, GuiHumanoidController)
-        return (
-            self._gui_agent_ctrl._humanoid_controller.obj_transform_base.translation
-        )
-
     def _get_gui_agent_feet_height(self):
         assert isinstance(self._gui_agent_ctrl, GuiHumanoidController)
         base_offset = (
             self._gui_agent_ctrl.get_articulated_agent().params.base_offset
         )
         agent_feet_translation = (
-            self._get_gui_agent_translation() + base_offset
+            self._gui_agent_ctrl.get_base_translation() + base_offset
         )
         return agent_feet_translation[1]
 

diff --git a/examples/hitl/rearrange_v2/README.md b/examples/hitl/rearrange_v2/README.md
@@ -0,0 +1,50 @@
+# Rearrange_v2 HITL application
+
+SIRo's 2024H1 data-collection app. Work in progress.
+
+## Installation
+See [habitat-hitl/README.md](../../../habitat-hitl/README.md).
+
+In addition to the core HITL data above, you need the private [fphab](https://huggingface.co/datasets/fpss/fphab) repo. Reach out to the SIRo team for access.
+```
+cd path/to/habitat-lab/data
+git clone --branch articulated-scenes --single-branch --depth 1 https://huggingface.co/datasets/fpss/fphab
+mv fphab fpss
+```
+
+## Data directory
+
+Run `rearrange_v2` from the Habitat-lab root directory. It will expect `data/` for Habitat-lab data, and it will also look for `examples/hitl/rearrange_v2/app_data/demo.json.gz` (included alongside source files in our git repo).
+
+## Example launch commands
+
+Local testing (local display and keyboard control):
+```bash
+python examples/hitl/rearrange_v2/rearrange_v2.py
+```
+
+Headed server (includes local display and keyboard control):
+```bash
+python examples/hitl/rearrange_v2/rearrange_v2.py habitat_hitl.networking.enable=True
+```
+
+Headless server:
+```bash
+python examples/hitl/rearrange_v2/rearrange_v2.py +experiment=headless_server
+```
+
+## Controls
+See on-screen help text. In addition, press `1` or `2` to select an episode.
+
+## Configuration
+See `config/rearrange_v2.yaml` and `config/experiment/headless_server.yaml`.
+
+## Browser client
+
+`rearrange_v2` has additional requirements for the [Unity VR client](../pick_throw_vr/README.md#vr).
+* Beware these instructions are a work in progress!
+* It is designed for use with a desktop browser-based (not VR) version of the Unity client.
+* Use the [`webgl-demo`](https://github.com/eundersander/siro_hitl_unity_client/tree/webgl-demo) branch.
+* Download Unity data folder (`data.zip`) from: https://drive.google.com/drive/folders/12VLJGf5_ntr5nztZn1KjzyzBG_UKDPme
+* Extract `data.zip` to Unity client's `Assets/Resources/data`.
+* Open and run the `PlayerMouseKeyboard` scene.
diff --git a/examples/hitl/rearrange_v2/app_data/demo.json.gz b/examples/hitl/rearrange_v2/app_data/demo.json.gz
diff --git a/examples/hitl/rearrange_v2/config/disable_spot_pick_and_move_backward.yaml b/examples/hitl/rearrange_v2/config/disable_spot_pick_and_move_backward.yaml
@@ -0,0 +1,24 @@
+# @package _global_
+
+habitat:
+  task:
+    actions:
+      agent_0_arm_action:
+        disable_grip: true
+
+habitat_baselines:
+  rl:
+    policy:
+      agent_0:
+        hierarchical_policy:
+          high_level_policy:
+            allowed_actions:
+              - nav_to_goal
+              - nav_to_obj
+              # - pick
+              # - place
+              - nav_to_receptacle_by_name
+              - turn_left
+              - turn_right
+              - move_forward
+              # - move_backward
diff --git a/examples/hitl/rearrange_v2/config/experiment/headless_server.yaml b/examples/hitl/rearrange_v2/config/experiment/headless_server.yaml
@@ -0,0 +1,12 @@
+# @package _global_
+
+habitat_hitl:
+  window: ~
+  networking:
+    enable: True
+    http_availability_server:
+      enable: True
+    client_max_idle_duration: 30.0
+  experimental:
+    headless:
+      do_headless: True
diff --git a/examples/hitl/rearrange_v2/config/rearrange_v2.yaml b/examples/hitl/rearrange_v2/config/rearrange_v2.yaml
@@ -0,0 +1,43 @@
+# @package _global_
+
+defaults:
+  - social_rearrange: pop_play
+  - hitl_defaults
+  - disable_spot_pick_and_move_backward
+  - _self_
+
+habitat:
+  # various config args to ensure the episode never ends
+  environment:
+    max_episode_steps: 0
+    iterator_options:
+      # For the demo, we want to showcase the episodes in the specified order
+      shuffle: False
+  task:
+    measurements:
+      rearrange_cooperate_reward:
+        end_on_collide: False
+      pddl_success:
+        must_call_stop: False
+  dataset:
+    data_path: examples/hitl/rearrange_v2/app_data/demo.json.gz
+
+habitat_baselines:
+  # todo: document these choices
+  eval:
+    should_load_ckpt: False
+  rl:
+    agent:
+      num_pool_agents_per_type: [1,1]
+
+habitat_hitl:
+  window:
+    title: "Rearrange"
+    width: 1300
+    height: 1000
+  gui_controlled_agent:
+    agent_index: 1
+    ang_speed: 300
+  hide_humanoid_in_gui: True
+  camera:
+    first_person_mode: True