Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

OVMM: Merge episode generation updates from main #1313

Merged
merged 15 commits into from
May 11, 2023
Merged

OVMM: Merge episode generation updates from main #1313

merged 15 commits into from
May 11, 2023

Conversation

JHurricane96
Copy link

@JHurricane96 JHurricane96 commented May 4, 2023 •
edited
Loading

Motivation and Context

Merging following PRs:

#1108, #1248, #1249, #1298, #1301

How Has This Been Tested

Types of changes

  • [Docs change] Addition or changes to the documentation
  • [Refactoring] Large changes to the code that improve its functionality or performance
  • [Dependency Upgrade] Upgrades one or several dependencies in habitat
  • [Bug Fix] (non-breaking change which fixes an issue)
  • [Development] A pull request that add new features to the habitat-lab task and environment codebase. Development Pull Requests must be small (less that 500 lines of code change), have unit testing, very extensive documentation and examples. These are typically new tasks, environments, sensors, etc... The review process for these Pull Request is longer because these changes will be maintained by our core team of developers, so make sure your changes are easy to understand!
  • [Experiment] A pull request that add new features to the habitat-baselines training codebase. Experiments Pull Requests can be any size, must have smoke/integration tests and be isolated from the rest of the code. Your code additions should not rely on other habitat-baselines code. This is to avoid dependencies between different parts of habitat-baselines. You must also include a README that will document how to use your new feature or trainer. You will be the maintainer of this code, if the code becomes stale or is not supported often enough, we will eventually remove it.

Checklist

  • My code follows the code style of this project.
  • I have updated the documentation if required.
  • I have read the CONTRIBUTING document.
  • I have completed my CLA (see CONTRIBUTING)
  • I have added tests to cover my changes if required.

@facebook-github-bot facebook-github-bot added the CLA Signed Do not delete this pull request or issue due to inactivity. label May 4, 2023
aclegg3 and others added 9 commits May 4, 2023 18:16
@aclegg3 @JHurricane96
TriangleMeshReceptacles and RearrangeGenerator Improvements (facebook…
2a69d1a 
…research#1108)

* updates to rearrange episode generation to support mesh receptacles and improved debugging

* docstrings and typing for DebugVisualizer util

* mesh receptacle debug_draw with global transforms

* add some tests for sampling accuracy

* bugfix - use default_sensor_uid instead of hardcoded "rgb" in DebugVisualizer

* adjust debug peek target to bb center to better capture objects not centered at COM

* add debug circles to the peek API

* add peek_scene variant for easliy getting images of a full scene or stage.
@aclegg3 @JHurricane96
RearrangeGenerator cleanup and unstable object culling (facebookresea…
344791e 
…rch#1248)

* fix unstable object culling compatibility with target sampling and change default to use this feature

* add comments and typing

* fix bug where only the first of a target sampler's object samplers was used
@aclegg3 @JHurricane96
Add support for glTF multi-receptacle assets (facebookresearch#1249)
01c6424 
* add support for glTF multi-receptacle assets with each sub-mesh instantiating a unique Receptacle.

* update receptacle names in test for new code

* refactor to use Meshdata internally

* convert triangle-like primitives to triangles.
@aclegg3 @JHurricane96
refactor to use Receptacle unique_name for sample filtering. Add supp…
73ba82c 
…ort for filter files configured in scene instance config user_defined fields.
@aclegg3 @JHurricane96
refactor snapping functionality to improve stability by estimating co…
c80c656 
…nsistency of support surface from bb corners projected in gravity direction
@aclegg3 @Skylion007 @JHurricane96
Update habitat-lab/habitat/sims/habitat_simulator/sim_utilities.py
209708e 
Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>
@aclegg3 @JHurricane96
turn off additional stability culling for now
3bcfd05
@aclegg3 @JHurricane96
bugfix
11595d1
@aclegg3 @JHurricane96
fix margin issues
9d3b537
@Skylion007
Copy link
Contributor

Ah, sorry saw thi is a merge PR, ignore my comments.

@JHurricane96 JHurricane96 requested review from yvsriram and ykarmesh May 10, 2023 17:08
@ykarmesh ykarmesh changed the base branch from object_rearrange to main May 10, 2023 20:21
@ykarmesh ykarmesh changed the base branch from main to object_rearrange May 10, 2023 20:21
@JHurricane96 JHurricane96 merged commit b0f38ec into facebookresearch:object_rearrange May 11, 2023
@JHurricane96 JHurricane96 deleted the ep-gen-merge branch May 11, 2023 15:52
yvsriram added a commit to yvsriram/habitat-lab that referenced this pull request May 13, 2023
@yvsriram
Revert "OVMM: Merge episode generation updates from main (facebookres…
d32dc06 
…earch#1313)"

This reverts commit b0f38ec.
yvsriram added a commit to yvsriram/habitat-lab that referenced this pull request May 14, 2023
@yvsriram
Revert "Revert "OVMM: Merge episode generation updates from main (fac…
95f7453 
…ebookresearch#1313)""

This reverts commit d32dc06.
yvsriram added a commit that referenced this pull request Jun 16, 2023
@yvsriram @JHurricane96
@jimmytyyang @ykarmesh
[OVMM] Fix issues in place and gaze training (#1344)
5850901 
* Category nav skill updates

- Remove rgb sensor and disable textures
- Add object segmentation sensor
- Enable einematic mode
- Add new receptacle segmentation sensor
- Adjust robot radius
- Switch to discrete actions, fix discrete actions for hab 2.0
- OmegaConf speed fixes
- Spawn agent on active navmesh island
- Start episodes with retracted agent arm

* Remove unused import

* Bug fixes

* remove prefix

* add gaze_policy

* update stretch urdf file

* add gaze

* remove semantic

* update stretch and spot yaml

* update the config

* robot api changes, add config files, use semantic sensor

* use abs obj idx

* fixes to grasp action

* remove debug statements

* terminate on wrong grasp

* Disable auto sleep in kinematic mode

Misc formatting fixes

* waypoint controller, config for terminate on wrong grasp

* distance from base

* fix and viz center

* gaze relaxed centering condition, reward update

* add reward scaling

* Episode generation and iteration changes

Training:
- New episode iterator to lazily load parts of an episode and reduce RAM usage
- Load navmesh from correct path

Episode generation stage 1:
- Add microtrain, minitrain and minival scenes to episode generator config
- Add dynamic object sampling based on available receptacle area
- Changes to receptacle navigability filtering logic
- Add flag to generate episodes for given scene only

Viewpoints, Episode generation stage 2:
- Fix category to ID map instead of varying based on objects in episode
- Add hard rearrangement object goals to episodes by default
- Cache receptacles and viewpoints during first step of episode generation
- Allow camera to tilt up/down once while generating viewpoints
- Add debug top-down map visualizations for viewpoints
- Set number of episodes to generate in modify episodes script

* revert hack made for missing objects

* fix camera angle reward

* fix camera_angle_reward

* update success condition

* dense drop penalty and change grasp threshold

* updated radius/height

* add option for using deterministic actions

* goal receptacle segmentation for place, waypoint 2 velocity controller update

* spawning using view_points

* support spawning on viewpoints

* update paths

* update configs

* gaze: spawn wrt viewpoints of object rather than start receptacle

* allow changing tilt for gaze

* added path to fphab

* solve bug in cat_nav_to_rec

* optional viewpoint caching

* update height of the camera

* updated the size of images created for the sensors

* make camera images rectangular and small

* Revert "OVMM: Merge episode generation updates from main (#1313)"

This reverts commit b0f38ec.

* change initial state for  gaze andand non-square resolution

* add grip action threshold

* Revert "Revert "OVMM: Merge episode generation updates from main (#1313)""

This reverts commit d32dc06.

* Revert "cache observations in RearrangeTask"

This reverts commit f189156.

* update config params

* place: add stability check

* set a picked_obj_idx whenever object gets snapped

* minor fix

* update configs

* place manip_mode action, min delta_pos, navmesh violate penalty

* add stretch height param in cat_ configs

* remove arm absolute stretch action

* rename delta_pos_limit in play_stretch.yaml

* add fpmodels to cat_nav_to_rec

* change minitrain to train, add drop pen to config

* refactor ovmm place sensors, use ee success thresh in place reward

* add reward for stability, update ovmm.yaml

* add object embeddings to place's observation space

* constraint base in manip mode

* add spawning and code for static place

* add object embedding sensor to benchmark/static_place

* gaze: update gym_obs_keys and resolution

* gaze: update gym_obs_keys

* remove all instances of goal_type

* [OVMM] Karmeshyadav/nav to obj (#1306)

* modified config for nav to rec to match overall to that of nav to obj

* change actions

* updated the nav to rec task config

* temporary visualization fix

* updated paths according to sriram's paths

* update camera height

* make images smaller for faster training

* change the nav to obj img size

* bugfixes after zoom call

* action related changes

* solve setter bug

* change from cat to ovmm

* add tilt angle

* reduce nav to pos success distance

---------

Co-authored-by: Karmesh Yadav <karmeshyadav@fb.com>
Co-authored-by: Sriram Yenamandra <yvsriram99@gmail.com>

* gaze: update config defaults

* snap to active island

* [OVMM] Nav Fixes (#1381)

* Fix semantic ID in recep seg sensor

Make semantic IDs start from 1 instead of 0

* Add teleport action to cat nav

* add recep categories to ovmm yaml

* update configs, hardcode picked idx (TODO)

* add tolerance to allow snapped points

* update range of teleport in ovmm

* added local changes for visualization

* always set start_rotation

* Update habitat-baselines/habitat_baselines/config/rearrange/rl_skill.yaml

Co-authored-by: Karmesh Yadav <ykarmesh@gmail.com>

* remove rearrange generator code

* Update habitat-lab/habitat/datasets/pointnav/pointnav_dataset.py

Co-authored-by: Arun Ramachandran <arun.jhurricane@gmail.com>

* Delete floorplanner.yaml

* Ovmm: RL skill updates and metrics (#1386)

* setting camera tilt and pan correctly; update tilt for static place

* fix relative gps and compass sensors

* fix change in nav/nav.py

* add blank out probability to nav seg sensors

* add blank out prob to configs

* rename functions

* pushing remaining renames, set tilt correctly in nav

* add start receptacle sensors for gaze

* update reward to allow time for objects to reach receptacle

* update camera tilt

* collision related changes

* set state for navmesh violation inside discrete actions

* add penalty for discrete move, use task state for navmesh violation

* reset navmesh violated in task step - now False if no action

* habitat-lab/habitat/tasks/rearrange/sub_tasks/pick_task.py

* update to v4

* add all new metrics: stage successes, place/pick success criterion

* use candidate objects hard

* remove  joint from nav,  is_holding from gaze

* handle resized sensors correctly

* add local changes: v4 upgrade, picked_object id resetting

* add condition for the object to be in view before picking

* Disable base collisions for nav start

* fix place success - objects at rest conditions

* fix receptacle segmentation sensor

* revert third person resolution change

* add robot collisions for place

* Revert "Adds hab-lab changes for including static objects in navmesh settings (#1400)"

This reverts commit 145c7e1.

* fixes for last change on robot collisions

* rename and move files to ovmm sub folders

* review comments

* Revert "Revert "Adds hab-lab changes for including static objects in navmesh settings (#1400)""

This reverts commit 5b48cc2.

* Revert "Adds hab-lab changes for including static objects in navmesh settings (#1400)"

This reverts commit 145c7e1.

* remove seed

* update configs: remove place kinematic, rename sensors

* update sensor resolutions in benchmark/ovmm

* remove change for including radius/height in navmesh folder name

* Update habitat-baselines/habitat_baselines/config/ovmm/rl_skill.yaml

Co-authored-by: Arun Ramachandran <arun.jhurricane@gmail.com>

---------

Co-authored-by: Karmesh Yadav <karmeshyadav@fb.com>
Co-authored-by: Arun Ramachandran <arun.jhurricane@gmail.com>

---------

Co-authored-by: Arun Ramachandran <arun.jhurricane@gmail.com>
Co-authored-by: Jimmy Yang <jimmytyyang@fb.com>
Co-authored-by: Karmesh Yadav <karmeshyadav@fb.com>
Co-authored-by: Karmesh Yadav <ykarmesh@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ykarmesh ykarmesh ykarmesh approved these changes

@Skylion007 Skylion007 Skylion007 approved these changes

@yvsriram yvsriram Awaiting requested review from yvsriram

Assignees
No one assigned
Labels
CLA Signed Do not delete this pull request or issue due to inactivity.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@JHurricane96 @Skylion007 @ykarmesh @facebook-github-bot @aclegg3