-
Notifications
You must be signed in to change notification settings - Fork 505
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[v0.2.3] [Stretch] play stretch demo #1014
Conversation
The functionality and video both looks great! Can we get Noah to add some more texture to that base? It is looking very flat. |
Thanks for the comment! I will sync with Noah and add more texture! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Generally looks good to me.
interactive_play.py
changes seem a bit adhoc, but the entire script is like that, so fine for now. In the future we should try to clean this up for generic robot support.
I am concerned about this mismatch of a kinematic action for Fetch but non-kinematic simulation. I synced with @jimmytyyang about this offline, but we should fix the simulation to also be kinematic, or else there will be weird behavior (like the arm will be able to penetrate surfaces easily). But maybe we can get this PR in now as is and work on that later? |
Hi @ASzot Thanks for the information! There is a discussion about if we should use kinematic or dynamics simulation. Fixing the simulation to be kinematic requires other substantial code changes such as how to detect collision and so on. We could get this PR in now to get the ball rolling as there is a lot of interest for people who want to try out Stretch, but I am open to any discussion! Thank you. |
Great point, @ASzot. Agreed, let's get the demo in now and then discuss the broader paradigm moving forward. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM 👍
* [v0.2.3] Add robot document (#1018) * [v0.2.3] [Stretch] play stretch demo (#1014) * [Fix] Documentation gym example path (#1016) Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * [0.2.3]Hydra migration 3 v2 (#1000) * updated md files * updated habitat2_gym_tutorial * updated Habitat2_Quickstart * updated Habitat_Lab * updated docs * partially removed YACS (note there are some inpedendent parts of code that still rely on YACS) * made all DictConfig imports to be from omegaconf using if TYPE_CHECKING * removed link to the outdated notebook * fixed tests * [Hydra Migration] Removing yacs and migratting the dataset generation code of rearrange (#1001) * removing yacs and migratting the dataset generation code of rearrange * addressing comments * typechecking corrections Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * renaming SensorConfig to LabSensorConfig * fixing svg link * fixing svg; * using relative path is better for svg * addressed comments * [0.2.3][Hydra Migration] Made agents to be of type Dict[str, AgentConfig] and added migration readme (#1007) * made habitat.simulator.agents be a Dict[str, AgentConfig] * addressed comments * refactored sim agents configuration * moved rearrange agents declarations to agents folder; finised agents configuration updates; updated docs * updated get_config import * Cfg sys migration readme (#2) * added habitat-lab config readme * addressed most of the comments * added main_agent alias * renamed habitat config reagme and added baselines config readme * Update habitat-lab/habitat/config/default.py Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> * added desctiption for the agents_order field * added a dot :) Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> * fixing conflicts Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * Fix links in MANIFEST.in (#991) * Fix broken links in MANIFEST.in * fixig video_render_view * fixing issue with gym maker * editing test that changed with recent config modifications * [v0.2.3] Fixing mypy (#1023) * Unpin habitat-lab requirements * Reorder dependencies * Fix mypy config * remove unnecessary mypy args * Update mypy version * Fix python lint * Fix pre-commit mypy further * fix mypy ignored errors * fixing mypy * fixing mypy * more accurate type:ignores * adding docs/ to exclude mypy * trying to fix python_lint * exlude arguments * exlude arguments * Container needs to be a string Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com> Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * removing the gym envs keys that were removed Co-authored-by: Jimmy Yang <55121504+jimmytyyang@users.noreply.github.com> Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> Co-authored-by: Ruslan <partsey2412@gmail.com> Co-authored-by: Mikaël Dallaire Côté <110583667+0mdc@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com>
* [bugfix] Reconfigure the sim even if the sim is not None * addressing comments, better way to decide wether or not to reconfigure sim_obj * same fix for the robot base * Added back caching to pick task * Fixed up nav to obj task * Fixed up open and close skills * Added back inits for nav to obj * Batched HRL policy * Updated task conditions * Fixed key mismatch in TP-SRL * Fixed issue with nav to obj skill * Fixed early stop issue * Fixed bad filter condition * Updated training configs * Refactored and cleaned up config files * Fixed up reward. Renamed configs * Added wandb resume * Refactored eval with debug sensor code. * Added get policy info * Hopefully fixed resuming, added safe drop * Added improved Fetch model * Removed refactored files * Added back default * Removed habitat folder * Fixed pick task * Fixed navigation skill * Removed redundancy in skill training configs * Addressed some comments * Removed nav pick * Update .gitignore Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> * PR comments * [v0.2.3] Hab2 fixup : merging main and resolving conflicts (#1022) * [v0.2.3] Add robot document (#1018) * [v0.2.3] [Stretch] play stretch demo (#1014) * [Fix] Documentation gym example path (#1016) Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * [0.2.3]Hydra migration 3 v2 (#1000) * updated md files * updated habitat2_gym_tutorial * updated Habitat2_Quickstart * updated Habitat_Lab * updated docs * partially removed YACS (note there are some inpedendent parts of code that still rely on YACS) * made all DictConfig imports to be from omegaconf using if TYPE_CHECKING * removed link to the outdated notebook * fixed tests * [Hydra Migration] Removing yacs and migratting the dataset generation code of rearrange (#1001) * removing yacs and migratting the dataset generation code of rearrange * addressing comments * typechecking corrections Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * renaming SensorConfig to LabSensorConfig * fixing svg link * fixing svg; * using relative path is better for svg * addressed comments * [0.2.3][Hydra Migration] Made agents to be of type Dict[str, AgentConfig] and added migration readme (#1007) * made habitat.simulator.agents be a Dict[str, AgentConfig] * addressed comments * refactored sim agents configuration * moved rearrange agents declarations to agents folder; finised agents configuration updates; updated docs * updated get_config import * Cfg sys migration readme (#2) * added habitat-lab config readme * addressed most of the comments * added main_agent alias * renamed habitat config reagme and added baselines config readme * Update habitat-lab/habitat/config/default.py Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> * added desctiption for the agents_order field * added a dot :) Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> * fixing conflicts Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * Fix links in MANIFEST.in (#991) * Fix broken links in MANIFEST.in * fixig video_render_view * fixing issue with gym maker * editing test that changed with recent config modifications * [v0.2.3] Fixing mypy (#1023) * Unpin habitat-lab requirements * Reorder dependencies * Fix mypy config * remove unnecessary mypy args * Update mypy version * Fix python lint * Fix pre-commit mypy further * fix mypy ignored errors * fixing mypy * fixing mypy * more accurate type:ignores * adding docs/ to exclude mypy * trying to fix python_lint * exlude arguments * exlude arguments * Container needs to be a string Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com> Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * removing the gym envs keys that were removed Co-authored-by: Jimmy Yang <55121504+jimmytyyang@users.noreply.github.com> Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> Co-authored-by: Ruslan <partsey2412@gmail.com> Co-authored-by: Mikaël Dallaire Côté <110583667+0mdc@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com> * Merged * Removed bad file and fixed IK * Env fixes * Fixed precommit * Fixed eval and fixed wandb logging * Fix for the resume in VER and the force threshold for open tasks * Update habitat-baselines/habitat_baselines/rl/ddppo/policy/resnet_policy.py Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com> * tp-srl fixes * tp-srl fixes * Fixed more tests * Cleaned up nav action * Added tests for all tasks in baseline trainer * Corrected test overrides * Improved agents loop in resnet_policy Improved agents loop in resnet_policy Co-authored-by: Ruslan <partsey2412@gmail.com> * Fixed policy docs and improvements. Added docs to spawn robot method. * [0.2.3] 959 PR review suggested changes (#1029) * removed habitat-baselines/habitat_baselines/config path prefix * improved adding render view uuids to the gym obs_keys * made config keys lowercase * Updated docs * Update habitat-lab/habitat/tasks/rearrange/rearrange_task.py Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> * Update habitat-baselines/habitat_baselines/rl/ddppo/policy/resnet_policy.py Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com> * Fixed formatting * tp-srl config fix * fixing documentation ? Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> Co-authored-by: ASzot <me@andrewszot.com> Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> Co-authored-by: Jimmy Yang <55121504+jimmytyyang@users.noreply.github.com> Co-authored-by: Ruslan <partsey2412@gmail.com> Co-authored-by: Mikaël Dallaire Côté <110583667+0mdc@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com> Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com> Co-authored-by: Oleksandr <maksymets.o@gmail.com> Co-authored-by: Oleksandr Maksymets <maksymets@gmail.com> Co-authored-by: SergioArnaud <sercharnaud@gmail.com>
…okresearch#959) * [bugfix] Reconfigure the sim even if the sim is not None * addressing comments, better way to decide wether or not to reconfigure sim_obj * same fix for the robot base * Added back caching to pick task * Fixed up nav to obj task * Fixed up open and close skills * Added back inits for nav to obj * Batched HRL policy * Updated task conditions * Fixed key mismatch in TP-SRL * Fixed issue with nav to obj skill * Fixed early stop issue * Fixed bad filter condition * Updated training configs * Refactored and cleaned up config files * Fixed up reward. Renamed configs * Added wandb resume * Refactored eval with debug sensor code. * Added get policy info * Hopefully fixed resuming, added safe drop * Added improved Fetch model * Removed refactored files * Added back default * Removed habitat folder * Fixed pick task * Fixed navigation skill * Removed redundancy in skill training configs * Addressed some comments * Removed nav pick * Update .gitignore Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> * PR comments * [v0.2.3] Hab2 fixup : merging main and resolving conflicts (facebookresearch#1022) * [v0.2.3] Add robot document (facebookresearch#1018) * [v0.2.3] [Stretch] play stretch demo (facebookresearch#1014) * [Fix] Documentation gym example path (facebookresearch#1016) Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * [0.2.3]Hydra migration 3 v2 (facebookresearch#1000) * updated md files * updated habitat2_gym_tutorial * updated Habitat2_Quickstart * updated Habitat_Lab * updated docs * partially removed YACS (note there are some inpedendent parts of code that still rely on YACS) * made all DictConfig imports to be from omegaconf using if TYPE_CHECKING * removed link to the outdated notebook * fixed tests * [Hydra Migration] Removing yacs and migratting the dataset generation code of rearrange (facebookresearch#1001) * removing yacs and migratting the dataset generation code of rearrange * addressing comments * typechecking corrections Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * renaming SensorConfig to LabSensorConfig * fixing svg link * fixing svg; * using relative path is better for svg * addressed comments * [0.2.3][Hydra Migration] Made agents to be of type Dict[str, AgentConfig] and added migration readme (facebookresearch#1007) * made habitat.simulator.agents be a Dict[str, AgentConfig] * addressed comments * refactored sim agents configuration * moved rearrange agents declarations to agents folder; finised agents configuration updates; updated docs * updated get_config import * Cfg sys migration readme (facebookresearch#2) * added habitat-lab config readme * addressed most of the comments * added main_agent alias * renamed habitat config reagme and added baselines config readme * Update habitat-lab/habitat/config/default.py Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> * added desctiption for the agents_order field * added a dot :) Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> * fixing conflicts Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * Fix links in MANIFEST.in (facebookresearch#991) * Fix broken links in MANIFEST.in * fixig video_render_view * fixing issue with gym maker * editing test that changed with recent config modifications * [v0.2.3] Fixing mypy (facebookresearch#1023) * Unpin habitat-lab requirements * Reorder dependencies * Fix mypy config * remove unnecessary mypy args * Update mypy version * Fix python lint * Fix pre-commit mypy further * fix mypy ignored errors * fixing mypy * fixing mypy * more accurate type:ignores * adding docs/ to exclude mypy * trying to fix python_lint * exlude arguments * exlude arguments * Container needs to be a string Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com> Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * removing the gym envs keys that were removed Co-authored-by: Jimmy Yang <55121504+jimmytyyang@users.noreply.github.com> Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> Co-authored-by: Ruslan <partsey2412@gmail.com> Co-authored-by: Mikaël Dallaire Côté <110583667+0mdc@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com> * Merged * Removed bad file and fixed IK * Env fixes * Fixed precommit * Fixed eval and fixed wandb logging * Fix for the resume in VER and the force threshold for open tasks * Update habitat-baselines/habitat_baselines/rl/ddppo/policy/resnet_policy.py Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com> * tp-srl fixes * tp-srl fixes * Fixed more tests * Cleaned up nav action * Added tests for all tasks in baseline trainer * Corrected test overrides * Improved agents loop in resnet_policy Improved agents loop in resnet_policy Co-authored-by: Ruslan <partsey2412@gmail.com> * Fixed policy docs and improvements. Added docs to spawn robot method. * [0.2.3] 959 PR review suggested changes (facebookresearch#1029) * removed habitat-baselines/habitat_baselines/config path prefix * improved adding render view uuids to the gym obs_keys * made config keys lowercase * Updated docs * Update habitat-lab/habitat/tasks/rearrange/rearrange_task.py Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> * Update habitat-baselines/habitat_baselines/rl/ddppo/policy/resnet_policy.py Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com> * Fixed formatting * tp-srl config fix * fixing documentation ? Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> Co-authored-by: ASzot <me@andrewszot.com> Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> Co-authored-by: Jimmy Yang <55121504+jimmytyyang@users.noreply.github.com> Co-authored-by: Ruslan <partsey2412@gmail.com> Co-authored-by: Mikaël Dallaire Côté <110583667+0mdc@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com> Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com> Co-authored-by: Oleksandr <maksymets.o@gmail.com> Co-authored-by: Oleksandr Maksymets <maksymets@gmail.com> Co-authored-by: SergioArnaud <sercharnaud@gmail.com>
…okresearch#959) * [bugfix] Reconfigure the sim even if the sim is not None * addressing comments, better way to decide wether or not to reconfigure sim_obj * same fix for the robot base * Added back caching to pick task * Fixed up nav to obj task * Fixed up open and close skills * Added back inits for nav to obj * Batched HRL policy * Updated task conditions * Fixed key mismatch in TP-SRL * Fixed issue with nav to obj skill * Fixed early stop issue * Fixed bad filter condition * Updated training configs * Refactored and cleaned up config files * Fixed up reward. Renamed configs * Added wandb resume * Refactored eval with debug sensor code. * Added get policy info * Hopefully fixed resuming, added safe drop * Added improved Fetch model * Removed refactored files * Added back default * Removed habitat folder * Fixed pick task * Fixed navigation skill * Removed redundancy in skill training configs * Addressed some comments * Removed nav pick * Update .gitignore Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> * PR comments * [v0.2.3] Hab2 fixup : merging main and resolving conflicts (facebookresearch#1022) * [v0.2.3] Add robot document (facebookresearch#1018) * [v0.2.3] [Stretch] play stretch demo (facebookresearch#1014) * [Fix] Documentation gym example path (facebookresearch#1016) Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * [0.2.3]Hydra migration 3 v2 (facebookresearch#1000) * updated md files * updated habitat2_gym_tutorial * updated Habitat2_Quickstart * updated Habitat_Lab * updated docs * partially removed YACS (note there are some inpedendent parts of code that still rely on YACS) * made all DictConfig imports to be from omegaconf using if TYPE_CHECKING * removed link to the outdated notebook * fixed tests * [Hydra Migration] Removing yacs and migratting the dataset generation code of rearrange (facebookresearch#1001) * removing yacs and migratting the dataset generation code of rearrange * addressing comments * typechecking corrections Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * renaming SensorConfig to LabSensorConfig * fixing svg link * fixing svg; * using relative path is better for svg * addressed comments * [0.2.3][Hydra Migration] Made agents to be of type Dict[str, AgentConfig] and added migration readme (facebookresearch#1007) * made habitat.simulator.agents be a Dict[str, AgentConfig] * addressed comments * refactored sim agents configuration * moved rearrange agents declarations to agents folder; finised agents configuration updates; updated docs * updated get_config import * Cfg sys migration readme (#2) * added habitat-lab config readme * addressed most of the comments * added main_agent alias * renamed habitat config reagme and added baselines config readme * Update habitat-lab/habitat/config/default.py Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> * added desctiption for the agents_order field * added a dot :) Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> * fixing conflicts Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * Fix links in MANIFEST.in (facebookresearch#991) * Fix broken links in MANIFEST.in * fixig video_render_view * fixing issue with gym maker * editing test that changed with recent config modifications * [v0.2.3] Fixing mypy (facebookresearch#1023) * Unpin habitat-lab requirements * Reorder dependencies * Fix mypy config * remove unnecessary mypy args * Update mypy version * Fix python lint * Fix pre-commit mypy further * fix mypy ignored errors * fixing mypy * fixing mypy * more accurate type:ignores * adding docs/ to exclude mypy * trying to fix python_lint * exlude arguments * exlude arguments * Container needs to be a string Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com> Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> * removing the gym envs keys that were removed Co-authored-by: Jimmy Yang <55121504+jimmytyyang@users.noreply.github.com> Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> Co-authored-by: Ruslan <partsey2412@gmail.com> Co-authored-by: Mikaël Dallaire Côté <110583667+0mdc@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com> * Merged * Removed bad file and fixed IK * Env fixes * Fixed precommit * Fixed eval and fixed wandb logging * Fix for the resume in VER and the force threshold for open tasks * Update habitat-baselines/habitat_baselines/rl/ddppo/policy/resnet_policy.py Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com> * tp-srl fixes * tp-srl fixes * Fixed more tests * Cleaned up nav action * Added tests for all tasks in baseline trainer * Corrected test overrides * Improved agents loop in resnet_policy Improved agents loop in resnet_policy Co-authored-by: Ruslan <partsey2412@gmail.com> * Fixed policy docs and improvements. Added docs to spawn robot method. * [0.2.3] 959 PR review suggested changes (facebookresearch#1029) * removed habitat-baselines/habitat_baselines/config path prefix * improved adding render view uuids to the gym obs_keys * made config keys lowercase * Updated docs * Update habitat-lab/habitat/tasks/rearrange/rearrange_task.py Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> * Update habitat-baselines/habitat_baselines/rl/ddppo/policy/resnet_policy.py Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com> * Fixed formatting * tp-srl config fix * fixing documentation ? Co-authored-by: vincentpierre <vincentpierre@users.noreply.github.com> Co-authored-by: ASzot <me@andrewszot.com> Co-authored-by: Vincent-Pierre BERGES <28320361+vincentpierre@users.noreply.github.com> Co-authored-by: Jimmy Yang <55121504+jimmytyyang@users.noreply.github.com> Co-authored-by: Ruslan <partsey2412@gmail.com> Co-authored-by: Mikaël Dallaire Côté <110583667+0mdc@users.noreply.github.com> Co-authored-by: Aaron Gokaslan <aaronGokaslan@gmail.com> Co-authored-by: Aaron Gokaslan <skylion.aaron@gmail.com> Co-authored-by: Oleksandr <maksymets.o@gmail.com> Co-authored-by: Oleksandr Maksymets <maksymets@gmail.com> Co-authored-by: SergioArnaud <sercharnaud@gmail.com>
Motivation and Context
Provide a demo for running Stretch. Download the stretch asset using data loader in Habitat sim.
Then run
python examples/interactive_play.py --never-end --cfg [path to stretch]/play_stretch.yaml
,then you can use a keyboard to control Stretch.
play_stretch_pr_demo.mp4
How Has This Been Tested
Locally
Types of changes
Checklist