Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Demo for the navigation task #3

Open
TimeEscaper opened this issue Sep 20, 2024 · 3 comments
Open

Demo for the navigation task #3

TimeEscaper opened this issue Sep 20, 2024 · 3 comments

Comments

@TimeEscaper
Copy link

Dear authors,

Thank you very much for releasing the code of the great work! We are now trying to deploy your model for the mobile robot navigation task. We are using inference_pretrained.ipynb as a reference and trying to adapt it for the navigation demo, but facing some troubles with that. Can you please outline the steps that should be done for running your model in navigation mode?

Thank you in advance!

@HomerW
Copy link
Contributor

HomerW commented Sep 23, 2024

To adapt the inference example from the notebook for navigation:

  • Use the navigation head by passing head_name='nav' into sample_actions
  • Use the omnimimic_gnm statistics to unnormalize the action by passing these into sample_actions
  • The observation dictionary should contain a single key image_nav containing the image observation

However, note that the full trajectory inference example in the notebook doesn't actually handle the timestep pad mask correctly. To actually roll out the policy on a robot you can use the evaluation server here. This is how we evaluated on mobile navigation robots. This server has a function stack_and_pad that will correctly stack the history of observations and create a correct timestep padding mask. Hope this helps!

@jisa
Copy link

jisa commented Oct 17, 2024

Cool project!

Using head_name='nav' and unnormalization_statistics=model.dataset_statistics["omnimimic_gnm_dataset"]["action"] leads to an error:
ValueError: Incompatible shapes for broadcasting: shapes=[(6, 4, 2), (100, 2)]

Where 6 is my input batch size. 4 is the number of steps predicted by the model. 100 is number of steps in the statistics. 2 is the dimensionality of a single action.

I.e. my understanding is that this is because the model predicts 4 steps, while the statistics is for 100 steps.

  • Is there a simple way to either make the model predict 100 steps or to trim the statistics to 4 steps? For performance reasons, I would prefer the later. Looking at the statistics, I understand that worst case I can trim it myself.
  • What is the interpretation of the outputs of the nav head? Some kind of waypoints? Are they all relative to the robot, or somehow relative to the previous waypoint?

@jisa
Copy link

jisa commented Oct 17, 2024

After trimming the stats myself with unnormalization_statistics=dict((stat_name, stat_value[:4,...]) for (stat_name, stat_value) in model.dataset_statistics["omnimimic_gnm_dataset"]["action"].items()), combined with knowledge from the paper and about previous related projects, I am guessing I am looking at metric waypoints relative to the robot, covering a short distance. The robot collecting the nav data is small and not very fast.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants