Instance ImageGoal Navigation Task #936
Conversation
facebook-github-bot
added
the
CLA Signed
| class InstanceImageGoalHFOVSensor(Sensor): | ||
| """A sensor that returns the horizontal field of view (HFOV) in degrees | ||
| of the current episode's instance image goal. | ||
| """ |
There was a problem hiding this comment.
Why is the HFoV so special among all the intrinsics?
There was a problem hiding this comment.
All the others are zero so it implies the complete K. We provide HFoV to allow task methods to apply various transforms.
There was a problem hiding this comment.
I see. Does it make sense for the task dataset to involve some randomization over entries in K?
There was a problem hiding this comment.
My thought is no. FOV tends to be an intentional choice whereas other instrinsics like distortion effects are not. Dealing with these effects does not seem like a beneficial complication at this time. I chatted with Devendra and he is on board with this as well.
There was a problem hiding this comment.
Yup, that's fair.
One last question -- are we letting people pick both H and V FoV or are we committing them to an aspect ratio?
There was a problem hiding this comment.
Our dataset commits to a 1:1 aspect ratio and 512x512 resolution, but the code and dataset format supports an arbitrary aspect ratio, resolution, and FOV on a per-image basis.
There was a problem hiding this comment.
Ok, fair enough. And you'll release the code for dataset generation as well?
There was a problem hiding this comment.
Yes! Not for this core task PR, but as a follow-up.
Instance Image Goal Navigation task. Given an image of an object, an agent is tasked with navigating to that specific object.
Instance Image Goal Navigation task. Given an image of an object, an agent is tasked with navigating to that specific object.
Motivation and Context
This PR introduces the Instance Image Goal Navigation task. Given an image of an object, an agent is tasked with navigating to that specific object. The goal criteria is adapted from ObjectNav to apply to a singular object. The goal image is specified by extrinsic and intrinsic parameters supplied by the dataset, i.e., the goal image can have a different camera height, viewing angle, and HFOV than the agent's observation space.
instance_imagegoal.mp4
A link to the dataset has been added to the datasets page.
How Has This Been Tested
Task tests added in
test/test_instance_image_nav_task.pymodeled after other nav task tests.This has been forward-tested with
habitat_baselinescode that trains a DDPPO agent. I plan to introduce a follow-up PR to support this.Types of changes
Checklist