Add fisheye camera model

[matsuren]

Motivation and Context

It will solve facebookresearch/habitat-sim#794.
Currently, equirectangular images are available aside from perspective images thanks to @Skylion007 . This pull request adds fisheye camera model. The fisheye model is based on the Double Sphere Camera Model. I chose this fisheye camera model because it can handle more than 180 degrees of FoV and the unprojection is fast according to their paper.
I'm not sure if you are interested in generating fisheye images, so currently, it's just draft pull request. Please tell me what you think.

Basically, the structure is the almost same as Cube2Equirec except that different projection on a sphere based on camera parameters, taking care of fov, and making sure each point is only assigned to single face.
So, when you will marge this camera model, maybe it's better to create a parent class like Cube2Omni, and Cube2Equirec and Cube2Fisheye are inhirited from the parent class with different implementation of different projection on a sphere. But, it will require a lot of changes, so I will leave it to you.

How Has This Been Tested

You can generate fisheye images by the following code.

import habitat
from habitat_baselines.utils.common import batch_obs
from habitat_baselines.common.obs_transformers import Cube2Fisheye
import math
from copy import deepcopy
import matplotlib.pyplot as plt
import torch

config = habitat.get_config("configs/tasks/pointnav.yaml")
config.defrost()
CAMERA_NUM = 6
orient = [
    [0, math.pi, 0],  # Back
    [-math.pi / 2, 0, 0],  # Down
    [0, 0, 0],  # Front
    [0, math.pi / 2, 0],  # Right
    [0, 3 / 2 * math.pi, 0],  # Left
    [math.pi / 2, 0, 0],  # Up
]
sensor_uuids = []
# Setup six cameras, Back, Down, Front, Left, Right, Up.
if "RGB_SENSOR" in config.SIMULATOR.AGENT_0.SENSORS:
    config.SIMULATOR.RGB_SENSOR.ORIENTATION = orient[0]
    for camera_id in range(CAMERA_NUM):
        camera_template = f"RGB_{camera_id}"
        camera_config = deepcopy(config.SIMULATOR.RGB_SENSOR)
        camera_config.ORIENTATION = orient[camera_id]

        camera_config.UUID = camera_template.lower()
        sensor_uuids.append(camera_config.UUID)
        setattr(config.SIMULATOR, camera_template, camera_config)
        config.SIMULATOR.AGENT_0.SENSORS.append(camera_template)
config.freeze()

# Load embodied AI task (PointNav) and a pre-specified virtual robot
env = habitat.Env(config=config)
observations = env.reset()
observations = env.reset()
batch = batch_obs([observations])
# Prepare cubemap
sensor_obs = [batch[sensor] for sensor in sensor_uuids]
sensor_dtype = sensor_obs[0].dtype
# Stacking along axis makes the flattening go in the right order.
imgs = torch.stack(sensor_obs, axis=1)
imgs = torch.flatten(imgs, end_dim=1)
imgs = imgs.permute((0, 3, 1, 2))  # NHWC => NCHW
imgs = imgs.float()

# Fisheye image parameters
fish_w = 512
fish_h = 512
fish_fov = 220
xi = 0.3
alpha = 0.3
fx = 0.15 * fish_w
fy = fx
cx = fish_w / 2
cy = fish_h / 2

model = Cube2Fisheye(fish_h, fish_w, fish_fov, cx, cy, fx, fy, xi, alpha)
out = model(imgs)
out = out.to(dtype=sensor_dtype)
out = out.permute((0, 2, 3, 1))  # NCHW => NHWC
out = out.numpy().squeeze()
plt.imshow(out)
plt.show()

The generated fisheye image:

Types of changes

• New feature (non-breaking change which adds functionality)

Checklist

• My code follows the code style of this project.
• My change requires a change to the documentation.
• I have updated the documentation accordingly.
• I have read the CONTRIBUTING document.
• I have completed my CLA (see CONTRIBUTING)
• I have added tests to cover my changes.
• All new and existing tests passed.

[Skylion007]

This is great and very exciting. All that is left is to hook it up to an ObservationTransformer like we have for Equirect generation. Also need to add tests briefly like we have for stitching equirect.

Are there any optimizations that we could use to lessen the number of sensors required btw? Like if the FOV is < 180 is it possible to construct the input from only 5 sensors? That complexity may not be worth it though.

Excellent work!

[matsuren]

Thank you for your comment! I added ObservationTransformer and test function for generating fisheye images.
I'm not familiar with ObservationTransformer and trainer in the test function, so basically I just copied your code for CubeMap2Equirec. Let me know if anything is wrong.

Are there any optimizations that we could use to lessen the number of sensors required btw? Like if the FOV is < 180 is it possible to construct the input from only 5 sensors? That complexity may not be worth it though.

I think it is possible, but I'd rather not do it because it will complicate the process and I'm not sure how much is the improvement.

Thank you!

[matsuren]

I see. Good to know I can combine multiple @pytest.mark.parametrize.

[raoqiang]

Example can not run:
config_not_found_error(repo=repo, tree=root)
File "/home/raoqiang/.local/lib/python3.9/site-packages/hydra/_internal/defaults_list.py", line 799, in config_not_found_error
raise MissingConfigException(
hydra.errors.MissingConfigException: In 'pointnav.yaml': Could not load 'lab_sensors/pointgoal_with_gps_compass_sensor'.

Config search path:
provider=hydra, path=pkg://hydra.conf
provider=main, path=file:///home/raoqiang/sda/project/hawk/habitat-lab/habitat-lab/habitat/config/habitat/task
provider=habitat, path=pkg://habitat.config
provider=schema, path=structured://
raoqiang@raoqiang-Precision-3650-Tower:~/sda/proj

[jdiazram]

Hi everyone. Nice job. I am new on Habitat, and I was wondering, based on the code above, if is possible to create images like this:

Thx for your help. Bye