Instance (or Semantic) Segmentation Camera Sensor

[asiron]

Hi I am interested in creating a Gazebo sensor that would output ground truth data for Instance segmentation task. Below, we have an example of raw image, semantic segmentation and instance segmentation respectively.

This feature request was already raised here #1192 (comment) before gazebo migrated to GitHub, but unfortunately @daveraja's links are broken and the progress seems to stop there. I think that feature would greatly benefit the ML/CV research community at large, that's why I started a separate issue.

As far as I can see, we would need to add a new sensor (InstanceSegmentationCamera) similar to gazebo::rendering::DepthCamera.(cc|hh), then building a gazebo sensor plugin and ROS plugin from it would be straightforward. The main difficulty is in rendering the segmentation image. We want to avoid rendering any shadows, and essentially the color for each model should be it's ID mapped onto the RGB color-space. The fragment shader would be a simple pass shader, and the vertex shader would have to receive the model's ID and use it to set the color. However, I am not sure how to pass the ID of the object to the vertex shader in the OGRE engine. Somebody already asked that question:

Original comment by Ian Chen (Bitbucket: Ian Chen, GitHub: iche033).

but I'm struggling with how to pass a variable down to that level.

if you're looking for an example on how to pass variable to shaders, here's one. You'll need to declare that variable in the materials script and in the shaders

but it seems like that is an inefficient way of passing data to shader (judged by the comment here), also that seems to pass only 1 variable, but we would need to pass an entire array of IDs for each vertex, or I am wrong ?

https://github.com/osrf/gazebo/blob/ab82c04da4afe36cb7e6b9842a1d7b4cdfa35570/gazebo/rendering/Distortion.cc#L79-L90

I am willing to put time to it and make it see the daylight, but I would appreciate some guidance.

[iche033]

gazebo's selection buffer mechanics may be useful to look into. We use it for pixel accurate mouse picking. The idea is that each object is assigned a different unique color, and when a mouse clicks on a pixel, we can get the color of the pixel and find the associated id of the object.

There are some explanations on the ogre wiki page. See if that's what you are looking for

[maxim0815]

This looks like a handy feature. Did you start working on it @asiron?
Since the sensors differ, I'm not sure if this belongs here. But maybe the idea with semantic information can be extended to ray-based sensors?