Overlay image with a segmentation mask

[charlielito]

Hi! It would be awesome to be able to overlay some segmentation maps (stored in a topic of image type) into another normal image. Something like this:

Currently, ASFAIK the only way to do something similar is using ImageMarkers, but it is not flexible enough and doesn't give us per-pixel visualization.

Do you know if you are planning to do this or how could we implement this?

[defunctzombie]

What is the format of your segmentation mask?

[charlielito]

It's an image where each pixel corresponds to the class id. For example, 0-> background, 1->sidewalk, 2->person and so on

[charlielito]

We are storing that info in our rosbags as an Image topic. We compressed the image using PNG so it is lossless

[defunctzombie]

Thanks for the details. This feature is on our roadmap. I had thought we had the ability to overlay a second image in the image panel but I don't think that's possible yet.

Besides overlaying the image is there other behaviors you'd like from such a feature?

[charlielito]

I think there are some open questions one might have:

1. How to map from class_idx to class names?
2. What kind of transparency when overlaying the image?
3. What kind of images should be supported as masks?
4. Which color map should be used?

For 1. I think one could use https://github.com/ros-perception/vision_msgs/blob/ros2/vision_msgs/msg/LabelInfo.msg message in a topic as we do for image topics with camera_info. It could be also a parameter in the GUI?

For 2. I think the easiest is also a parameter directly in Fogxlove

For 3. it makes sense to support sensor_msgs/msg/Image and sensor_msgs/msg/CompressedImage but with PNG encoding since JPG would destroy the mask.

For 4. There are multiple options. Some standard colormaps could be used. It could be also configured as a param in the GUI as well. Or use some param file like here. It would make sense to have a legend of color meanings if the user provided 1. It would be nice also to be able to disable/enable certain classes like weights and biases segmentation plugin

[jmm-slamcore]

Something like this would be great! Agree with all the points above. A lot of times semantic masks can't be visualised without a colour map being applied because the pixel values for classes in the image start from 0 and the whole image looks black/grey with just a few classes.

[frapit]

Until this is supported natively by the Image panel, you can easily accomplish the blending with a user script. Quick and dirty, borrows a lot from the default template:

Userscript

import { Input, Message, Time } from "./types";
import { RawImage } from "@foxglove/schemas";

type Image = Message<"sensor_msgs/Image">;
type Output = Message<"foxglove.RawImage">;

export const inputs = [
  "/image_color",
  "/segmentation",
];

export const output = "/overlay";

let lastReceiveTime: Time = { sec: 0, nsec: 0 };
let lastReceivedImg: Image;

let inSync: Boolean = false;

const image_overlay: {
  image?: Image;
  segmentation?: Image;
} = {};

type GlobalVariables = { blendAlpha: number };

function blendImages(
  image1Data: Uint8Array,
  image2Data: Uint8Array,
  blendPercentage: number,
): Uint8Array {
  if (image1Data.length !== image2Data.length) {
    throw new Error("Image data lengths do not match");
  }

  blendPercentage = Math.max(0, Math.min(100, blendPercentage));

  const blendedImageData = new Uint8Array(image1Data.length);

  // Blend pixel values
  for (let i = 0; i < image1Data.length; i += 3) {
    const b1 = image1Data[i];
    const g1 = image1Data[i + 1];
    const r1 = image1Data[i + 2];

    const b2 = image2Data[i];
    const g2 = image2Data[i + 1];
    const r2 = image2Data[i + 2];

    const blendedB = Math.round(
      (b1 * (100 - blendPercentage) + b2 * blendPercentage) / 100,
    );
    const blendedG = Math.round(
      (g1 * (100 - blendPercentage) + g2 * blendPercentage) / 100,
    );
    const blendedR = Math.round(
      (r1 * (100 - blendPercentage) + r2 * blendPercentage) / 100,
    );

    blendedImageData[i] = blendedB;
    blendedImageData[i + 1] = blendedG;
    blendedImageData[i + 2] = blendedR;
  }

  return blendedImageData;
}

// This function is called with messages from your input topics.
// The first argument is an event with the topic, receive time, and message.
// Use the `Input<...>` helper to get the correct event type for your input topic messages.
export default function script(
  event:
    | Input<"/image_color">
    | Input<"/segmentation">,
  globalVars: GlobalVariables,
): Output | undefined {
  if (event.topic === inputs[1]) {
    image_overlay.segmentation = event.message;
    inSync = false;
  } else if (event.topic === inputs[0]) {
    lastReceiveTime.sec = event.receiveTime.sec;
    lastReceiveTime.nsec = event.receiveTime.nsec;
    lastReceivedImg = event.message;
  }
  if (!inSync) {
    // Compare Timestamps 100 ms range
    if (
      Math.abs(
        event.receiveTime.sec * 1e9 +
          event.receiveTime.nsec -
          (lastReceiveTime.sec * 1e9 + lastReceiveTime.nsec),
      ) < 1e8
    ) {
      image_overlay.image = lastReceivedImg;
      inSync = true;
    }
  }

  if (
    image_overlay.image != undefined &&
    image_overlay.segmentation != undefined
  ) {
    let out_image: Output = {
      timestamp: image_overlay.segmentation.header.stamp,
      frame_id: image_overlay.segmentation.header.frame_id,
      encoding: image_overlay.image.encoding,
      height: image_overlay.image.height,
      width: image_overlay.image.width,
      step: image_overlay.image.step,
      data: blendImages(
        image_overlay.image.data,
        image_overlay.segmentation.data,
        globalVars.blendAlpha,
      ),
    };
    return out_image;
  }

  return undefined;
}

For equal timestamps you can skip the timestamp logic.

[charlielito]

Wow thanks! I think as @jmm-slamcore noted here https://github.com/orgs/foxglove/discussions/897#discussioncomment-8162730 the image would look gray/dark because there is no mapping of class_ids to color_map. Also typically the segmentation topic is a 1-channel image which wouldn't work with this code, but it's a great starting point!

Just a question since I am not very familiar with typescript or foxglove codebase. How would you run this script or how would you integrate this into foxglove?

[frapit]

Of course this was only the solution for my usecase of two color images, one with a already colored mask and one camera image. For one channel images you would need to integrate some colormap and more difficult operations should probably use some image manipulation library instead of directly altering the UInt8 Array.

This is very easy to use, inside Foxglove just open a new Panel of type UserScript and paste the code in there. This creates a new internal topic to which you can subscribe. You can alter global variables with something like a variable slider. Thats it.

foxglove_userscript-2024-02-23_10.20.46.mp4

[charlielito]

this is awesome @frapit !! I'll try to adapt the script and put an example here