Realsense D455 with Kimera/Hydra pipelen. IR pattern problem.

[MikeDmytrenko]

Dear developers,

I'm trying to run Hydra pipeline with D455. I`ve successfully managed to spin Kimera-VIO with emitter disabled. However, for a successful mesh generation, a reliable depth image is also required which is only possible it seems with IR enabled. If IR is enabled Kimera-VIO gets confused and doesn't work. How is it possible to have both?

I've seen implementations such as this, where they use the emitter_on_off option and then utilize half of the frames for depth and half for infra streams. However, it is made for an old realsense SDK which doesn't seem to support D455. At least when I tried it, it didn`t produce the desired effect. Are you aware of any alternatives or what could be the best course of action? Thanks!

[MartyG-RealSense]

Hi @MikeDmytrenko The semi-random dot pattern projected from the IR emitter of a 400 Series camera casts dots onto surfaces that the camera can then use as a texture source for depth information. When the emitter is disabled and the dot pattern therefore removed from the scene, the amount of detail on the depth image is reduced.

You can reduce the visibility of the dot pattern by reducing the Laser Power value. Reducing the Laser Power value to '1' from its default value of '150' will keep the emitter enabled but with no visible pattern, whilst '0' will disable the emitter.

Alternatively, the 400 Series cameras can make use of any ambient light in a scene to analyze surfaces for depth detail, so you may be able to achieve an improved depth image in the absence of the dot pattern by increasing the light illumination in the scene if it is possible to do so.

In regard to the GitHub project that you linked to, emitter_on_off is still a valid parameter in the RealSense ROS wrapper to alternate the emitter on and off per frame. Support for the D455 camera model was not introduced until SDK version 2.38.1 though.

The most recent SDK and ROS wrapper combination at the time of writing this is:

ROS2
2.51.1 SDK and wrapper 4.51.1

ROS1
2.50.0 SDK and wrapper 2.3.2

[MartyG-RealSense]

Hi @MikeDmytrenko Do you require further assistance with this case, please? Thanks!

[MikeDmytrenko]

Hi, yeah thanks for your reply. Maybe for someone else, it could be useful - I've managed to modify the source code so that I can have both good quality depth and IR-free infra streams. The streams are published at half the frequency of fps set in the launch file. For some reason, it doesn't work every time I launch it, not sure what it depends on. Maybe you might have an idea what could be going wrong half of the time?

Here is the piece of code which achieves the desired behaviour (not every time):

if(0 != info_publisher.getNumSubscribers() ||
       0 != image_publisher.first.getNumSubscribers())
    {
        auto& cam_info = camera_info.at(stream);
        if (cam_info.width != width)
        {
            updateStreamCalibData(f.get_profile().as<rs2::video_stream_profile>());
        }
        cam_info.header.stamp = t;
        cam_info.header.seq = seq[stream];
        info_publisher.publish(cam_info);

        if ((stream.first == rs2_stream::RS2_STREAM_COLOR && !(seq[stream] % 2)) || 
             (stream.first == rs2_stream::RS2_STREAM_DEPTH    && !(seq[stream] % 2) ) ||
             (stream.first == rs2_stream::RS2_STREAM_INFRARED &&   seq[stream] % 2  ) )
        {
        sensor_msgs::ImagePtr img;
        img = cv_bridge::CvImage(std_msgs::Header(), encoding.at(stream.first), image).toImageMsg();
        img->width = width;
        img->height = height;
        img->is_bigendian = false;
        img->step = width * bpp;
        img->header.frame_id = cam_info.header.frame_id;
        img->header.stamp = t;
        img->header.seq = seq[stream];

        image_publisher.first.publish(img);
        }
         image_publisher.second->update();
        ROS_DEBUG("%s stream published", rs2_stream_to_string(f.get_profile().stream_type()));
    }

[MartyG-RealSense]

IntelRealSense/librealsense#7221 may be a helpful ROS reference for your sometimes-halving FPS when using depth and color.

[MartyG-RealSense]

Hi @MikeDmytrenko Do you require further assistance with this case, please? Thanks!

[MikeDmytrenko]

Hi, I've managed to solve the unstable behavior in this case by reduxing the frame rate to 30fps, which is not ideal since now I'm getting 15 fps per depth and infra streams. With 60 fps enable I'd get HID set_power 1 failed, which I guess is a performance issue.

If you don't have anything to add to this, feel free to close the issue.

[MartyG-RealSense]

Are you making use of the D455 camera's IMU, please? At MIT-SPARK/Kimera-VIO-ROS#113 (comment) a RealSense user who was using a D435 with Kimera and was experiencing the HID set_power 1 failed error resolved their problems by setting the accel FPS to 63 and gyro to 200.

---

In regard to the ZJU-FAST-Lab project not working with D455, that project's instructions install the 2.30.0 version of the RealSense SDK from packages, but D455 support was not introduced in the SDK until 2.38.1. So you could amend the instructions to install 2.38.1 from packages, or install the latest version of the SDK (2.51.1 at the time of writing this).

If installing a specific past version then IntelRealSense/librealsense#5773 (comment) advises how you can use the command apt-cache showpkg librealsense2 to list all the versions available and then select the appropriate tags.

[MartyG-RealSense]

Hi @MikeDmytrenko Do you require further assistance with this case, please? Thanks!