3D Depth image distorted, not able to get a proper visual

[ghokulji]

librealsense	2.39.0 RELEASE
OS	Linux
Name	Intel RealSense D435I
Serial Number	923322072183
Firmware Version	05.12.08.200
Advanced Mode	YES
Camera Locked	YES
Usb Type Descriptor	3.2
Product Line	D400
Asic Serial Number	926623051259
Firmware Update Id	926623051259

Please provide a description of the problem
Hello,

I have setup the d435i camera and tested it out. The 2d image is fine, but there is an issue with the 3d image it is not projected properly.
This is the 2D image:

This is the 3D image. As you can see the 3d image is not projected properly. Any suggestions on how to fix this issue.

Also,when I run the realsense-viewer, these are the warnings and errors I get:

`

20/10 12:28:59,492 WARNING [139818885437184] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61

 20/10 12:29:00,325 ERROR [139818348566272] (handle-libusb.h:51) failed to open usb interface: 0, error: RS2_USB_STATUS_ACCESS
 20/10 12:29:00,326 WARNING [139818977789696] (sensor.cpp:632) Exception was thrown when inspecting Raw RGB Camera property Enable / disable backlight compensation
 20/10 12:29:00,326 WARNING [139818977789696] (sensor.cpp:632) Exception was thrown when inspecting Raw RGB Camera property UVC image brightness
 20/10 12:29:00,326 WARNING [139818977789696] (sensor.cpp:632) Exception was thrown when inspecting Raw RGB Camera property UVC image contrast
 20/10 12:29:00,326 WARNING [139818977789696] (sensor.cpp:632) Exception was thrown when inspecting Raw RGB Camera property Controls exposure time of color camera. Setting any value will disable auto exposure
 20/10 12:29:00,326 WARNING [139818977789696] (sensor.cpp:632) Exception was thrown when inspecting Raw RGB Camera property UVC image gamma setting
 20/10 12:29:00,326 WARNING [139818977789696] (sensor.cpp:632) Exception was thrown when inspecting Raw RGB Camera property UVC image hue
 20/10 12:29:00,326 WARNING [139818977789696] (sensor.cpp:632) Exception was thrown when inspecting Raw RGB Camera property UVC image saturation setting
 20/10 12:29:00,326 WARNING [139818977789696] (sensor.cpp:632) Exception was thrown when inspecting Raw RGB Camera property UVC image sharpness setting
 20/10 12:29:00,326 WARNING [139818977789696] (sensor.cpp:632) Exception was thrown when inspecting Raw RGB Camera property Controls white balance of color image. Setting any value will disable auto white balance
 20/10 12:29:00,326 WARNING [139818977789696] (sensor.cpp:632) Exception was thrown when inspecting Raw RGB Camera property Enable / disable auto-exposure
 20/10 12:29:00,326 WARNING [139818977789696] (sensor.cpp:632) Exception was thrown when inspecting Raw RGB Camera property Enable / disable auto-white-balance
 20/10 12:29:01,993 WARNING [139818885437184] (messenger-libusb.cpp:42) control_transfer returned error, index: 300, error: No data available, number: 3d
 20/10 12:29:36,304 WARNING [139818885437184] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 20/10 12:29:39,393 WARNING [139818885437184] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
 20/10 12:29:47,535 WARNING [139818885437184] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 20/10 12:29:49,607 WARNING [139818885437184] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61
 20/10 12:29:51,680 WARNING [139818885437184] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61

`

Any suggestions or advice on how to resolve these errors, I am new to using the d435i.

[MartyG-RealSense]

Hi @ghokulji What computer or computing device are you using the D435i with, please? And which method are you using to build librealsense? (source code with patching, source code with RSUSB, Debian packages, etc)

There was a past case that had the same set of errors on Raspberry Pi 4 when building with the CMake flag FORCE_RSUSB_BACKEND=true.

#5310

If you are building librealsense from source code, you could try adding the flag -DCMAKE_BUILD_TYPE=Release if you are not doing so already in order to build librealsene with optimizations.

[ghokulji]

Hello @MartyG-RealSense,
I am using a Dell Laptop and I used this method for installation of librealsense https://github.com/IntelRealSense/librealsense/blob/development/doc/distribution_linux.md

Among the methods you have mentioned for building librealsense, which one should be used for which situation? I was confused at first and decided to go with https://github.com/IntelRealSense/librealsense/blob/development/doc/distribution_linux.md

[MartyG-RealSense]

Continuously generating control_transfer returned errors can indicate that there is a communication problem with the device. If it is only being generated in the log several times at the start of streaming and then no more then it may not be a serious problem.

RSUSB (a method of building from source code) is good for quick and easy set up and prototyping, as it can be installed over an internet connection, is not dependent on Linux versions or kernel versions and does not require patching. It does have drawbacks though which may render it unsuited for use in commercial projects.

The distribution method of building from packages is the one that Intel strongly recommends whenever possible.

The manual method of building from source code is good for commercial-standard projects.

The advantages and disadvantages of RSUSB versus kernel patching are discussed in the link below. Scroll down to the section of the linked-to comment that is headed What are the advantages and disadvantages of using libuvc vs patched kernel modules?

#5212 (comment)

[ghokulji]

Thanks for the explanation, I will check it out and read about it more.
Is there any explanation as to why the 3D image appears like that ( I have attached the image in the question above). Also,in the below image, I am placing it in front of a plain wall, but it doesn't display the whole wall.

I am trying to do RTAB- Mapping with the d435i and it requires the depth and rgb to be aligned, it would be great help if you can point me to some documentation from where I can align the depth and rgb.

Thnaks!

[MartyG-RealSense]

Images that look like the ones that you kindly supplied would cause me to think that performing a re-calibration of the camera's sensors with the Dynamic Calibrator software would be a good course of action. The large amount of errors in the log though cause me not to rule out that they may be having an influence on the results though.

Page 14 onward of the user guide for the Dynamic Calibrator tool explain how to install the tool on Linux.

https://www.intel.com/content/www/us/en/support/articles/000026723/emerging-technologies/intel-realsense-technology.html

Before that, let's try mapping the RGB onto the depth in the 3D point cloud mode of the Viewer, assuming that it is not already aligned.

1. Enable both the RGB and the Depth streams.

2. In 3D mode, left-click on the Texture option at the top of the Viewer window and select from its drop-down menu the option Intel RealSense D435i Color

The RGB should now map onto the point cloud.

[ghokulji]

I tried out the steps you have mentioned, this is the image I got which was proper.

Also another thing I noticed was that after 1 metre it doesn't display anything and shows a blank output. It only shows objects near. I am also getting the " (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: No data available, number: 61" ERROR continuously.

[MartyG-RealSense]

Please check in the Post-Processing section of the Stereo Module options in the side-panel of the Viewer for a filter called Threshold Filter and expand it open. Check the value of the Max Distance slider please. This option limits how far the camera can depth-sense when the filter is enabled.

You can either increase the value of the option to '10' (the maximum default depth sensing range of the 400 Series cameras) or left-click on the icon beside the Threshold Filter option to toggle it from blue color (On) to red (Off).

[ghokulji]

I changed the max distance parameter to 10m, but still it doesn't show beyond 1m approx.
This is the 2d depth and rgb

This is the 3D image, you can see that it is distorted.

Any idea what can be the cause behind this?

[MartyG-RealSense]

I ran some tests with the 3D mode by observing a pair of jeans that I have that are a similar color to your satchel bag. There did not seem to be any issues with depth-detecting that shade of color from the same distance from object that you are using. I was unable to replicate the extent of the missing detail in your own image though.

The 2D depth image looks okay, which makes me think that the problem likely is unrelated to depth range being capped.

Could you turn the icon beside the Post-Processing section heading to red please so that all post-processing filters are disabled? This will make the results more like those of a self-created application (which have no filters enabled by default) and eliminate the possibility that another filter is negatively affecting the image.

[ghokulji]

Hey @MartyG-RealSense

I tried out the method you have mentioned.

This is the image with Post-Processing disabled:

This is with Post-Processing enabled

The only difference I could see was that with the post-processing disabled the image was a bit wavering, whereas when enabled it was smoothened out and stable.

These are some of the errors I get repeatedly:

ERROR (synthetic-stream.cpp:48) Exception was thrown during user processing callback: Error occured during execution of the processing block! See the log for more info

ERROR (synthetic-stream.cpp:48) Exception was thrown during user processing callback: Out of frame resources!

WARNING (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11 [I get this warning continuously]

[MartyG-RealSense]

I believe that the errors are most likely related to the RGB stream rather than the depth stream. The Inspecting Raw RGB Camera errors in your opening message support this.

Next, could you try completely disabling the RGB Camera post-processing options by left-clicking on the blue icon beside the option to turn it to red (Off) please? When doing so, does it reduce the errors at all?

---

If RGB is the problem, an alternative may be to map infrared to depth instead.

First, enable Infrared 1 In the Stereo Module options. Do not enable RGB.

Start the depth and infrared streams by enabling the Stereo Module streaming.

In 3D mode, go to the Texture option. Because RGB is disabled and Infrared is enabled, the menu option Intel RealSense D435i Infrared 1 is now available. Select this option.

The infrared image is now mapped to depth on the point cloud.

The image will have the infrared dot pattern on it. If that poses a problem for your project, you can remove the pattern by setting the Emitter Enabled drop-down menu in the side-panel to Off, at the expense of some depth detail.

[ghokulji]

So, I tried disabling the rgb post-processing,this is the result.

This is the output of the infrared1

Is this the expected output of the realsense d435i? Also, may I know how to increase the width range, that is breadth-wise covering more space.

[MartyG-RealSense]

The D435 and D435i models have monochrome color on both of their infrared channels. D415 and D455 can have some color in the image if you set the infrared to the RGB8 format. This is because the left IR imager on these models can provide RGB color. The color shows up most strongly at close range.

You can increase the size of the horizontal field of view (FOV) by arranging more than one camera horizontally and overlapping their fields of view, and then stitching the individual point clouds together into a single combined point cloud. You can also arrange cameras vertically to widen the vertical FOV size.

[ghokulji]

Hi @MartyG-RealSense ,

Okay, I guess I will be able to carry on from here. Is there any documentation I can refer for setting up the post-processing parameters?
Thanks a lot for your amazing support!

[MartyG-RealSense]

You are very welcome :)

If you are using the RealSense Viewer then Intel's guide to post-processing should be fine for explaining the different types of filter.

https://dev.intelrealsense.com/docs/depth-post-processing

[ghokulji]

Thank you, I will close this issue :)