Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Realsense D455 IMU Problems #13591

Open
SnoopyGeeek opened this issue Dec 10, 2024 · 12 comments
Open

Realsense D455 IMU Problems #13591

SnoopyGeeek opened this issue Dec 10, 2024 · 12 comments

Comments

@SnoopyGeeek
Copy link


Required Info
Camera Model { D455 }
Firmware Version (Open RealSense Viewer
Operating System & Version {Ubuntu 22.04
Kernel Version (Linux Only) (e.g. 4.14.13)
Platform Jetson Orin AGX
SDK Version 2.55.1}
Language {C++}
Segment { }

Issue Description

I upgraded my agx with SDKManager from Jetpack 5.0 to Jetpack 6.1. I followed the instruction of build librealsense from the source using libuvc following the link: https://github.com/IntelRealSense/librealsense/blob/master/doc/libuvc_installation.md.
Everything looks fine when I opened realsense-viewer. It can show up the streams of stereo module and rgb module. However the realsense-viewer collapse when I turned motion module on. It seems like IMU module has some problems.
image

At the same time, I installed ros2 wrapper and tried ros2 launch realsense2_camera rs_launch.py and I met the same problem as #issue10902:#10902. Here is the error:

[INFO] [launch]: All log files can be found below /home/amrapali/.ros/log/2022-09-14-17-31-12-940522-ubuntu-47043
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [realsense2_camera_node-1]: process started with pid [47046]
[realsense2_camera_node-1] [INFO] [1663156874.616471375] [camera.camera]: RealSense ROS v4.0.4
[realsense2_camera_node-1] [INFO] [1663156874.616810512] [camera.camera]: Built with LibRealSense v2.51.1
[realsense2_camera_node-1] [INFO] [1663156874.616917425] [camera.camera]: Running with LibRealSense v2.51.1
[realsense2_camera_node-1] [INFO] [1663156875.111891766] [camera.camera]: Device with serial number 018322071462 was found.
[realsense2_camera_node-1]
[realsense2_camera_node-1] [INFO] [1663156875.114755875] [camera.camera]: Device with physical ID 2-3.3-3 was found.
[realsense2_camera_node-1] [INFO] [1663156875.114876803] [camera.camera]: Device with name Intel RealSense D435I was found.
[realsense2_camera_node-1] [INFO] [1663156875.116949036] [camera.camera]: Device with port number 2-3.3 was found.
[realsense2_camera_node-1] [INFO] [1663156875.117116493] [camera.camera]: Device USB type: 3.2
[realsense2_camera_node-1] [INFO] [1663156875.140857334] [camera.camera]: getParameters...
[realsense2_camera_node-1] [WARN] [1663156875.146055885] [camera.camera]: JSON file provided doesn't exist! (/application/bot_ws/src/realsense/realsense-ros/realsense2_camera/config/high_accuracy.json)
[realsense2_camera_node-1] [INFO] [1663156875.146225677] [camera.camera]: Device Name: Intel RealSense D435I
[realsense2_camera_node-1] [INFO] [1663156875.146318062] [camera.camera]: Device Serial No: 018322071462
[realsense2_camera_node-1] [INFO] [1663156875.146394926] [camera.camera]: Device physical port: 2-3.3-3
[realsense2_camera_node-1] [INFO] [1663156875.146482798] [camera.camera]: Device FW version: 05.13.00.50
[realsense2_camera_node-1] [INFO] [1663156875.146619311] [camera.camera]: Device Product ID: 0x0B3A
[realsense2_camera_node-1] [INFO] [1663156875.148091926] [camera.camera]: Sync Mode: Off
[realsense2_camera_node-1] [WARN] [1663156878.791794606] [camera.camera]: re-enable the stream for the change to take effect.
[realsense2_camera_node-1] [WARN] [1663156880.052438754] [camera.camera]: re-enable the stream for the change to take effect.
[realsense2_camera_node-1] [INFO] [1663156880.114970453] [camera.camera]: Stopping Sensor: Stereo Module
[realsense2_camera_node-1] [INFO] [1663156880.249313989] [camera.camera]: Starting Sensor: Stereo Module
[realsense2_camera_node-1] 14/09 17:31:20,469 WARNING [281472476891552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[realsense2_camera_node-1] [INFO] [1663156880.522522681] [camera.camera]: Open profile: stream_type: Depth(0), Format: Z16, Width: 640, Height: 480, FPS: 15
[realsense2_camera_node-1] [INFO] [1663156880.525715431] [camera.camera]: Stopping Sensor: RGB Camera
[realsense2_camera_node-1] [INFO] [1663156880.534353773] [camera.camera]: Starting Sensor: RGB Camera
[realsense2_camera_node-1] [INFO] [1663156880.648117347] [camera.camera]: Open profile: stream_type: Color(0), Format: RGB8, Width: 640, Height: 480, FPS: 15
[realsense2_camera_node-1] [INFO] [1663156880.649612841] [camera.camera]: RealSense Node Is Up!
[realsense2_camera_node-1] [WARN] [1663156880.663337798] [camera.camera]:
[realsense2_camera_node-1] 14/09 17:31:20,936 WARNING [281472476891552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[realsense2_camera_node-1] 14/09 17:31:21,089 WARNING [281472476891552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11
[realsense2_camera_node-1] 14/09 17:31:21,342 WARNING [281472476891552] (messenger-libusb.cpp:42) control_transfer returned error, index: 768, error: Resource temporarily unavailable, number: 11

@edfus
Copy link

edfus commented Dec 10, 2024

Facing the same issue here. Is there any update or potential solution available? @MartyG-RealSense

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Dec 10, 2024

Hi @SnoopyGeeek Does the Motion Module in realsense-viewer work if you boot up the computer from shutdown using the power button and then launch realsense-viewer and only enable Motion Module on its own?

The realsense-viewer has a known issue where for some RealSense users, if Depth, RGB and Motion Module are all enabled simultaneously then a stream failure occurs. There is no fix for this except to only enable Motion Module on its own, or just two streams instead of three (such as Depth + Motion Module, or RGB + Motion Module).

In the RealSense ROS wrapper, (which does not have the three-stream IMU issue in realsense-viewer) the IMU topics are disabled by default and will not be published. You can enable them in the launch instruction by adding to it enable_accel:=true enable_gyro:=true unite_imu_method:=2

ros2 launch realsense2_camera rs_launch.py enable_accel:=true enable_gyro:=true unite_imu_method:=2

This enables the accel and gyro streams and publishes them in a single combined /imu topic.

@SnoopyGeeek
Copy link
Author

Hi @SnoopyGeeek Does the Motion Module in realsense-viewer work if you boot up the computer from shutdown using the power button and then launch realsense-viewer and only enable Motion Module on its own?

The realsense-viewer has a known issue where for some RealSense users, if Depth, RGB and Motion Module are all enabled simultaneously then a stream failure occurs. There is no fix for this except to only enable Motion Module on its own, or just two streams instead of three (such as Depth + Motion Module, or RGB + Motion Module).

In the RealSense ROS wrapper, (which does not have the three-stream IMU issue in realsense-viewer) the IMU topics are disabled by default and will not be published. You can enable them in the launch instruction by adding to it enable_accel:=true enable_gyro:=true unite_imu_method:=2

ros2 launch realsense2_camera rs_launch.py enable_accel:=true enable_gyro:=true unite_imu_method:=2

This enables the accel and gyro streams and publishes them in a single combined /imu topic.

No, single Motion Module does not work for me in my situation. It shows reccieved no frames .Here is my result:
d96fe34ceecf0f92da76adcfbb99bab1
As for realsense ros wrapper, I tired running your script. The result remained the same as before---control_transfer returned error, The result screenshot is as follows:
615f8674d7a130c29c124bf24074a34d
Could you plz help me out here. Our team has been stuck with the realsense driver problem for weeks.

@MartyG-RealSense
Copy link
Collaborator

Do the IMU streams continue to be unavailable if you reset the camera?

This can be done in the Viewer by going to the 'Reset Hardware' option under the drop-down menu of the 'More' option near the top of the options side-panel.

In ROS it can be done by adding initial_reset:=true to the launch instruction.

@SnoopyGeeek
Copy link
Author

Do the IMU streams continue to be unavailable if you reset the camera?

This can be done in the Viewer by going to the 'Reset Hardware' option under the drop-down menu of the 'More' option near the top of the options side-panel.

In ROS it can be done by adding initial_reset:=true to the launch instruction.

Yes, I tried reset the camera, the problem still remain the same.
I noticed an interesting thing is that D435i work fine in the morning, but found the same problem as D455 in the afternoon. What a magic!
jdOMtC8SPUrSGaV4tIVr

@MartyG-RealSense
Copy link
Collaborator

I note that your images have librealsense versions 2.55.1 and 2.56.3. If these pictures are taken on the same computer then this would indicate that there is more than one librealsense version installed on the computer. There should only be one librealsense version installed, as multiple librealsense installations on the same computer could result in conflicts.

There is also a ROS log with librealsense 2.51.1 and ROS wrapper 4.0.4. Again, there should only be one librealsense version on the computer. The ROS wrapper version that should also be used with 2.51.1 is 4.51.1. The 4.0.4 version of the wrapper is nearly three years old and should not be used with modern librealsense versions.

@SnoopyGeeek
Copy link
Author

I note that your images have librealsense versions 2.55.1 and 2.56.3. If these pictures are taken on the same computer then this would indicate that there is more than one librealsense version installed on the computer. There should only be one librealsense version installed, as multiple librealsense installations on the same computer could result in conflicts.

There is also a ROS log with librealsense 2.51.1 and ROS wrapper 4.0.4. Again, there should only be one librealsense version on the computer. The ROS wrapper version that should also be used with 2.51.1 is 4.51.1. The 4.0.4 version of the wrapper is nearly three years old and should not be used with modern librealsense versions.

Sorry, the image and log file in the initial issue publishment were copied from other place with the same problem, because I did not have the results to publish at hand. Except that, the other images and results are directly from my agx screenshots. As you see, I have librealsense V2.55.1 and realsense ROS wrapper V4.55.1 installed.

@MartyG-RealSense
Copy link
Collaborator

Having eliminated other possible causes, it may be that you are having problems with your IMU because you are using JetPack 6. RealSense cameras equipped with an IMU on a Jetson using JP6 can have problems ranging from no IMU data to the camera not being detected at all. This is because JP6 removed an instruction called 'hidraw' that RealSense IMU-equipped models (known as HID devices) relied upon.

Sometimes using the libuvc_installation.md instructions like you did resolves the problem, but not on every occasion. Some have found that whilst using the libuvc_installation.sh build script did not resolve their IMU issue, building from source code with the similar RSUSB backend method did - please see #9931 (comment) for a guide to this method.

The other way of adding IMU support on a Jetson using JP6 on a USB cable connection is to perform a complicated installation procedure for the MIPI driver at the link below.

https://github.com/IntelRealSense/realsense_mipi_platform_driver

Using JetPack 5.1.3 instead of 6.1 should also enable to the IMU to work, though I understand that JP6 is needed for Ubuntu 22.04, so using 5.1.3 would likely require downgrading to Ubuntu 20.04.

@SnoopyGeeek
Copy link
Author

Having eliminated other possible causes, it may be that you are having problems with your IMU because you are using JetPack 6. RealSense cameras equipped with an IMU on a Jetson using JP6 can have problems ranging from no IMU data to the camera not being detected at all. This is because JP6 removed an instruction called 'hidraw' that RealSense IMU-equipped models (known as HID devices) relied upon.

Sometimes using the libuvc_installation.md instructions like you did resolves the problem, but not on every occasion. Some have found that whilst using the libuvc_installation.sh build script did not resolve their IMU issue, building from source code with the similar RSUSB backend method did - please see #9931 (comment) for a guide to this method.

The other way of adding IMU support on a Jetson using JP6 on a USB cable connection is to perform a complicated installation procedure for the MIPI driver at the link below.

https://github.com/IntelRealSense/realsense_mipi_platform_driver

Using JetPack 5.1.3 instead of 6.1 should also enable to the IMU to work, though I understand that JP6 is needed for Ubuntu 22.04, so using 5.1.3 would likely require downgrading to Ubuntu 20.04.

Perhaps the reason is that JP6 is too new to be supported by librealsense? Bro, do you know when will it be fully supported, like I can get pre-compiled deb file by using apt install command. Do you put it on schedule?

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Dec 12, 2024

JetPack 6 is supported officially by librealsense from beta 2.56.1 onwards. This does not resolve the IMU related issues with JP6 though, because it is related to JP6 removing its 'hiddraw' function and not a fault in librealsense. So performing a libuvc backend source code installation of librealsense or installing the MIPI driver is still required to provide IMU support when using JP6.

A major release of librealsense (one which has packages) is planned for 'end of year'.

@SnoopyGeeek
Copy link
Author

JetPack 6 is supported officially by librealsense from beta 2.56.1 onwards. This does not resolve the IMU related issues with JP6 though, because it is related to JP6 removing its 'hiddraw' function and not a fault in librealsense. So performing a libuvc backend source code installation of librealsense or installing the MIPI driver is still required to provide IMU support when using JP6.

A major release of librealsense (one which has packages) is planned for 'end of year'.

Ok, does it mean that we can only expect nvidia to fix the jetpack6-realsense imu compatibility issue, is that right?

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Dec 12, 2024

I would assume not. At the moment, the suggested workarounds are the only fixes available for JetPack 6 users with an IMU-equipped RealSense camera.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants