Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[D435i connection issue on Jetson Orin Nano] #13553

Closed
sooyoungmoon opened this issue Nov 27, 2024 · 19 comments
Closed

[D435i connection issue on Jetson Orin Nano] #13553

sooyoungmoon opened this issue Nov 27, 2024 · 19 comments

Comments

@sooyoungmoon
Copy link

sooyoungmoon commented Nov 27, 2024


Required Info
Camera Model { D435i }
Firmware Version 5.14.0
Operating System & Version Jetpack 6.0 Ubuntu 22.04.5 LTS
Kernel Version (Linux Only) 5.15.136-tegra
Platform NVIDIA Jetson Orin Nano
SDK Version (librealsense) 2.55.1
Language C++
Segment Robot

Hi, I am trying to do SLAM using Intel realsense2 D435i camera on my Jetson Orin Nano board.
But I'm experiencing connection issue with a D435i camera on my Jetson Orin Nano board.

Description

Issue: D435i Camera is not detected in realsense-viewer on my Jetson Orin Nano board

Progress until now:

  1. Built librealsense from source Linux Ubuntu Installation > An error occurred when I tried to execute ./scripts/patch-realsense-ubuntu-lts-hwe.sh
Installed dependencies except for QtCreator (optional)
$ git clone https://github.com/IntelRealSense/librealsense.git
$ ./scripts/setup_udev_rules.sh

Install librealsense2
	1. $ git clone https://github.com/IntelRealSense/librealsense.git 
	2. $ ./scripts/setup_udev_rules.sh  
	
	3.Build and apply patched kernel modules for Ubuntu 20/22 (focal/jammy) with LTS kernel 5.13, 5.15, 5.19, 6.2, 6.5
	
		$ ./scripts/patch-realsense-ubuntu-lts-hwe.sh > following error occurred!
		E: Unable to locate package linux-headers-5.15.136-tegra
		E: Couldn't find any package by glob 'linux-headers-5.15.136-tegra'
		E: Couldn't find any package by regex 'linux-headers-5.15.136-tegra'
		  	
Building librealsense2 SDK
- ...
$ cmake ../ -DBUILD_EXAMPLES=true
$ sudo make uninstall && make clean && make && sudo make install > no error
  1. Executed realsense-viewer > Camera was not detected!
$lsusb => D435i camera has been connected to the board, but not in the usb device list.
$rs-enumerate-devices => 'No device detected' 
  1. Followed the steps in LibUVC-backend installation
The same result as in 1) (i.e., camera is not detected in realsense-viewer and not listed in the usb device list) 
  1. Tried to follow the build steps explained in [Intel® RealSense™ camera driver for GMSL* interface] (https://github.com/IntelRealSense/realsense_mipi_platform_driver), but got stuck

    I cannot find where the script ./apply_patches.sh exist.

Questions:

Q#1. Do I have to download & build MIPI driver? I'm concerned that If it is mandatory or not. 	  

Q#2. I don't have a RealSense de-serialize board. Does it mean that I cannot install the MIPI platform driver to my jetson orin nano board?

Q#3 When I tried to execute the script ./scripts/patch-realsense-ubuntu-lts-hwe.sh, the following error occurred. 
  E: Couldn't find any package by glob 'linux-headers-5.15.136-tegra'
  E: Couldn't find any package by regex 'linux-headers-5.15.136-tegra'
      > What should I do to solve this?

Thank you for your support. I'm looking forward to your comment.

@MartyG-RealSense
Copy link
Collaborator

Hi @sooyoungmoon When installing from source code on Jetson, the installation_jetson.md instructions at the dedicated Jetson installation page at the link below should be used.

https://github.com/intelRealSense/librealsense/blob/master/doc/installation_jetson.md#building-from-source-using-native-backend

Jetson also has its own dedicated patch script called patch-realsense-ubuntu-L4T.sh that should be used instead of the hwe patch script.

https://github.com/IntelRealSense/librealsense/blob/master/scripts/patch-realsense-ubuntu-L4T.sh


You can also build from source using the libuvc backend procedure, which I note that you have aleady tried.


As you are using JetPack 6 with an IMU-equipped D435i camera, the non-detection is likely to be related to JP6 because it no longer has a function called hiddraw that RealSense IMU cameras depended on. As you are using Ubuntu 22.04, resolving the detection problem by downgrading to JetPack 5.1.3 may not be an option for you.


In regard to the approach of adding JP6 support for IMU by installing the MIPI driver, you can seek advice about it at the driver's dedicated support page at the link below.

https://github.com/IntelRealSense/realsense_mipi_platform_driver/issues

Q1: Building the MIPI driver is not mandatory if you can add IMU support by using JP 5.1.3 (for example, by downgrading to Ubuntu 20.04) or using the libuvc baackend installation procedure (which did not solve the problem for you in this particular case). If neither a JP downrade or the libuvc procedur work for you though then the MIPI driver would be the final option available, other than changing to a non-IMU camera model such as D435.

Q2: A deserializer board is not required to use the MIPI driver for the purpose of adding IMU support to a camera being used on a USB cable connection.

Q3: As mentioned above, the L4T patch script should be used with Jetson boards istead of the HWE patch script.


Intel also strongly recommend enabling the barrel jack power connector for Nano type Jetson boards, using the instructions at the link below.

https://jetsonhacks.com/2019/04/10/jetson-nano-use-more-power/

@shivani-varsha
Copy link

I had the same issue with D455 realsense camera connectivity with Jetson with Jetpack 6.0 , Ubuntu 22.04 and ROS2 humble version.
Will it be compatible if i downgrade to jetpack 5.1.3 , ubuntu 20.04 and ros1 noetic version, by just starting by cloning the librealsense package?

@MartyG-RealSense
Copy link
Collaborator

MartyG-RealSense commented Nov 29, 2024

@shivani-varsha The stated downgrade configuration should work.

@shivani-varsha
Copy link

@shivani-varsha The stated downgrade configuration should work.

Thanks for verifying.

@sooyoungmoon
Copy link
Author

sooyoungmoon commented Nov 29, 2024 via email

@MartyG-RealSense
Copy link
Collaborator

Yes they are two installation options, though if you use the RSUSB method then you do not need to use a kernel patch.

If you use method 2 then the Jetson kernel patch script patch-realsense-ubuntu-L4T.sh should be used.

https://github.com/IntelRealSense/librealsense/blob/master/scripts/patch-realsense-ubuntu-L4T.sh

I look forward to your next report. Good luck!

@sooyoungmoon
Copy link
Author

sooyoungmoon commented Dec 6, 2024

Hello @MartyG-RealSense , thank you for your support and
I managed to connect my D435i camera to Jetson Orin Nano!
I found an issue #13116 and followed your suggestion of uninstalling all the librealsense packages and reinstalling them (building from source using RSUSB backend in my case). Now I can see my camera is connected as follows:
(but, it seems that video streaming is not perfect, because I can see only small portion of images within the camera's view as shown in the picture below)

$ realsense-viewer

Screenshot from 2024-12-06 15-40-44

Question: Why don't the incoming images cover the whole camera's view and what can I do to get clear images?

$ rs-enumerate-devices

Device info: 
    Name                          : 	Intel RealSense D435I
    Serial Number                 : 	841612070518
    Firmware Version              : 	5.16.0.1
    Recommended Firmware Version  : 	5.16.0.1
    Physical Port                 : 	1-2.3-7
    Debug Op Code                 : 	15
    Advanced Mode                 : 	YES
    Product Id                    : 	0B3A
    Camera Locked                 : 	YES
    Usb Type Descriptor           : 	2.1
    Product Line                  : 	D400
    Asic Serial Number            : 	850123051257
    Firmware Update Id            : 	850123051257
    Dfu Device Path               : 	

Stream Profiles supported by Stereo Module
 Supported modes:
    STREAM      RESOLUTION     FORMAT      FPS
    Infrared 1  1280x720       Y8          @ 6 Hz
    Infrared 1   848x480       Y8          @ 10/8/6 Hz
    Infrared 1   640x480       Y8          @ 30/15/6 Hz
    Infrared 1   640x360       Y8          @ 30 Hz
    Infrared 1   480x270       Y8          @ 60/30/15/6 Hz
    Infrared 2  1280x720       Y8          @ 6 Hz
    Infrared 2   848x480       Y8          @ 10/8/6 Hz
    Infrared 2   640x480       Y8          @ 30/15/6 Hz
    Infrared 2   640x360       Y8          @ 30 Hz
    Infrared 2   480x270       Y8          @ 60/30/15/6 Hz
    Depth       1280x720       Z16         @ 6 Hz
    Depth        848x480       Z16         @ 10/8/6 Hz
    Depth        640x480       Z16         @ 30/15/6 Hz
    Depth        640x360       Z16         @ 30 Hz
    Depth        480x270       Z16         @ 60/30/15/6 Hz
    Depth        256x144       Z16         @ 90 Hz

Stream Profiles supported by RGB Camera
 Supported modes:
    STREAM      RESOLUTION     FORMAT      FPS
    Color       1920x1080      RGB8        @ 8 Hz
    Color           |          Y8          @ 8 Hz
    Color           |          BGRA8       @ 8 Hz
    Color           |          RGBA8       @ 8 Hz
    Color           |          BGR8        @ 8 Hz
    Color           |          YUYV        @ 8 Hz
    Color       1280x720       RGB8        @ 15/10/6 Hz
    Color           |          Y8          @ 15/10/6 Hz
    Color           |          BGRA8       @ 15/10/6 Hz
    Color           |          RGBA8       @ 15/10/6 Hz
    Color           |          BGR8        @ 15/10/6 Hz
    Color           |          YUYV        @ 15/10/6 Hz
    Color        640x480       RGB8        @ 30/15/6 Hz
    Color           |          Y8          @ 30/15/6 Hz
    Color           |          BGRA8       @ 30/15/6 Hz
    Color           |          RGBA8       @ 30/15/6 Hz
    Color           |          BGR8        @ 30/15/6 Hz
    Color           |          YUYV        @ 30/15/6 Hz
    Color        424x240       RGB8        @ 60/30/15/6 Hz
    Color           |          Y8          @ 60/30/15/6 Hz
    Color           |          BGRA8       @ 60/30/15/6 Hz
    Color           |          RGBA8       @ 60/30/15/6 Hz
    Color           |          BGR8        @ 60/30/15/6 Hz
    Color           |          YUYV        @ 60/30/15/6 Hz

Stream Profiles supported by Motion Module
 Supported modes:
    STREAM      FORMAT         FPS
    Accel       MOTION_XYZ32F  @ 250/63 Hz
    Gyro        MOTION_XYZ32F  @ 400/200 Hz

@shivani-varsha Please refer to my case and you will be able to connect your camera to the Jetson Orin Nano. Good luck!

@MartyG-RealSense
Copy link
Collaborator

Hi @sooyoungmoon I'm pleased to hear that your camera is now detected.

In regard to the images, you can see more complete images if you click on the '2D' option in the top corner of the RealSense Viewer window.

image

The '3D' mode that the above image is from is the Viewer's point cloud mode. Point cloud images are often like this. You could try changing the Preset* camera configuration option near the top of the Viewer's opions side-panel from 'Custom' to 'Medium Density' to see whether it improves the point cloud, as the Medium Density configuration preset provides a good balance between accuracy and the amount of depth detail on the image.

@sooyoungmoon
Copy link
Author

sooyoungmoon commented Dec 10, 2024

Summary: I could get clear image through the 2D mode.
Symptom: After I installed RTABMAP_ROS binary version and removed ros-humble-librealsense2, color image stream is not shown in realsense-viewer (**Suspicious msg: 'formats-converter.cpp:267 - Request: RGB8 Color, Resolved to: YUYV Color**) 
Question: What could be the reason for this and how can I solve the issue? 

Dear @MartyG-RealSense Thank you for your support!
As you suggested, I could get more complete image by clicking '2D' option of RealSense Viewer.
And I understand when I chose '3D' mode, the point cloud is shown like the picture I uploaded. I changed the Viewer's option from 'Custom' to 'Medium Density' but there was no noticeable difference on the surface.
Currently, I'm trying to get images from a realsense_camera_node from realsense_ros github repository.
But unfortunately, after I changed a few things (shown below) in my jetson orin nano, the color image stream is not shown in the realsense-viewer. The image stream is not shown in rviz too, when I launched realsense_camera_node in ROS2 workspace.

  • Changes I have made since my previous comment:

    1. I installed RTABMAP_ROS binary version using the command below (Some dependencies were also automatically installed (ex. ros-humble-librealsense2))
       $ sudo apt install ros-humble-rtabmap
    1. I removed the dependencies for ros-humble-rtabmap because I want to build librealsense from source (to get my camera detected, as you already know).

    2. I launched realsense_camera_node with other ROS2 nodes, but I could get image stream in rviz.

    3. I built librealsense again from source using RSUSB backend

    4. $ rs-enumerate-devices

         moon@moon-desktop:~/librealsense_build/librealsense-master/build$ rs-enumerate-devices
         Device info: 
         Name                          : 	Intel RealSense D435I
         Serial Number                 : 	841612070518
         Firmware Version              : 	5.16.0.1
         Recommended Firmware Version  : 	5.16.0.1
         Physical Port                 : 	1-2.3-17
         Debug Op Code                 : 	15
         Advanced Mode                 : 	YES
         Product Id                    : 	0B3A
         Camera Locked                 : 	YES
         Usb Type Descriptor           : 	2.1
         Product Line                  : 	D400
         Asic Serial Number            : 	850123051257
         Firmware Update Id            : 	850123051257
         Dfu Device Path               : 	
      
        Stream Profiles supported by Stereo Module
        Supported modes:
        STREAM      RESOLUTION     FORMAT      FPS
        Infrared 1  1280x720       Y8          @ 6 Hz
        Infrared 1   848x480       Y8          @ 10/8/6 Hz
        Infrared 1   640x480       Y8          @ 30/15/6 Hz
        Infrared 1   640x360       Y8          @ 30 Hz
        Infrared 1   480x270       Y8          @ 60/30/15/6 Hz
        Infrared 2  1280x720       Y8          @ 6 Hz
        Infrared 2   848x480       Y8          @ 10/8/6 Hz
        Infrared 2   640x480       Y8          @ 30/15/6 Hz
        Infrared 2   640x360       Y8          @ 30 Hz
        Infrared 2   480x270       Y8          @ 60/30/15/6 Hz
        Depth       1280x720       Z16         @ 6 Hz
        Depth        848x480       Z16         @ 10/8/6 Hz
        Depth        640x480       Z16         @ 30/15/6 Hz
        Depth        640x360       Z16         @ 30 Hz
        Depth        480x270       Z16         @ 60/30/15/6 Hz
        Depth        256x144       Z16         @ 90 Hz
      
        Stream Profiles supported by RGB Camera
        Supported modes:
        STREAM      RESOLUTION     FORMAT      FPS
        Color       1920x1080      RGB8        @ 8 Hz
        Color           |          Y8          @ 8 Hz
        Color           |          BGRA8       @ 8 Hz
        Color           |          RGBA8       @ 8 Hz
        Color           |          BGR8        @ 8 Hz
        Color           |          YUYV        @ 8 Hz
        Color       1280x720       RGB8        @ 15/10/6 Hz
        Color           |          Y8          @ 15/10/6 Hz
        Color           |          BGRA8       @ 15/10/6 Hz
        Color           |          RGBA8       @ 15/10/6 Hz
        Color           |          BGR8        @ 15/10/6 Hz
        Color           |          YUYV        @ 15/10/6 Hz
        Color        640x480       RGB8        @ 30/15/6 Hz
        Color           |          Y8          @ 30/15/6 Hz
        Color           |          BGRA8       @ 30/15/6 Hz
        Color           |          RGBA8       @ 30/15/6 Hz
        Color           |          BGR8        @ 30/15/6 Hz
        Color           |          YUYV        @ 30/15/6 Hz
        Color        424x240       RGB8        @ 60/30/15/6 Hz
        Color           |          Y8          @ 60/30/15/6 Hz
        Color           |          BGRA8       @ 60/30/15/6 Hz
        Color           |          RGBA8       @ 60/30/15/6 Hz
        Color           |          BGR8        @ 60/30/15/6 Hz
        Color           |          YUYV        @ 60/30/15/6 Hz
      
      Stream Profiles supported by Motion Module
      Supported modes:
      STREAM      FORMAT         FPS
      Accel       MOTION_XYZ32F  @ 250/63 Hz
      Gyro        MOTION_XYZ32F  @ 400/200 Hz
      
    5. $ realsense-viewer
      Again, only the depth stream is shown (image stream is not shown)
      Please refer to the following image.
      Screenshot from 2024-12-10 22-49-27

@MartyG-RealSense
Copy link
Collaborator

I recall a similar past problem when installing rtabmap_ros where realsense-viewer worked correctly again after rtabmap_ros was uninstalled.

@sooyoungmoon
Copy link
Author

sooyoungmoon commented Dec 14, 2024

Thank you for your advice!
Currently I'm trying to implement SLAM on nvidia-isaac-ros environment using Docker.
I think we can close this issue because the connection issue is already solved.
I will post another issue if necessary. Thanks!

@sooyoungmoon
Copy link
Author

Closing this issue.

@MartyG-RealSense
Copy link
Collaborator

You are very welcome, @sooyoungmoon - thanks very much for the update and please do post another issue if you need to. Good luck!

@jrateliff
Copy link

Could you state how you actually solved the connection issue? The tread is open-ended and does not communicate the resolution.

@sooyoungmoon
Copy link
Author

Could you state how you actually solved the connection issue? The tread is open-ended and does not communicate the resolution.

Hi @jrateliff, I managed to connect my D435 camera to a jetson orin nano board using the following steps (actually, they were discussed in the middle of this thread).

The steps I tried: (please refer to #13116 (#13116) )

  1. Deleted all the librealsense packages on Jetson board by running the command
    $ dpkg -l | grep "realsense" | cut -d " " -f 3 | xargs sudo dpkg --purge

  2. Reinstalled librealsense by building from source using RSUSB backend

You can test the result using rs-enumerate-devices command and realsense-viewer to see whether the camera is connected to your Jetson board.
Good luck!

@xingtongliuliumei
Copy link

My D435i is successfully connected to the Jetson Nano, and I can open and display it using realsense-viewer. However, I am unable to retrieve IMU data, and the program freezes. Another issue is that in my Python script, when calling profile = pipeline.start(config), it shows 'no device connected', although lsusb clearly shows that the device is connected." need your help

@MartyG-RealSense
Copy link
Collaborator

Hi @xingtongliuliumei Are you using JetPack 6 please? If you are then JP6 can have problems accessing the IMU, or detecting the camera at all. This is because - as mentioned earlier in this discussion at #13553 (comment) - JP6 lacks a function called hiddraw that IMU-equipped RealSense camera models such as D435i depend upon.

@xingtongliuliumei
Copy link

xingtongliuliumei commented Jan 17, 2025

Yes, I’m using JetPack 6.1, and I installed it with the and add -DBUILD_WITH_python=true flag. The main issue is that while I can open RealSense Viewer, I still cannot access the device in the Python script. It shows 'No device connected'.

@MartyG-RealSense
Copy link
Collaborator

@xingtongliuliumei Also including the flag -DFORCE_RSUSB_BACKEND=true to build librealsense in RSUSB mode can sometimes help when using an IMU-equipped RealSense camera with JP6, though it does not solve the problem in every case, especially ones that involve camera non-detection in pyrealsense2.

Have you tried launching your script with sudo admin permissions by putting sudo in front of the launch command? For example, sudo python3 test.py

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants