Realsense D455 Processing Speed #260
Comments
|
Hi,
If you want to speed up,
1) Only 10-15hz image might be enough
2) less slam features
3) less msckf features
You can tune these parameters to speed up the system.
best,
Yulin
…On Sat, Jul 16, 2022 at 8:30 AM myboyhood ***@***.***> wrote:
Thansk for the OpenVINS project, I have learnd it about 6 months!
The question is the time-consuming performance in feature delay,
initialize and update stage is normal ?
My result is record as following [ Realsense D455 , mono color 640 * 480
30Hz + IMU 200Hz] [running on my laptop computer 6*2.6GHz CPU Core]:
[TIME]: 0.0032 seconds for tracking
[TIME]: 0.0136 seconds for propagation
[TIME]: 0.1252 seconds for MSCKF update (10 feats)
[TIME]: 0.1449 seconds for SLAM update (66 feats)
[TIME]: 0.1194 seconds for SLAM delayed init (4 feats)
[TIME]: 0.0101 seconds for re-tri & marg (11 clones in state)
[TIME]: 0.4165 seconds for total (camera 0)
If I change to stereo version, The time will increase up to even total
1.44s per period.
As I know, when we play the ETH euroc dataset to test OpenVINS, we always
set the queue size in callback as 9999(as large as possible) in order to
process all images one by one. we ignore the real-time performance.
But when we test in real-time environment, For example, 30Hz image stream
request VIO system to spend below 30ms to process image and imu info. I
find it is hard for OpenVINS system.
I also tried VINS-Mono ro VINS-Fusion before, The optimization backend is
also time-costly.
Kind friends could help me ?
—
Reply to this email directly, view it on GitHub
<#260>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADJAQCF337FKN2HRQNNUX6DVULIP3ANCNFSM53YH4RUA>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
--
With Best Regards,
Linde Yang / Yulin Yang
*UNIVERSITY of DELAWARE*
PhD, Department of Mechanical Engineering
328 Spencer Lab, Newark, DE, USA, 19716-3140
Mobile: +(01) 302-690-3926
mailto: ***@***.***> ***@***.***
Web: https://yangyulin.net <https://yangyulin.net>
|
|
Can you provide the config you are using? And if possible an example bag
with the roslaunch command to run it.
…On Sat, Jul 16, 2022, 11:30 AM myboyhood ***@***.***> wrote:
Thansk for the OpenVINS project, I have learnd it about 6 months!
The question is the time-consuming performance in feature delay,
initialize and update stage is normal ?
My result is record as following [ Realsense D455 , mono color 640 * 480
30Hz + IMU 200Hz] [running on my laptop computer 6*2.6GHz CPU Core]:
[TIME]: 0.0032 seconds for tracking
[TIME]: 0.0136 seconds for propagation
[TIME]: 0.1252 seconds for MSCKF update (10 feats)
[TIME]: 0.1449 seconds for SLAM update (66 feats)
[TIME]: 0.1194 seconds for SLAM delayed init (4 feats)
[TIME]: 0.0101 seconds for re-tri & marg (11 clones in state)
[TIME]: 0.4165 seconds for total (camera 0)
If I change to stereo version, The time will increase up to even total
1.44s per period.
As I know, when we play the ETH euroc dataset to test OpenVINS, we always
set the queue size in callback as 9999(as large as possible) in order to
process all images one by one. we ignore the real-time performance.
But when we test in real-time environment, For example, 30Hz image stream
request VIO system to spend below 30ms to process image and imu info. I
find it is hard for OpenVINS system.
I also tried VINS-Mono ro VINS-Fusion before, The optimization backend is
also time-costly.
Kind friends could help me ?
—
Reply to this email directly, view it on GitHub
<#260>, or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAQ6TYUJM24UWVXH6C5MKSLVULIPZANCNFSM53YH4RUA>
.
You are receiving this because you are subscribed to this thread.Message
ID: ***@***.***>
|
Hi, I am really appreciated to your suggestion.
As analysis above, three part [propagation part] [MSCKF update] [SLAM delayed init] take most time. Firstly, I doubt the running speed of my computer, maybe the single 2.6GHz cpu core running OpenVINS is really laborious. Secondly, I found Realsense D455 is not stable. When the image stream is set to 15Hz and IMU is set to 200Hz. After a while, the image frequency drops to 13 or 8Hz strangely, imu data is also unstable (the USB3.1 cable loose? or the device is overheat? I am really troubled with those unstable devices, sad :-( ) . So I really want to get help which device RPNG usually use. I see xsens MTi in your paper, but I have no idea about the camera. Finally, I am PhD candidate at robotics institution of Shanghai Jiao Tong University. Our lab mainly focuses on uav perception, navigation and control. We hope to have more friendship with RPNG, here is my mentor's [bio link] and introduction [bilibili video] |
goldbattle
added
the
user-platform
|
Hi,
When using a 15hz image, you don't need to reset the realsense sensor.
Instead, you can modify the main file for MSCKF code and skip images/ros
messages when feeding images. For example, you can still use the camera to
capture a 30hz image but skip one image from every two images when feeding
to the msckf estimator.
Realsense sensors in most cases work pretty well in their default settings.
I have tested quite a few Realsense sensors before.
Rare cases that the sensor will fail. But I want to point out that the time
offset between the camera and IMU of real sensor sensor sometimes drifts
(around 5ms for RS T265)
The imu propagation should not be so slow (as 20ms, that is 0.02s). I guess
the computer might be slow. 🙈
If you just want to run the OpenVINS fast enough on computer, try to apply
a better laptop😎
(If the above measures still does not help the speed, you can also try to
use less clones (not recommended), or detect less features from the image
or scale the image to lower resolution. These might lower the performance a
lot )
For collaborations, thanks for your interests but please directly reach out
to Prof. Huang. Let's keep github as a place for technical discussion.
…On Sat, Jul 16, 2022 at 6:56 PM myboyhood ***@***.***> wrote:
Hi, If you want to speed up, 1) Only 10-15hz image might be enough 2) less
slam features 3) less msckf features You can tune these parameters to speed
up the system. best, Yulin
… <#m_4365259778044307397_>
On Sat, Jul 16, 2022 at 8:30 AM myboyhood *@*.*> wrote: Thansk for the
OpenVINS project, I have learnd it about 6 months! The question is the
time-consuming performance in feature delay, initialize and update stage is
normal ? My result is record as following [ Realsense D455 , mono color 640
* 480 30Hz + IMU 200Hz] [running on my laptop computer 62.6GHz CPU Core]:
[TIME]: 0.0032 seconds for tracking [TIME]: 0.0136 seconds for propagation
[TIME]: 0.1252 seconds for MSCKF update (10 feats) [TIME]: 0.1449 seconds
for SLAM update (66 feats) [TIME]: 0.1194 seconds for SLAM delayed init (4
feats) [TIME]: 0.0101 seconds for re-tri & marg (11 clones in state)
[TIME]: 0.4165 seconds for total (camera 0) If I change to stereo version,
The time will increase up to even total 1.44s per period. As I know, when
we play the ETH euroc dataset to test OpenVINS, we always set the queue
size in callback as 9999(as large as possible) in order to process all
images one by one. we ignore the real-time performance. But when we test in
real-time environment, For example, 30Hz image stream request VIO system to
spend below 30ms to process image and imu info. I find it is hard for
OpenVINS system. I also tried VINS-Mono ro VINS-Fusion before, The
optimization backend is also time-costly. Kind friends could help me ? —
Reply to this email directly, view it on GitHub <#260
<#260>>, or unsubscribe
https://github.com/notifications/unsubscribe-auth/ADJAQCF337FKN2HRQNNUX6DVULIP3ANCNFSM53YH4RUA
<https://github.com/notifications/unsubscribe-auth/ADJAQCF337FKN2HRQNNUX6DVULIP3ANCNFSM53YH4RUA>
. You are receiving this because you are subscribed to this thread.Message
ID: @.*
*> -- With Best Regards, Linde Yang / Yulin Yang UNIVERSITY of DELAWARE
PhD, Department of Mechanical Engineering 328 Spencer Lab, Newark, DE, USA,
19716-3140 Mobile: +(01) 302-690-3926 mailto: @.**> @.** Web:
https://yangyulin.net https://yangyulin.net
Hi, I am really appreciated to your suggestion.
1.
I would try 15Hz image stream on Realsense D455 later
2.
About the backend processing time, I carefully make statistics about
each step.
[tracking part]: using Fast + optiflow costs 3ms
[propagation part]: The time is linear to the size of IMU integration.
For example, 10 times IMU integration and state argument costs about 10ms.
if image stream is set to 10Hz, IMU frequency keeps 200Hz, in each period,
there will be 20 IMU data to integrate and state argument, about 20ms.
[MSCKF update]: each MSCKF feature update would cost average 12.5ms.
This process includes triangulation, state update.(here I use max_clones =
11, which means the number of camera pose used in MSCKF feats is less than
11)
[SLAM update]: each SLAM feats update would cost 2.2ms, For example,
if I set max_slam_feats = 20, it will cost 44ms in SLAM update.
[SLAM delayed init]: each SLAM delayed init process would cost 30ms.
This process includes feature triangulation and G-N optimization, Since the
observation count is 11 times, triangulation would be a little
time-consuming. if I set max_clones = 5, it will speed up but the accuracy
of feats and reprojection in state update would suffer.
[re-tri & marg]:This part costs stably 9~10ms.
As analysis above, three part [propagation part] [MSCKF update] [SLAM
delayed init] take most time.
Firstly, I doubt the running speed of my computer, maybe the single 2.6GHz
cpu core running OpenVINS is really laborious.
Secondly, I found Realsense D455 is not stable. When the image stream is
set to 15Hz and IMU is set to 200Hz. After a while, the image frequency
drops to 13 or 8Hz strangely, imu data is also unstable (the USB3.1 cable
loose? or the device is overheat? I am really troubled with those unstable
devices, sad :-( ) . So I really want to get help which device RPNG usually
use. I see xsens MTi in your paper, but I have no idea about the camera.
Finally, I am PhD candidate at robotics institution of Shanghai Jiao Tong
University. Our lab mainly focuses on uav perception, navigation and
control. We hope to have more friendship with RPNG, here is my mentor's [bio
link <https://me.sjtu.edu.cn/teacher_directory1/dongwei2.html>] and
introduction [bilibili video
<https://www.bilibili.com/video/BV1gv411J76K?share_source=copy_web>]
—
Reply to this email directly, view it on GitHub
<#260 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADJAQCCDG5MLRRHMPJ5HIA3VUNR4NANCNFSM53YH4RUA>
.
You are receiving this because you commented.Message ID:
***@***.***>
--
With Best Regards,
Linde Yang / Yulin Yang
*UNIVERSITY of DELAWARE*
PhD, Department of Mechanical Engineering
328 Spencer Lab, Newark, DE, USA, 19716-3140
Mobile: +(01) 302-690-3926
mailto: ***@***.***> ***@***.***
Web: https://yangyulin.net <https://yangyulin.net>
|
Hi. Thanks for your reply. I tried the rosbag again on my desktop (3.6GHz per core)as suggested by [yangyulin](https://github.com/yangyulin) . Get info as following:
I share my rosbag and launch file on [Amazon Drive] Thanks for your kind help ! |
|
Please use the newest version of OpenVINS (you launch file is for a very old version). You will need to create a config yaml file specific to your platform: Additionally, you need to make sure to change the realsense launch to fuse the gyroscope and accel. EDIT: Also make sure you perform calibration of the camera and IMU using kalibr. You should not trust the values reported by intel from their driver for intrinsics and extrinsics between the IMU and camera. Here are some example results I have used in the past for the d455 in its stereo mode. |
Yes! The latest version develop_v2.6.1 has been implemented on my computer. It is really faster!
Is the speed improved by multi-thread, async or other method? It is now can run on my Realsense D455 at 30Hz infra stereo mode. |
goldbattle
changed the title
|
Glad you were able to get it working. I am not sure what caused it originally to be slow, but likely was just a much older version of the project, or maybe different compile flags. |
|
Example D455 config file is now in the repo here: https://github.com/rpng/open_vins/tree/master/config/rs_d455 |
|
Thank you very much. I am graceful for the testing. The above problem confused me for a long time in the last year. Then, I use s-g filter to filter the raw imu data, the imu data become very stable and the time-delay is not obvious. After this s-g filter, I can execute OpenVINS successful. |
|
The UAV rotors will have an impact on how noisy the IMU readings are. I think that physically damping the sensor platform (e.g. rubber rings) or filtering the readings is reasonable to try to address this. Additionally inflating the IMU noises from the stationary ones. The question is by how much to increase it... Do you have the exact filter you are using? Are you able to share a bag with the noisy IMU data in it? |
|
Thank you very much. I did refer the paper path: FF4-odom-imu_2022-12-10-21-22-24.bag
version: 2.0
duration: 1:45s (105s)
start: Dec 10 2022 21:22:24.30 (1670678544.30)
end: Dec 10 2022 21:24:09.45 (1670678649.45)
size: 53.7 MB
messages: 91865
compression: none [71/71 chunks]
types: nav_msgs/Odometry [cd5e73d190d741a2f92e81eda573aca7]
sensor_msgs/Imu [6a62c6daae103f4ff57a132d6f95cec2]
topics: /T265_2/camera/odom/sample 21064 msgs : nav_msgs/Odometry
/imu/data 21032 msgs : sensor_msgs/Imu
/imu/data_filter 16938 msgs : sensor_msgs/Imu
/ov_msckf/odomimu 16434 msgs : nav_msgs/Odometry
/ov_msckf/odomimu_filtered 16397 msgs : nav_msgs/OdometryI found OpenVINS is not smooth as T265, So I have to adopt s-g filte for I also provide my s-g filter in gitee myboyhood/aggressive201lss hoping for your reply ! |
Thansk for the OpenVINS project, I have learnd it about 6 months!
The question is the time-consuming performance in feature delay, initialize and update stage is normal ?
My result is record as following [ Realsense D455 , mono color 640 * 480 30Hz + IMU 200Hz] [running on my laptop computer 6*2.6GHz CPU Core]:
If I change to stereo version, The time will increase up to even total 1.44s per period.
As I know, when we play the ETH euroc dataset to test OpenVINS, we always set the queue size in callback as 9999(as large as possible) in order to process all images one by one. we ignore the real-time performance.
But when we test in real-time environment, For example, 30Hz image stream request VIO system to spend below 30ms to process image and imu info. I find it is hard for OpenVINS system.
I also tried VINS-Mono ro VINS-Fusion before, The optimization backend is also time-costly.
Kind friends could help me ?
The text was updated successfully, but these errors were encountered: