rclcpp::exceptions::RCLError

[doisyg]

Bug report

Required Info:

• Operating System:
  • Ubuntu 22.04
• ROS2 Version: Humble last sync, debian install
• Version or commit hash: from source, last humble branch, 91883ce
• DDS implementation:
  • Cyclone, localhost only

Steps to reproduce issue

Struggling with this one. It is maybe not nav2 related but reporting here in case somebody run into the same issue.

Let the navigation run for several hours. After about 2 hours I get a crash on the controller server (compiled with backward_ros to get stack trace):

Jul 27 02:14:09 bash[813572]: [controller_server-66] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
Jul 27 02:14:09 bash[813572]: [controller_server-66] what(): rcl_wait() failed: error not set
Jul 27 02:14:09 bash[813572]: [controller_server-66] Stack trace (most recent call last) in thread 816228:
Jul 27 02:14:09 bash[813572]: [controller_server-66] #14 Object "", at 0xffffffffffffffff, in
Jul 27 02:14:09 bash[813572]: [controller_server-66] #13 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc4eee369ff, in
Jul 27 02:14:10 bash[813572]: [controller_server-66] #12 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc4eeda4b42, in
Jul 27 02:14:10 bash[813572]: [controller_server-66] #11 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7fc4ef0342c2, in
Jul 27 02:14:10 bash[813572]: [controller_server-66] #10 Object "/opt/ros/humble/lib/librclcpp.so", at 0x7fc4ef44dc80, in rclcpp::executors::SingleThreadedExecutor::spin()
Jul 27 02:14:10 bash[813572]: [controller_server-66] #9 Object "/opt/ros/humble/lib/librclcpp.so", at 0x7fc4ef445972, in rclcpp::Executor::get_next_executable(rclcpp::AnyExecutable&, std::chrono::duration<long, std::ratio<1l, 1000000000l> >)
Jul 27 02:14:10 bash[813572]: [controller_server-66] #8 Object "/opt/ros/humble/lib/librclcpp.so", at 0x7fc4ef4435cd, in rclcpp::Executor::wait_for_work(std::chrono::duration<long, std::ratio<1l, 1000000000l> >)
Jul 27 02:14:10 bash[813572]: [controller_server-66] #7 Object "/opt/ros/humble/lib/librclcpp.so", at 0x7fc4ef43f9c8, in rclcpp::exceptions::throw_from_rcl_error(int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, rcutils_error_state_s const*, void (*)())
Jul 27 02:14:10 bash[813572]: [controller_server-66] #6 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7fc4ef00627d, in std::rethrow_exception(std::__exception_ptr::exception_ptr)
Jul 27 02:14:10 bash[813572]: [controller_server-66] #5 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7fc4ef0062f6, in std::terminate()
Jul 27 02:14:10 bash[813572]: [controller_server-66] #4 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7fc4ef00628b, in
Jul 27 02:14:10 bash[813572]: [controller_server-66] #3 Object "/usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.30", at 0x7fc4eeffabfd, in
Jul 27 02:14:10 bash[813572]: [controller_server-66] #2 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc4eed387f2, in abort
Jul 27 02:14:10 bash[813572]: [controller_server-66] #1 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc4eed52475, in raise
Jul 27 02:14:10 bash[813572]: [controller_server-66] #0 Object "/usr/lib/x86_64-linux-gnu/libc.so.6", at 0x7fc4eeda6a7c, in pthread_kill
Jul 27 02:14:10 bash[813572]: [controller_server-66] Aborted (Signal sent by tkill() 814467 1000)
Jul 27 02:14:10 bash[813572]: [ERROR] [controller_server-66]: process has died [pid 814467, exit code -6, cmd '/opt/wyca/elodie/nav2_controller/lib/nav2_controller/controller_server --ros-args --params-file /home/wyca/.wyca/nav2_params/controller_server_params.yaml --params-file /home/wyca/.wyca/nav2_params/local_costmap_params.yaml --params-file /home/wyca/.wyca/nav2_params/costmaps_plugins_params.yaml --params-file /tmp/launch_params_est2liq2 -r /cmd_vel:=/twist_mux_root/twist_nav'].

This could be due to the specific controller plugins that I use, however I had it for AMCL too. Which makes me suspect that there is maybe something more general broken:

[amcl-59] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[amcl-59] what(): rcl_wait() failed: error not set
[ERROR] [amcl-59]: process has died [pid 1193703, exit code -6, cmd '/home/gd/elodie2_ws/install/nav2_amcl/lib/nav2_amcl/amcl --ros-args -r __node:=amcl --params-file /tmp/launch_params_8hezp2j8 --params-file /tmp/launch_params_q53mveik --params-file /tmp/launch_params_4slx9bvo --params-file /tmp/launch_params_58rq2q1l --params-file /tmp/launch_params_f1_xdpan --params-file /home/gd/elodie2_ws/install/elodie_config/share/elodie_config/default/nav2_params/amcl.yaml -r map:=map_amcl -r scan:=scan_filtered'].

[doisyg]

This issue seems to be confirmed by at least 2 other users.

[arslogavitabrevis]

We are also having this problem with some random nav2 nodes. Yesterday it was he planner and today we got the controller. The error is the same:
controller_server-9] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[controller_server-9] what(): rcl_wait() failed: error not set

We are using nav2 binary version 1.1.0-2.

[doisyg]

Thanks for confirming it ? Are you on Humble, Ubuntu 22.04 and Cyclone DDS too ?

[SteveMacenski]

The ticket in rclcpp / RMWs is probably the way to go for this, it doesn't seem like this is anything to do with Nav2 in particular, but an odd networking event error. We happen to just see it in servers with a bunch of traffic.

[doisyg]

Closing in favor of ros2/rclcpp#1977
@arslogavitabrevis, can we continue over there ?