Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AMCL core dumps due to a race condition when used with the map server. #4537

Closed
techtasie opened this issue Jul 15, 2024 · 9 comments · Fixed by #4605
Closed

AMCL core dumps due to a race condition when used with the map server. #4537

techtasie opened this issue Jul 15, 2024 · 9 comments · Fixed by #4605
Labels
bug Something isn't working

Comments

@techtasie
Copy link

Bug report

Required Info:

  • Operating System:
    • fedora 40
  • ROS2 Version: jazzy
    • jazzy
  • Version or commit hash: 1.3.1
    • from source: no changes from the source.
  • DDS implementation:
    • cyclonedds

Steps to reproduce issue

In the core dump, you see that map_ can be a null pointer and therefore you get an unhandled null pointer exception.
Uncommenting those lines circumvented the crash for us:

diff --color -ur navigation2-release-release-jazzy-nav2_amcl-1.3.1-1/src/amcl_node.cpp navigation2-release-release-jazzy-nav2_amcl-1.3.1-1-new/src/amcl_node.cpp
--- navigation2-release-release-jazzy-nav2_amcl-1.3.1-1/src/amcl_node.cpp	2024-06-25 20:01:09.000000000 +0200
+++ navigation2-release-release-jazzy-nav2_amcl-1.3.1-1-new/src/amcl_node.cpp	2024-07-15 17:43:12.792477289 +0200
@@ -540,13 +540,13 @@
       global_frame_id_.c_str());
     return;
   }
-  if (abs(msg->pose.pose.position.x) > map_->size_x ||
-    abs(msg->pose.pose.position.y) > map_->size_y)
-  {
-    RCLCPP_ERROR(
-      get_logger(), "Received initialpose from message is out of the size of map. Rejecting.");
-    return;
-  }
+  // if (abs(msg->pose.pose.position.x) > map_->size_x ||
+  //   abs(msg->pose.pose.position.y) > map_->size_y)
+  // {
+  //   RCLCPP_ERROR(
+  //     get_logger(), "Received initialpose from message is out of the size of map. Rejecting.");
+  //   return;
+  // }
 
   // Overriding last published pose to initial pose
   last_published_pose_ = *msg;

There is a race condition between the receiving of the map topic subscriber and setting the initial pose. On our robot, this race condition almost always happens but could be hard to reproduce on other setups.

Expected behavior

AMCL localization at initial pose or at least not crashing.

Actual behavior

AMCL coredumps

Additional information

I have attached our core dump. For the core dump, I rebuilt only AMCL in Debug mode; everything else was built as ReleaseWithDebug.
core.amcl.1000.c11f12581e0949ccb1dcd9e3d30b45d0.13740.1721053490000000.tar.gz
We are using a custom map server that may experience a slight delay when starting up. Therefore, AMCL could be launched without a map.

@SteveMacenski
Copy link
Member

That was recently added in #4416 by @GoesM during his fuzzing testing.

That makes sense, I think we should check if map_ != nullptr as part of that if statement before dereferencing it. @techtasie can you submit that change and make sure it solves your issue?

i.e.

if (map_ && (abs(msg->pose.pose.position.x) > map_->size_x || abs(msg->pose.pose.position.y) > map_->size_y))

@SteveMacenski SteveMacenski added the bug Something isn't working label Jul 15, 2024
@techtasie
Copy link
Author

@SteveMacenski Thank you for your fast reply I will test it thoroughly tomorrow during robocup.

@GoesM
Copy link
Contributor

GoesM commented Jul 16, 2024

Oh, so sorry for that! This is entirely caused by my mistake! I didn't notice that this map might be a null pointer.

@techtasie
Copy link
Author

Don't we need to still check for the nullptr like this?

  if ( (!first_map_received_) || (map_ && (abs(msg->pose.pose.position.x) > map_->size_x ||
    abs(msg->pose.pose.position.y) > map_->size_y))
  {
    RCLCPP_ERROR(
      get_logger(), "Received initialpose from message is out of the size of map. Rejecting.");
    return;
  }

@techtasie
Copy link
Author

I thing the way it is supposed to work is that the initial pose fires directly after the map has been received. Because in some circumstances you want to set the initial pose but get the map a second or so after amcl launch.

@GoesM
Copy link
Contributor

GoesM commented Jul 16, 2024

I thing the way it is supposed to work is that the initial pose fires directly after the map has been received. Because in some circumstances you want to set the initial pose but get the map a second or so after amcl launch.

I see. Thanks for your share √ 😀

@SteveMacenski
Copy link
Member

I think in that case we just need to check for map_ != nullptr, no? That would have been the behavior prior to the change (i.e. could set the initial pose before a map) so I think that's solid to continue to do

@GoesM
Copy link
Contributor

GoesM commented Jul 16, 2024

That would have been the behavior prior to the change

no problem, I misunderstood something before and I've understood it now.

@SteveMacenski
Copy link
Member

#4605 to resolve. thanks for the note @techtasie !

SteveMacenski added a commit that referenced this issue Aug 24, 2024
* Updated README Table once Jazzy jobs turn over (#4482)

* add new Jazzy matrix

* missing header

* test toolg

* retry

* done!

* trim

* trim

* fix OS[0]

* shutdown services in destructor of `ClearCostmapService` (#4495)

Signed-off-by: GoesM_server <GoesM@buaa.edu.cn>
Co-authored-by: GoesM_server <GoesM@buaa.edu.cn>

* adjusting backup speed to be more reasonable (#4501)

* Adding Costmap filters to system tests and cleaning up old gazebo classic files (#4502)

* removing old files

* removing old refs to gazebo classic

* porting test body

* including in root

* Dock panel (#4458)

* Initial docking panel

Signed-off-by: Alberto Tudela <ajtudela@gmail.com>

* Only one goal status

Signed-off-by: Alberto Tudela <ajtudela@gmail.com>

* Added dock  pose

Signed-off-by: Alberto Tudela <ajtudela@gmail.com>

* Fix size of text

Signed-off-by: Alberto Tudela <ajtudela@gmail.com>

* Update rviz

Signed-off-by: Alberto Tudela <ajtudela@gmail.com>

* Update rviz config

Signed-off-by: Alberto Tudela <ajtudela@gmail.com>

---------

Signed-off-by: Alberto Tudela <ajtudela@gmail.com>

* Fix default view (#4504)

* Fix logo in nav2 panel (#4505)

* Fix logo in nav2 panel

Signed-off-by: Alberto Tudela <ajtudela@gmail.com>

* Move icon

Signed-off-by: Alberto Tudela <ajtudela@gmail.com>

---------

Signed-off-by: Alberto Tudela <ajtudela@gmail.com>

* Fix world to map coordinate conversion (#4506)

Signed-off-by: HovorunBh <fipogh@gmail.com>

* Update README.md

Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>

* Add dock id (#4511)

* Implement dock id

Signed-off-by: redvinaa <redvinaa@gmail.com>

* Update tests

Signed-off-by: redvinaa <redvinaa@gmail.com>

* Update docs

Signed-off-by: redvinaa <redvinaa@gmail.com>

* Fix virtual override error

Signed-off-by: redvinaa <redvinaa@gmail.com>

---------

Signed-off-by: redvinaa <redvinaa@gmail.com>

* fix(nav2_costmap_2d): make obstacle layer not current on enabled toggle (#4507)

Signed-off-by: Kemal Bektas <kemal.bektas@node-robotics.com>
Co-authored-by: Kemal Bektas <kemal.bektas@node-robotics.com>

* min_turning_r_ getting param fix (#4510)

* min_turning_r_ getting param fix

Signed-off-by: Ivan Radionov <i.a.radionov@gmail.com.com>

* Update nav2_mppi_controller/include/nav2_mppi_controller/motion_models.hpp

Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>
Signed-off-by: Ivan Radionov <i.a.radionov@gmail.com.com>

---------

Signed-off-by: Ivan Radionov <i.a.radionov@gmail.com.com>
Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>
Co-authored-by: Ivan Radionov <i.a.radionov@gmail.com.com>
Co-authored-by: Steve Macenski <stevenmacenski@gmail.com>

* fixing gz sim launch file by using gz directly (#4514)

* port wait behavior to new gazebo (#4471)

Signed-off-by: stevedan <stevedan.o.omodolor@gmail.com>
Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>
Co-authored-by: Steve Macenski <stevenmacenski@gmail.com>

* Completely rewritten spin, backup, and drive on heading tests to remove flakiness (#4515)

* port backup behavior to new gazebo

Signed-off-by: stevedan <stevedan.o.omodolor@gmail.com>

* port drive on heading behavior to new gazebo

Signed-off-by: stevedan <stevedan.o.omodolor@gmail.com>

* completely rewritten spin test

* lint

* complete flaky test rewrite

---------

Signed-off-by: stevedan <stevedan.o.omodolor@gmail.com>
Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>
Co-authored-by: stevedan <stevedan.o.omodolor@gmail.com>

* Return out of map update if frames mismatch. Signed-off-by Joey Yang (#4517)

Signed-off-by: Joey Yang <joeyyang.ai@gmail.com>

* Fix error_code_id (#4522)

Signed-off-by: redvinaa <redvinaa@gmail.com>

* completely shutdown dyn_params_handler_ (s)  (#4521)

* completely shutdown dny_params_handler_ in nav2_amcl

Signed-off-by: GoesM_server <GoesM@buaa.edu.cn>

* completely shutdown dyn_param_handler_ in controller_server

Signed-off-by: goes <GoesM@buaa.edu.cn>

* compeletly shutdown dyn_params_handler in nav2_costmap_2d

Signed-off-by: goes <GoesM@buaa.edu.cn>

* compeletly shutdown dyn_param_handler in nav2_docking

Signed-off-by: goes <GoesM@buaa.edu.cn>

* completely shutdown dyn_params_handler in nav2_velocity_smoother

Signed-off-by: goes <GoesM@buaa.edu.cn>

* compeletly shutdown dyn_param_handler in waypoint_follower

Signed-off-by: goes <GoesM@buaa.edu.cn>

* typo fixed

Signed-off-by: goes <GoesM@buaa.edu.cn>

* graceful-controller & dwb_controller

Signed-off-by: goes <GoesM@buaa.edu.cn>

* mppi-controller

Signed-off-by: goes <GoesM@buaa.edu.cn>

* navfn_planner & regulated_..controller

Signed-off-by: goes <GoesM@buaa.edu.cn>

* rotation_..controller & samc_planners

Signed-off-by: goes <GoesM@buaa.edu.cn>

* A*planner

Signed-off-by: goes <GoesM@buaa.edu.cn>

* code style

Signed-off-by: goes <GoesM@buaa.edu.cn>

* 1

Signed-off-by: goes <GoesM@buaa.edu.cn>

* fixed

Signed-off-by: goes <GoesM@buaa.edu.cn>

* fix the usage of weak_ptr

Signed-off-by: goes <GoesM@buaa.edu.cn>

* code-style

Signed-off-by: goes <GoesM@buaa.edu.cn>

* weak_ptr released

Signed-off-by: goes <GoesM@buaa.edu.cn>

* code style

Signed-off-by: goes <GoesM@buaa.edu.cn>

* code style

Signed-off-by: goes <GoesM@buaa.edu.cn>

* code style

Signed-off-by: goes <GoesM@buaa.edu.cn>

* code style update

Signed-off-by: goes <GoesM@buaa.edu.cn>

* back

Signed-off-by: goes <GoesM@buaa.edu.cn>

* rebase conflict resovled

Signed-off-by: goes <GoesM@buaa.edu.cn>

* rebase error fixed

Signed-off-by: goes <GoesM@buaa.edu.cn>

* fixed2

Signed-off-by: goes <GoesM@buaa.edu.cn>

* rebase fixed 3

Signed-off-by: goes <GoesM@buaa.edu.cn>

* 33

Signed-off-by: goes <GoesM@buaa.edu.cn>

* shared_ptr into weak_ptr

Signed-off-by: GoesM <goesm@buaa.edu.cn>

* remove adundant node.resest()

Signed-off-by: GoesM <goesm@buaa.edu.cn>

---------

Signed-off-by: GoesM_server <GoesM@buaa.edu.cn>
Signed-off-by: goes <GoesM@buaa.edu.cn>
Signed-off-by: GoesM <goesm@buaa.edu.cn>
Co-authored-by: GoesM_server <GoesM@buaa.edu.cn>

* check nullptr in smoothPlan() (#4544)

* check nullptr in smoothPlan()

Signed-off-by: GoesM <goesm@buaa.edu.cn>

* code-style

Signed-off-by: GoesM <goesm@buaa.edu.cn>

* code-style

Signed-off-by: GoesM <goesm@buaa.edu.cn>

* simple change

Signed-off-by: GoesM <goesm@buaa.edu.cn>

---------

Signed-off-by: GoesM <goesm@buaa.edu.cn>
Co-authored-by: GoesM <goesm@buaa.edu.cn>

* check nullPtr in `computeControl()` (#4548)

Signed-off-by: goes <GoesM@buaa.edu.cn>
Co-authored-by: goes <GoesM@buaa.edu.cn>

* Straight analytic expansions (#4549)

* Add test to verify analytic expansions are straight

Signed-off-by: James Ward <j.ward@sydney.edu.au>

* Use continuous scaling of angle when performing analytic expansion

Only applies to Hybrid A* - behaviour in lattice planner is unchanged

Signed-off-by: James Ward <j.ward@sydney.edu.au>

---------

Signed-off-by: James Ward <j.ward@sydney.edu.au>

* Rviz tool to get cost of costmap cell (#4546)

* Added GetCost srv

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Added Service in costmap_2d

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Added Rviz tool

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Fixed Styling

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Fixed Styles and Linting

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Fixed Linting

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Added Bool use_footprint to srv

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Added unit test for costmap costcell cost service

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Fixed unit test script

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Added theta, Updated unit test, Updated rviz tool service call logic

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Updated requested changes

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

---------

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Switch to new-style static_transform_publisher arguments. (#4563)

These arguments have been the preferred way to use things
since at least Humble.  This avoids warnings when running it for the tests.

Signed-off-by: Chris Lalancette <clalancette@gmail.com>

* Updated slack link (#4565)

Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>

* Update README.md (#4589)

Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>

* fix flickering visualization (#4561)

* Fix Flickering visualization

Signed-off-by: Vladyslav Hrynchak <vladyslav.hrynchak@logivations.com>

* Refactoring Costmap2DPublisher and Costmap2DROS

Signed-off-by: Vladyslav Hrynchak <vladyslav.hrynchak@logivations.com>

* Refactoring costmap_2d_ros.cpp

Signed-off-by: Vladyslav Hrynchak <vladyslav.hrynchak@logivations.com>

* Refactoring Costmap2DPublisher and Costmap2DROS

Signed-off-by: Vladyslav Hrynchak <vladyslav.hrynchak@logivations.com>

* Update costmap_2d_publisher.cpp

Signed-off-by: Vladyslav Hrynchak <vladyslav.hrynchak@logivations.com>

* Change map_vis_z from float to double

Signed-off-by: Vladyslav Hrynchak <vladyslav.hrynchak@logivations.com>

* Add comment to  map_vis_z_ parameter

Signed-off-by: Vladyslav Hrynchak <vladyslav.hrynchak@logivations.com>

---------

Signed-off-by: Vladyslav Hrynchak <vladyslav.hrynchak@logivations.com>

* Copy fix-terminate diff from opennav_docking repo (#4598)

* Copy fix-terminate diff from opennav_docking repo

Signed-off-by: redvinaa <redvinaa@gmail.com>

* Lint

Signed-off-by: redvinaa <redvinaa@gmail.com>

---------

Signed-off-by: redvinaa <redvinaa@gmail.com>

* Fix race condition in AMCL for #4537 (#4605)

* Fixed timed_behavior.hpp (#4602)

Signed-off-by: Vladyslav Hrynchak <vladyslav.hrynchak@logivations.com>

* Adding new Nav2 loopback simulator (#4614)

* adding Nav2 loopback sim

Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>

* drop performance by half

* lintin

* Add multirobot usecase comment

* fixing copy paste error

* fixing review comments

---------

Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>

* Added laser data from map in nav2_loopback_sim (#4617)

* Added laser data from map

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Fixed Linting

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Fixed Linting

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Fixed few requested changes

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Fixed linting

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Requested changes

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Linting

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Added parameters and fixed requested changes

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* linting

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Added scan  using LineIterator

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* LineIterator max_distance or range_max

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* min of max_distance or range_max

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* final updates working correctly

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Fix lint

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* requested changes

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>

* Update nav2_loopback_sim/nav2_loopback_sim/loopback_simulator.py

Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>

* Update nav2_loopback_sim/nav2_loopback_sim/loopback_simulator.py

Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>

---------

Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>
Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>
Co-authored-by: Steve Macenski <stevenmacenski@gmail.com>

* Making base frame ID for map to base link transform based on base frame ID parameter (#4632)

Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>

* Update Smac Planner for rounding to closest bin rather than flooring (#4636)

* adding stamped option for loopbacks im (#4637)

Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>

---------

Signed-off-by: GoesM_server <GoesM@buaa.edu.cn>
Signed-off-by: Alberto Tudela <ajtudela@gmail.com>
Signed-off-by: HovorunBh <fipogh@gmail.com>
Signed-off-by: Steve Macenski <stevenmacenski@gmail.com>
Signed-off-by: redvinaa <redvinaa@gmail.com>
Signed-off-by: Kemal Bektas <kemal.bektas@node-robotics.com>
Signed-off-by: Ivan Radionov <i.a.radionov@gmail.com.com>
Signed-off-by: stevedan <stevedan.o.omodolor@gmail.com>
Signed-off-by: Joey Yang <joeyyang.ai@gmail.com>
Signed-off-by: goes <GoesM@buaa.edu.cn>
Signed-off-by: GoesM <goesm@buaa.edu.cn>
Signed-off-by: James Ward <j.ward@sydney.edu.au>
Signed-off-by: Jatin Patil <jatinpatil2003@gmail.com>
Signed-off-by: Chris Lalancette <clalancette@gmail.com>
Signed-off-by: Vladyslav Hrynchak <vladyslav.hrynchak@logivations.com>
Co-authored-by: GoesM <130988564+GoesM@users.noreply.github.com>
Co-authored-by: GoesM_server <GoesM@buaa.edu.cn>
Co-authored-by: Alberto Tudela <ajtudela@gmail.com>
Co-authored-by: Bohdan <72872431+HovorunBh@users.noreply.github.com>
Co-authored-by: Vince Reda <60265874+redvinaa@users.noreply.github.com>
Co-authored-by: Kemal Bektas <34746077+bektaskemal@users.noreply.github.com>
Co-authored-by: Kemal Bektas <kemal.bektas@node-robotics.com>
Co-authored-by: Ivan Radionov <45877502+JJRedmond@users.noreply.github.com>
Co-authored-by: Ivan Radionov <i.a.radionov@gmail.com.com>
Co-authored-by: Stevedan Ogochukwu Omodolor <61468301+stevedanomodolor@users.noreply.github.com>
Co-authored-by: stevedan <stevedan.o.omodolor@gmail.com>
Co-authored-by: Joey Yang <joeyyang.ai@gmail.com>
Co-authored-by: James Ward <j.ward@sydney.edu.au>
Co-authored-by: Jatin Patil <89979346+JatinPatil2003@users.noreply.github.com>
Co-authored-by: Chris Lalancette <clalancette@gmail.com>
Co-authored-by: Vladyslav Hrynchak <vladyslav.hrynchak@logivations.com>
josephduchesne pushed a commit to josephduchesne/navigation2 that referenced this issue Dec 10, 2024
Signed-off-by: Joseph Duchesne <josephgeek@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants