rclcpp::Node constructor and destructor crash with multithreading

[rotu]

Use of rclcpp::Node is unsafe in an asynchronous setting and can cause segfaults and undefined behavior if such objects are constructed and/or destructed simultaneously.

See ros2/rosbag2#329 where this directly caused a reproducible crash. While the particular crash was averted in ros2/rosbag2#338, the constructor/destructor still use the unsafe functions rcl_node_init/rcl_node_fini.

Either these functions should be made threadsafe or they should be documented as unsafe and audited/instrumented to detect such unsafe usage.

[rotu]

@thomas-moulard re: https://github.com/ros-tooling/aws-oncall/issues/107 (can't comment on my own ticket in aws-oncall repo since I'm not a collaborator)

[fujitatomoya]

@rotu

does anybody work on the fix?

[rotu]

As far as I can tell, no. @dirk-thomas is listed as the package maintainer so maybe he knows.

[ivanpauno]

@rotu how it was concluded that the thread safety issue is in Node constructor/destructor?
From the information available in ros2/rosbag2#329, it's not obvious to me.
Can you clarify how that conclusion was achieved?

I will try to track down the issue, but it would be great to have clearer information (and preferably, a simple example with reproduction steps).

[dirk-thomas]

@dirk-thomas is listed as the package maintainer

While I am listed in the manifest this repo is maintainer by the whole @ros2/team.

[ivanpauno]

I think I understand the issue:

• rcl_node_init calls rcl_logging_rosout_init_publisher_for_node
• rcl_logging_rosout_init_publisher_for_node modifies some global state.

We really should get rid of those global variables in rcl ...

I don't see a way of completely fixing the issue, as the rosout output handler does access the same map (https://github.com/ros2/rcl/blob/94b5a1d7d0899aa84a1026e21488bee95e67bbd8/rcl/src/rcl/logging_rosout.c#L241), and adding a lock to mutually exclude all node constructor calls, node destructor calls and all logs seems like a pretty bad idea.

For the time being, I think that adding a global mutex protecting node construction/destruction will greatly reduce the issue (there might be a multithread crash between one thread constructing/destructing a node and another one logging something).

@wjwwood @hidmic any ideas?

[ivanpauno]

For the time being, I think that adding a global mutex protecting node construction/destruction will greatly reduce the issue (there might be a multithread crash between one thread constructing/destructing a node and another one logging something).

Thinking about this again, I think we should actually add mutexes in rcl itself, protecting its global variables (e.g. https://github.com/ros2/rcl/blob/85ef95514d6ee47bb8285009add84e956595c018/rcl/src/rcl/logging_rosout.c#L83 and https://github.com/ros2/rcl/blob/85ef95514d6ee47bb8285009add84e956595c018/rcl/src/rcl/logging.c#L37).

I know there was some resistance about adding locks in rcl, but until we get rid of all the global state I think it's the best solution (loggers seem to be the only place where we have global state).

[rotu]

I think that locks in rcl are an excellent idea. If you instrument rcl for it, you could even add an option to compile with or without locks, or to verbosely log lock contention to report unsafe usage of the API.

[rotu]

@dirk-thomas is listed as the package maintainer

While I am listed in the manifest this repo is maintainer by the whole @ros2/team.

Please update the package to reflect reality

https://www.ros.org/reps/rep-0149.html#maintainer-multiple-but-at-least-one

[wjwwood]

👎 to adding locks to rcl. Why not add the locking in rclcpp?

We really should get rid of those global variables in rcl ...

This is the right solution that I have pointed out several times during the development of the logging API's. It is also needed to address long standing issues with static destruction order... Any other solutions are band-aids in my opinion and this particular proposed one happens to also be a step backwards in design as well, in my opinion.

[wjwwood]

and adding a lock to mutually exclude all node constructor calls, node destructor calls and all logs seems like a pretty bad idea.

Why?