Various tests hanging sometimes on linux with fastrtps

[dhood]

This times out in different points in the test, e.g.
https://ci.ros2.org/view/nightly/job/nightly_linux_repeated/963/testReport/junit/(root)/projectroot/test_find_weak_nodes/

-- run_test.py: invoking following command in '/home/rosbuild/ci_scripts/ws/src/ros2/rclcpp/rclcpp':
 - /home/rosbuild/ci_scripts/ws/build/rclcpp/test_find_weak_nodes --gtest_output=xml:/home/rosbuild/ci_scripts/ws/build/rclcpp/test_results/rclcpp/test_find_weak_nodes.gtest.xml
Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from TestFindWeakNodes
[ RUN      ] TestFindWeakNodes.allocator_strategy_with_weak_nodes

or
https://ci.ros2.org/view/nightly/job/nightly_linux_repeated/939/testReport/junit/(root)/projectroot/test_find_weak_nodes/

-- run_test.py: invoking following command in '/home/rosbuild/ci_scripts/ws/src/ros2/rclcpp/rclcpp':
 - /home/rosbuild/ci_scripts/ws/build/rclcpp/test_find_weak_nodes --gtest_output=xml:/home/rosbuild/ci_scripts/ws/build/rclcpp/test_results/rclcpp/test_find_weak_nodes.gtest.xml
Running main() from gtest_main.cc
[==========] Running 2 tests from 1 test case.
[----------] Global test environment set-up.
[----------] 2 tests from TestFindWeakNodes
[ RUN      ] TestFindWeakNodes.allocator_strategy_with_weak_nodes
[       OK ] TestFindWeakNodes.allocator_strategy_with_weak_nodes (14 ms)
[ RUN      ] TestFindWeakNodes.allocator_strategy_no_weak_nodes

Those tests are very simple and there isn't anything they should be waiting on, making me think it's a deadlock. A while ago this was flaky on osx with verbose mutex locking issues, which might still be the underlying problem. https://ci.ros2.org/view/nightly/job/nightly_osx_repeated/768/testReport/junit/(root)/projectroot/test_find_weak_nodes/

This test is only being run for fastrtps. It would be good to know if it's also an issue for connext.

[dhood]

With only connext it passed 200 times: https://ci.ros2.org/job/ci_linux/4071/ and 417 times here: https://ci.ros2.org/job/ci_linux/4085/ (failed the 417th retry because of #93). Still a chance it's flaky with a very low probability, but unlikely.

so it looks like this issue is unique to fastrtps.

[dhood]

More apparent deadlocks happened last night on ARM:
https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/369/testReport/junit/(root)/projectroot/gtest_spin__rmw_fastrtps_cpp/
https://ci.ros2.org/view/nightly/job/nightly_linux-aarch64_repeated/369/testReport/junit/(root)/projectroot/test_time_source/

I was suspicious that these might have been caused by something since today is the first time I've seen them, but even with the repos from yesterday both of those tests still failed when repeated:
https://ci.ros2.org/job/ci_linux-aarch64/1228/testReport/(root)/projectroot/test_time_source/
https://ci.ros2.org/job/ci_linux-aarch64/1227/testReport/junit/(root)/projectroot/gtest_spin__rmw_fastrtps_cpp/

[dhood]

I update the title because I haven't ever confirmed a deadlock was the reason the tests were/are hanging.

[clalancette]

We haven't seen this in a while on nightly_linux_repeated (at least a week), so I'm going to close this out for now. We can reopen if it reappears.