wait_for_service not being woken by graph events

[dhood]

While investigating the appropriate timeout for ros2/system_tests#259, I noticed a correlation between the timeout used in wait_for_service calls (20s) and the time taken for tests to run successfully.

The tests that have two wait_for_service calls with timeouts of 20s each take one of 6, 26, or 46 seconds to run. Change the wait_for_service call to each be 30s each and the tests take one of 6, 36, or 66 seconds to run. Change the wait_for_service call to be multiple 1s wait_for_service calls and the tests never take longer than 9s.

Note that the tests still pass, they just spend an unnecessary amount of time in the wait_for_service calls, presumably because the waitset is not triggered by any graph event of the service coming up.

Given that wait_for_service passes in the end, my money is on the graph event triggering before we wait on the waitset. Therefore we are waiting for something that has already occurred.

We have come across this in rmw_fastrtps_cpp before: what we need is an equivalent to ros2/rmw_fastrtps#147, which prevents guard conditions from being triggered between the time we check them to decide if we should wait, and the time we actually wait.

This seems related to #201 but distinct in that this is a race condition in services showing up as opposed to #201 being a race condition in services going away.

[dhood]

As a correction, while #201 is primarily about services not being reported correctly as not available, it also references the issue of services not being reported as available, which prompted this workaround: ros2/rclcpp#262

I've modified that workaround in ros2/rclcpp#476 which fixes this specific issue, but not the underlying cause