Arbitrator is getting delayed due to failed calls

[MishkaMN]

Summary

During VRU validation testing, carma vehicle stopped generating trajectory right in the middle of the intersection. Upon investigation, the location of the vehicle was not the main issue and just happened to shut down there.

As you can see from below, 4.5.0 included new functionality in arbitrator to retry maneuver call back if failed as sometimes it can randomly fail due to ROS2 middleware issue. The retry however has timeout of 500ms and tries 10 times, which is 5 seconds. This results in arbitrator not being able to generate maneuver for entire 5 seconds despite all but 1 (which is optional) plugin successfully generated the high priority maneuver. Please observe the timestamps below:

guidance/plugins/approaching_emergency_vehicle_plugin/plan_maneuvers, retrying, attempt no: 0
1713294495.8355138 [carma_component_container_mt-25] 1713294495.835187321 | WARN | multiplex_service_call_for_capability:86 | Following client timed out: /guidance/plugins/approaching_emergency_vehicle_plugin/plan_maneuvers, retrying, attempt no: 1
1713294496.3357017 [carma_component_container_mt-25] 1713294496.335434161 | WARN | multiplex_service_call_for_capability:86 | Following client timed out: /guidance/plugins/approaching_emergency_vehicle_plugin/plan_maneuvers, retrying, attempt no: 2
1713294496.8359416 [carma_component_container_mt-25] 1713294496.835693847 | WARN | multiplex_service_call_for_capability:86 | Following client timed out: /guidance/plugins/approaching_emergency_vehicle_plugin/plan_maneuvers, retrying, attempt no: 3
1713294497.3361895 [carma_component_container_mt-25] 1713294497.335941837 | WARN | multiplex_service_call_for_capability:86 | Following client timed out: /guidance/plugins/approaching_emergency_vehicle_plugin/plan_maneuvers, retrying, attempt no: 4
1713294497.8365588 [carma_component_container_mt-25] 1713294497.836221242 | WARN | multiplex_service_call_for_capability:86 | Following client timed out: /guidance/plugins/approaching_emergency_vehicle_plugin/plan_maneuvers, retrying, attempt no: 5
1713294498.3368809 [carma_component_container_mt-25] 1713294498.336513135 | WARN | multiplex_service_call_for_capability:86 | Following client timed out: /guidance/plugins/approaching_emergency_vehicle_plugin/plan_maneuvers, retrying, attempt no: 6
1713294498.8371410 [carma_component_container_mt-25] 1713294498.836800445 | WARN | multiplex_service_call_for_capability:86 | Following client timed out: /guidance/plugins/approaching_emergency_vehicle_plugin/plan_maneuvers, retrying, attempt no: 7
1713294499.3374329 [carma_component_container_mt-25] 1713294499.337086187 | WARN | multiplex_service_call_for_capability:86 | Following client timed out: /guidance/plugins/approaching_emergency_vehicle_plugin/plan_maneuvers, retrying, attempt no: 8
1713294499.8377602 [carma_component_container_mt-25] 1713294499.837396271 | WARN | multiplex_service_call_for_capability:86 | Following client timed out: /guidance/plugins/approaching_emergency_vehicle_plugin/plan_maneuvers, retrying, attempt no: 9

Since the arbitrator was mostly using outdated maneuver, the tactical plugin was also kept using outdated trajectory. And around the time when vehicle ran out of valid trajectory, it drifted forward to get out of the road and shut down with error:

1713294494.9067583 [carma_component_container_mt-29] 1713294494.906498326 | ERROR | bumper_pose_cb:583 | Vehicle has left the route entirely. Unable to compute new shortest path.
1713294494.9073830 [carma_component_container_mt-29] 1713294494.907010253 | WARN | change_state:416 | Error occurred while doing error handling.
1713294494.9074733 [carma_component_container_mt-29] 1713294494.907036251 | ERROR | on_error:128 | Uncaught Exception from node: route_following_plugin exception: Vehicle not on route, unable to compute shortest path. while in ACTIVE state.
1713294494.9075103 [carma_component_container_mt-29] 1713294494.907085823 | ERROR | on_error:142 | Sent on_error system alert

One issue that contributes to this is the fact that the failed plugin which is not active, still published to plugin_discovery. This is logged here: #2384

Version

4.5.0 (Current)

Expected Behavior

Arbitrator should not get locked due to one failing plugin

Actual Behavior

See above.

Steps to Reproduce the Actual Behavior

1. Intentionally make one of the strategic plugin inactive
2. Run a simple scenario
3. Observe that arbitrator is publishing maneuvers late for 5s

Related Work

Related issue logged here: #2384
JIRA for this: https://usdot-carma.atlassian.net/browse/CAR-6039
No response

[MishkaMN]

This should be resolved now with only max 2 retry attempts.