Triggering Autonomous Mode for custom planner in microservice architecture

[TobiasMascetta]

We are currently using autoware and autoware.universe as a middleware for an autonomous driving project. We have split the different modules (system, planning, ...) into different microservices and are now in the process of integrating our own algorithms.

Problem Overview

Unfortunately, we seem to be stuck at getting our custom planner to run. Similar to issue # #3783 , we are stuck at the point that the auto mode cannot be triggered (greyed out button in rviz). However, for previous versions of autoware (under ROS2 Galactic), our code worked. We currently suspect the component_state_monitor of our planning module to be the problem, however we cannot rule out problems with our microservice architecture completely.
Our current state can be seen here:

The operation_mode can be seen here:

The autoware state can be seen here. It works as expected up to state 3, where it gets stuck.

The routing state (topic /api/routing/state) is 2.

The motion state (topic /api/motion/state) is 1 and the service /api/motion/accept_start response that it is not the "starting" state.

What we have tried so far

1. Here is the output for the recommended guide.py (note the control sometimes is not listed here, but planning always):

We interprete the guide.py output that there is a rate error in our planning, but we could not find out how the rate error is exactly defined. Please let us now, if you have any suggestions here.

2. As described in issue #3783 , we checked the two planning topics listed here. Both the topic /planning/mission_planning/route and /planning/scenario_planning/trajectory are published once.

3. The topic /system/component_state_monitor/component/autonomous/* only is false for the planning module.

4. Trying to change states for either the component_state_machine or the autoware state manually via ros2 topic pub and a hand-crafted message yields the response that "The mode change is blocked by the system"

5. Using the service /api/operation_mode/change_to_autonomous also yields the response "The mode change is blocked by the system", but the change to all other modes (e.g. stop mode) works fine.

6. We also tried changing the operation mode as described here, but were unsuccessful

Questions and Need for Advice / Additional Documentation

As you can probably tell by the lack of strategy in our trouble-shooting, we are kind of lost when it comes to understanding the details of the different states, modes etc. and, especially, what triggers what when with which topic to activate the autonomous mode.

If you have additional documentation about the state change, please let us know.

Thank you

[mitsudome-r]

In order to switch to autonomous mode, is_autonomous_mode_available in /api/operation_mode/state must become true. (published from here)

The field becomes true if all the dependent module states(planning state, localization state, perception state, etc) are true. code

Each module states are published by component state monitor which looks if all the expected messages are coming out from each module, which are defined by the config here.

Therefore, if you are replacing planning module with your own custom implementation, I would first look into the topics for planning and modify them for your system. (by default it is looking at /planning/mission_planning/route and /planning/scenario_planning/trajectory.

[mitsudome-r]

@isamu-takagi Please add a comment if I'm missing or wrong in my explanation.

[isamu-takagi]

The file actually used have been copied to autoware_launch repository, so please edit the following file.
https://github.com/autowarefoundation/autoware_launch/blob/main/autoware_launch/config/system/component_state_monitor/topics.yaml

[TobiasMascetta]

Thank you very much for your help!
We actually came a lot closer to the goal and are now able to trigger autonomous mode.

So for other interested users, here is some documentation:

Update

Old Problem Partially Solved

• the autonomous mode can be triggered now. It actually was a rate error combined with a shadowed config file for the component_state_monitor config.yaml

--> Solution:

• 1. Listen to the /diagnostics topic (ros2 topic echo) and find out, what topic produces an error. In planning, it is probably topic_state_monitor_scenario_planning_trajectory (aka /planning/scenario_planning/trajectory)

2. From step one, use the message to find out, which rate is failing (there is an expected and an actual value in the data of the message)
3. Find the correct config.yaml for the component_state_monitor, see here and adjust the rates for accordingly, until the diagnostic messages have the type OK. Also, if you are using a docker compose architecture, make sure you are modifying the correct file in the correct docker
4. Modify the frequency of your planner and the topics accordingly. The trajectory msg needs to be published periodically.

Problem Description - What we have tried so far

• After triggering the autonomous mode, the system_error_monitor throws a STALE error (for each field) for the trajectory resulting in an emergency maneuver

• There seems to be a STALE error for /autoware/planning/perfoamce_monitoring/trajectory_validation

• Disabling the trajectory topic by modifying the values in the config.yaml for the system_error_monitor results in the car not going into an emergency maneuver, but just standing still

• However, the trajectory being published and control topics seem to work as desired.

• The velocity and position profile seem to be correct and the trajectory for the first step is visualized correctly

Below are some pictures of the current state:

Questions

We still seem to have some problem with our trajectory msg. We tried to dive deeper into the system_error_monitor but could not figure out exactly what the error condition is, that it reacted to. Our assumption currently is that the system_error_monitor indeed raises a valid error which (after we disable the system_error_monitor partially) results into the car just not moving.

If you have further suggestions, please let us know.

Thank you for your help and best regards.