Docker Container for Autoware distribution

[dylan-gonzalez]

It is insufficient to just have AutoWare run on a very particular setup, there should be a way to generalise the AutoWare installation on other devices easily.

Experiment and use the provided docker installation. Identify pros and cons of this method as compared to source to determine suitability for distribution.

[zcdai]

[Imported logs from few days ago]
Tried docker things, launching with GPU support didn't work until NVIDIA/nvidia-docker#1243 fixed it

General things about docker:
Still getting ros sourcing issues when trying to run.
Building seems to go well, sometimes ansible doesn't work on certain containers but this is inconsistent?
Everything performed on a non-fresh Ubuntu install, looks difficult to adapt docker image to everyone's personal devices.

Following the tutorial, the docker repo with NVIDIA support does not seem to work, use the barebones ROS one instead.

[maarkks]

Followed the development installation here
https://autowarefoundation.github.io/autoware-documentation/main/installation/autoware/docker-installation/

Started the installation process using ansible. Discovered errors regarding CUDA dependencies. NVIDIA properties were also mentioned, but proceeded. Workspace setup should be straightforward (?) after this step.

Tried to launch the current installation of docker separately (using the general user), encountered an error stating that KVM is not enabled on host. This relates to the enabling of virtualisation support - needs to be turned on in the system bios.
Retried this on the fresh AWSIM user profile, with the same problem. Redownloaded the docker packages and application, but still have this same problem.

[npnquang]

Docker Engine still works somehow even the beast does not support virtualization. Keep in mind that we need Docker Engine for the setup. Right now we are on the downloading step so it still might fail when we run the Docker Image.

[npnquang]

These docker images are essential for running docker container of autoware.
Right now the container is launchable, but we need to see if the container works correctly as expected.
Link to the installation

Important

DO NOT delete those Docker image, especially the one from the autowarefoundation repository

• CUDA installation is 11.6
• cudnn and tensorRT installation is still unsure

[npnquang]

• When building the autoware,
  colcon build --symlink-install --cmake-args -DCMAKE_BUILD_TYPE=Release
  CUDA is not found by autoware while it is installed.
  

• The CUDA version returned from nvcc --version and nvidia-smi are different.
  

[npnquang]

We have Autoware working now, but the UI looks nothing like this tutorial

Important

• Remember to source the setup file and then run this command to launch it.
  ros2 launch autoware_launch e2e_simulator.launch.xml vehicle_model:=sample_vehicle sensor_model:=awsim_sensor_kit map_path:=<your mapfile location>
• Right now the docker container is mounted with the local autoware directory, so we do not have to build every time we launch the container.
• The same thing happens for the autoware_map directory.

[dylan-gonzalez]

Following on from @npnquang
Looking at the tutorial he linked to run Autoware with AWSIM.

Run the executable for AWSIM at ~/AWSIM/AWSIM_demo.x86_64.
Made sure to switch to the awsim-stable branch as they state.

Run Docker container
rocker --nvidia --x11 --user --volume $HOME/autoware --volume $HOME/autoware_map --volume $HOME/autoware_data-- ghcr.io/autowarefoundation/autoware-universe:latest-cuda

Run Autoware (make sure to specify absolute path for mapfile location):
source install/setup.bash
ros2 launch autoware_launch e2e_simulator.launch.xml vehicle_model:=sample_vehicle sensor_model:=awsim_sensor_kit map_path:=/home/mcav/autoware_map/pointcloud_map.pcd

Can't seem to load the map properly.
Trying a mapfile of /home/mcav/autoware_map/ (might require the map directory which has a .pcd and .osm file, https://autowarefoundation.github.io/autoware-documentation/main/support/troubleshooting/)
--> This worked!!

Following these steps now to test a simulation scenario.

Simulation scenarios work great (bit slow, maybe had a few issues with traffic light recognition it seemed?)

[dylan-gonzalez]

03/11/23 with @npnquang

We tried to run Autoware with AWSIM again and it we were getting errors with some of the modules.
Autoware RVIZ screen UI would show with the map, but camera feed was displaying 'No Image', with AUTO operation mode greyed out.

We will need to take a look at this discussion, which seems to be very similar to our problem, and try the recommended fixes.

[npnquang]

When trying to fix the persistent issues of Autoware, I tried listen to the /api/operation_mode/state topic but this error is returned:
The message type 'autoware_adapi_v1_msgs/msg/OperationModeState' is invalid

I tried to reinstall autoware again but this error is raised:

[dylan-gonzalez]

For the 'invalid msg type' error, you need to have had sourced the workspace in that terminal before listening to the topics. Otherwise ROS won't have any knowledge of the custom msg type you're trying to listen to

[npnquang]

The issue is raised when the autoware is running so I don't think it was because I didn't source

[dylan-gonzalez]

But if you open a new terminal to do a ros2 topic echo then you will have to source it again in that new terminal window if the topics have custom msg typea

[npnquang]

After doing the installation once again for the relatively nonsense reason above, I realized that for the docker installation, we do not need anything even cuda. Let the ./setup-dev-env.sh docker do the thing for us. This is the best way to avoid any version conflicts.

Moreover, I also realized that we do not need cudnn and tensorRT when following the docker installation, while we do when following the source installation.

When checking the version of CUDA, we only need to check nvcc --version. The command nvidia-smi is not the correct one to check.

[npnquang]

When running Autoware, I listened to /api/operation_mode/state topic and it seems that nothing is sent to this topic. The screen remains the same for quite a while.

When listening to /diagnostics topic, this message is sent over and over again

[npnquang]

It seems that the installation guide did not mention anything regarding the autoware_data folder but I realized that we should mount that folder to the Docker container as well since some of the errors raised when loading up Autoware relates to that folder.

After mounting the autoware_data folder to the Docker container, the above errors are resolved.
So the correct command to run the container is
rocker --nvidia --x11 --user --volume $HOME/autoware --volume $HOME/autoware_map --volume $HOME/autoware_data -- ghcr.io/autowarefoundation/autoware-universe:latest-cuda

UPDATE: We can confirm that autoware_data is the folder that supports the operation of Autoware now and it is generated when the autoware is built.

[npnquang]

We have the steering and velocity received by Autoware now but the problems with the Recognition Result is still unsolved. In addition, the /api/operation_mode/state topic is still quiet when listening to it.

[dylan-gonzalez]

how did you get steering and velocity to be received by Autoware?

[npnquang]

I just rebuilt the autoware and the velocity and steering angle is not "not received" anymore.

[dylan-gonzalez]

ok so seems like checking out to the awsim-stable branch fixed that
I think it might also be a good idea to pull the docker image again as I remember reading somewhere that they made some changes (I could be wrong though)

[npnquang]

I have pulled it again multiple times and the image remains unchanged.

[npnquang]

I have put a post to ask about our issues now on the GitHub page of Autoware.

[dylan-gonzalez]

i've bumped your issue to the discord so that we get responses quicker
https://discordapp.com/channels/953808765935816715/1172387460844625963

[dylan-gonzalez]

Debugging - here's what I've done

AWSIM

1. Run AWSIM
2. ros2 topic list to see what topics are running

3. I see a few of them use message definitions from Autoware. If I try to echo these topics I will get the following error: The message type is invalid. To fix this, you need to source the autoware workspace: cd ~/autoware && source install/setup.bash.
4. the AWSIM topics are being published fine. So I can conclude the issue is not with AWSIM.

Autoware Universe

1. Running the docker container and running autoware_launch gives the same results as above ^ as expected. Looking at the terminal output, shows this error that occurs routinely (i.e keeps trying to initialise but fails):

2. searching up default_ad_api. Brought me to this issue. Then realised that you can run a planning simulation without a running simulator, and just using RVIZ. So will try this.

Planning Simulation

• realised autoware_data is empty --> followed these steps
• running autoware in this case without the docker container works fine. Followed the steps to do the 2d pose estimation, 2d goal pose, etc. No camera view - same as in the tutorial, guessing there isn't a camera on the ego vehicle?
• running autoware in the docker container with autoware_data mounted works
• running autoware in the docker container with autoware_data not mounted seems to work? Had a few issues initialising a 2d goal pose, with the goal pose being invalid. But then seemed to work with other poses that I set. I'm guessing running autoware without autoware_data will have reduced functionality since autoware_data contains a bunch of pre-trained models I think.

Trying Autoware + AWSIM again

• running autoware outside of docker: ran it, saw it had the same issue, procrastinated on my phone for like 5 mins, look back up and now its working. Re-tried again: pose initialisation takes like 10 seconds, then it's all good! I'm not sure why it works outside of the docker container (otherwise what is the poitn of the docker container...)
• pretty sure the main issue is just that the host network is not being correctly bridged to the docker container. When you do ros2 topic list inside the docker container, it is empty. When I add --net host to the rocker command, it seems to bridge the network (as ros2 topic list is no longer empty) but I don't think it's entering the container properly as the PS1 prompt stays as mcav@the-beast.

I'm a bit confused as to how Autoware is running outside of the docker container. @npnquang did you do a source installation or docker installation development?

Changes I've made:

• included a ~/maps directory to hold all maps. ~/maps/autoware_map is to be used for the Planning Simulation, and ~/maps/awsim_map is to be used when running the AWSIM simulator.

[npnquang]

We have a docker installation development.

[dylan-gonzalez]

We have concluded that Autoware Universe is working inside and outside the docker container.

The reason why it doesn't work properly with AWSIM when Autoware is run inside the docker container is due to networking issues.

[sxppro]

@dylan-gonzalez @npnquang can we give the Autoware maintainers an update in this issue: https://github.com/orgs/autowarefoundation/discussions/3970

[sxppro]

(unless it was resolved through some other channel oops)