Run carla on Nvidia Tesla GPU

[MahdiCat]

We want to install carla on a server with NVIDIA Tesla P100. Has anyone ever succeeded? Do we need to install Grid Driver or other driver? And what are the legal driver versions?
It would be great if there was a tutorial for installing carla on P100.
Thank you very much.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[qhaas]

Our colleagues have had success running CARLA 0.9.5 on a Tesla V100, the successor to the Tesla P100, using the CARLA Docker image provided by the CARLA team running via Singularity. AFAIK, no GRID driver is needed in Linux when not running inside a VM.

As a sanity check, this is how you would run it via Singularity, nvidia-docker will likely work as well (especially since the images were written specifically for Docker)

# 'import' the docker image into singularity
singularity pull docker://carlasim/carla:0.9.5

# Create scratch directories on the host (Singularity will assign ownership of /home/carla to root)
install -d CarlaUE4/Saved Engine/Saved

# Launch the command from above with select folders mounted into the scratch directories
singularity exec --nv -B $(pwd)/CarlaUE4/Saved:/home/carla/CarlaUE4/Saved,$(pwd)/Engine/Saved:/home/carla/Engine/Saved carla_0.9.5.sif /bin/bash /home/carla/CarlaUE4.sh /Game/Carla/Maps/Town01 -benchmark -carla-server -fps=10 -carla-no-hud

[Kailiangdong]

log.txt
@qhaas
My gpu is p100.Thank you.I follow you method. But when I execute the last step, it crashs and not work.
Can you help me to check the log and find the fault.
Thank you

[qhaas]

I was getting a similar error on one of our systems, not sure what fixed it... either a system restart or a CUDA process no longer sharing the same GPU as CARLA (UE4). CARLA (0.9.5 at least) is hardwired, it seems, to use GPU 0, so verify that there is no other CUDA job (Type C) running on GPU 0 with nvidia-smi -i 0

Also, make sure Singularity is up-to-date, outdated versions seem to develop strange issues.

I suspect that the following might have more than one cause given the amount of results from Googling it:

Failed to find symbol file, expected location:
"/home/carla/CarlaUE4/Binaries/Linux/CarlaUE4.sym"

For example: #1494

[Kailiangdong]

@qhaas Thank you.
How to set gpu in this command ?
"singularity exec --nv -B $(pwd)/CarlaUE4/Saved:/home/carla/CarlaUE4/Saved,$(pwd)/Engine/Saved:/home/carla/Engine/Saved carla_0.9.5.sif /bin/bash /home/carla/CarlaUE4.sh /Game/Carla/Maps/Town01 -benchmark -carla-server -fps=10 -carla-no-hud"

[Kailiangdong]

@qhaas
I struggled in Carla for three days. May I ask why you use Docker to run Carla? Is it because Tesla(v100,p100)?
I am a fresher and at first I didn't use docker and meet problem with opengl and nvidia driver.

[qhaas]

We haven't figured out a way to select a specific GPU in Singularity or outside containers without root or SLURM... there is a sub-discussion about how to select a GPU other than GPU 0 in #225

We use Singularity to run CARLA's Docker image because we don't have root access to the system with the V100s installed and to avoid dependency hell.

[Kailiangdong]

@qhaas Thank you very much.
I don't use docker finally, but upgrade nvidia Gpu driver to 410. Carla can run. But from xorg I can't see anything.
Do you also follow this tutorial for remote display?
https://carla.readthedocs.io/en/latest/carla_headless/

[Kailiangdong]

@qhaas Hi. Could you tell me your version of Opengl which successfully run Carla? Thank you.

[qhaas]

Tested with CARLA 0.9.6 (compiled version from the vendor), CentOS 7.7, was able to run it on the host system and inside the container.

glxinfo | grep 'OpenGL version'
OpenGL version string: 3.0.0 NVIDIA 430.50