-
Notifications
You must be signed in to change notification settings - Fork 54
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RuntimeError: nvrtc: error: invalid value for --gpu-architecture (-arch) #1
Comments
Hi, this seems to be an issue with PyTorch and nvrtc for the new RTX 4090 GPUs. Please check this issue, and more specifically, this comment. As suggested in the thread, perhaps upgrading a python version to a nightly build may resolve the issue as detailed in this comment. |
Hello, Thank you for your help. I will test it and let you know about the result. |
Hi, have you solved this problem? |
I also had this issue, and I was able to solve it with
and restarting the computer (as suggested in the comment pytorch/pytorch#87595 (comment)). I'm running pop OS 22.04 on a 4080 with cuda_11.5.r11.5/compiler.30672275_0 |
@pompomO @rasrab1992 Can you please try the above solution here to see if it works? If not, please try out the fix provided in #3. |
I'm facing the same issue with GPU RTX 4060 and Ubuntu 20.04, and going to test the solutions provided here. I will post the results here after testing your solutions. |
I have tested this solution, but the issue is not solved. |
@MJavadZallaghi what version of torch, torchvision and torchaudio are you using? I'm on pytorch/pytorch#87595 (comment) says that updating these to newer versions can solve the issue. You might also need to update the cuda version. Im on |
@EtorArza This is version list of all the installed package inside the # packages in environment at /home/mjavadzallaghi/anaconda3/envs/rlgpu:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_kmp_llvm conda-forge
absl-py 2.1.0 pyhd8ed1ab_0 conda-forge
aerial-gym 1.0.0 dev_0 <develop>
aiohttp 3.7.4.post0 py37h5e8e339_1 conda-forge
async-timeout 3.0.1 py_1000 conda-forge
attrs 23.2.0 pyh71513ae_0 conda-forge
blas 2.122 mkl conda-forge
blas-devel 3.9.0 22_linux64_mkl conda-forge
blinker 1.6.3 pyhd8ed1ab_0 conda-forge
brotli-python 1.0.9 py37hd23a5d3_7 conda-forge
bzip2 1.0.8 hd590300_5 conda-forge
c-ares 1.28.1 hd590300_0 conda-forge
ca-certificates 2024.3.11 h06a4308_0
cachetools 5.3.3 pyhd8ed1ab_0 conda-forge
certifi 2024.2.2 pyhd8ed1ab_0 conda-forge
cffi 1.15.1 py37h43b0acd_1 conda-forge
chardet 4.0.0 py37h89c1867_3 conda-forge
charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge
click 8.1.3 py37h89c1867_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
cryptography 38.0.2 py37h5994e8b_1 conda-forge
cuda-cudart 12.1.105 0 nvidia
cuda-cupti 12.1.105 0 nvidia
cuda-libraries 12.1.0 0 nvidia
cuda-nvrtc 12.1.105 0 nvidia
cuda-nvtx 12.1.105 0 nvidia
cuda-opencl 12.4.127 0 nvidia
cuda-runtime 12.1.0 0 nvidia
cudatoolkit 11.1.1 hb139c0e_13 conda-forge
cycler 0.11.0 pypi_0 pypi
ffmpeg 4.3 hf484d3e_0 pytorch
fonttools 4.38.0 pypi_0 pypi
freetype 2.12.1 h267a509_2 conda-forge
fvcore 0.1.5.post20210915 py37 fvcore
gmp 6.3.0 h59595ed_1 conda-forge
gnutls 3.6.13 h85f3911_1 conda-forge
google-auth 2.23.0 pyh1a96a4e_0 conda-forge
google-auth-oauthlib 0.4.6 pyhd8ed1ab_0 conda-forge
grpc-cpp 1.48.1 h30feacc_1 conda-forge
grpcio 1.48.1 py37he7b19e7_1 conda-forge
icu 73.2 h59595ed_0 conda-forge
idna 3.7 pyhd8ed1ab_0 conda-forge
imageio 2.31.2 pypi_0 pypi
importlib-metadata 4.11.4 py37h89c1867_0 conda-forge
intel-openmp 2024.1.0 pypi_0 pypi
iopath 0.1.9 py37 iopath
isaacgym 1.0rc4 dev_0 <develop>
jpeg 9b h024ee3a_2
kiwisolver 1.4.5 pypi_0 pypi
lame 3.100 h166bdaf_1003 conda-forge
ld_impl_linux-64 2.40 h55db66e_0 conda-forge
libabseil 20220623.0 cxx17_h05df665_6 conda-forge
libblas 3.9.0 22_linux64_mkl conda-forge
libcblas 3.9.0 22_linux64_mkl conda-forge
libcublas 12.1.0.26 0 nvidia
libcufft 11.0.2.4 0 nvidia
libcufile 1.9.1.3 0 nvidia
libcurand 10.3.5.147 0 nvidia
libcusolver 11.4.4.55 0 nvidia
libcusparse 12.0.2.55 0 nvidia
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.2.0 h77fa898_7 conda-forge
libgfortran-ng 13.2.0 h69a702a_7 conda-forge
libgfortran5 13.2.0 hca663fb_7 conda-forge
libhwloc 2.10.0 default_h2fb2949_1000 conda-forge
libiconv 1.17 hd590300_2 conda-forge
liblapack 3.9.0 22_linux64_mkl conda-forge
liblapacke 3.9.0 22_linux64_mkl conda-forge
libnpp 12.0.2.50 0 nvidia
libnsl 2.0.1 hd590300_0 conda-forge
libnvjitlink 12.1.105 0 nvidia
libnvjpeg 12.1.1.14 0 nvidia
libpng 1.6.43 h2797004_0 conda-forge
libprotobuf 3.21.8 h6239696_0 conda-forge
libsqlite 3.45.3 h2797004_0 conda-forge
libstdcxx-ng 13.2.0 hc0a3c3a_7 conda-forge
libtiff 4.0.9 he6b73bb_1 conda-forge
libuv 1.48.0 hd590300_0 conda-forge
libxml2 2.12.6 h232c23b_2 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
llvm-openmp 18.1.5 ha31de31_0 conda-forge
markdown 3.6 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.1 py37h540881e_1 conda-forge
matplotlib 3.5.3 pypi_0 pypi
mkl 2024.0.0 pypi_0 pypi
mkl-devel 2024.1.0 ha770c72_692 conda-forge
mkl-include 2024.1.0 ha957f24_692 conda-forge
multidict 6.0.2 py37h540881e_1 conda-forge
ncurses 6.4.20240210 h59595ed_0 conda-forge
nettle 3.6 he412f7d_0 conda-forge
ninja 1.11.1.1 pypi_0 pypi
numpy 1.21.6 py37h976b520_0 conda-forge
oauthlib 3.2.2 pyhd8ed1ab_0 conda-forge
olefile 0.47 pyhd8ed1ab_0 conda-forge
openh264 2.1.1 h780b84a_0 conda-forge
openssl 3.3.0 hd590300_0 conda-forge
packaging 24.0 pypi_0 pypi
pillow 9.5.0 pypi_0 pypi
pip 24.0 pyhd8ed1ab_0 conda-forge
portalocker 2.6.0 py37h89c1867_0 conda-forge
protobuf 4.21.8 py37hd23a5d3_0 conda-forge
pyasn1 0.5.1 pyhd8ed1ab_0 conda-forge
pyasn1-modules 0.3.0 pyhd8ed1ab_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyjwt 2.8.0 pyhd8ed1ab_1 conda-forge
pyopenssl 23.2.0 pyhd8ed1ab_1 conda-forge
pyparsing 3.1.2 pypi_0 pypi
pysocks 1.7.1 py37h89c1867_5 conda-forge
python 3.7.12 hf930737_100_cpython conda-forge
python-dateutil 2.9.0.post0 pypi_0 pypi
python_abi 3.7 4_cp37m conda-forge
pytorch 1.8.1 py3.7_cuda11.1_cudnn8.0.5_0 pytorch
pytorch-cuda 12.1 ha16c6d3_5 pytorch
pytorch3d 0.7.0 py37_cu111_pyt181 pytorch3d
pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge
pyyaml 6.0 py37h540881e_4 conda-forge
re2 2022.06.01 h27087fc_1 conda-forge
readline 8.2 h8228510_1 conda-forge
requests 2.31.0 pyhd8ed1ab_0 conda-forge
requests-oauthlib 2.0.0 pyhd8ed1ab_0 conda-forge
rsa 4.9 pyhd8ed1ab_0 conda-forge
scipy 1.7.3 py37hf2a6cf1_0 conda-forge
setuptools 69.0.3 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
sqlite 3.45.3 h2c6b66d_0 conda-forge
tabulate 0.9.0 pyhd8ed1ab_1 conda-forge
tbb 2021.12.0 pypi_0 pypi
tensorboard 2.11.2 pyhd8ed1ab_0 conda-forge
tensorboard-data-server 0.6.1 py37h52d8a92_0
tensorboard-plugin-wit 1.8.1 pyhd8ed1ab_0 conda-forge
termcolor 2.3.0 pyhd8ed1ab_0 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
torchaudio 0.8.1 py37 pytorch
torchvision 0.9.1 py37_cu111 pytorch
tqdm 4.66.4 pyhd8ed1ab_0 conda-forge
typing-extensions 4.7.1 hd8ed1ab_0 conda-forge
typing_extensions 4.7.1 pyha770c72_0 conda-forge
urllib3 1.26.18 pyhd8ed1ab_0 conda-forge
werkzeug 2.2.3 pyhd8ed1ab_0 conda-forge
wheel 0.42.0 pyhd8ed1ab_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yacs 0.1.8 pyhd8ed1ab_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
yarl 1.7.2 py37h540881e_2 conda-forge
zipp 3.15.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.13 hd590300_5 conda-forge
zstd 1.5.6 ha6fb4c9_0 conda-forge
|
@MJavadZallaghi upgrading your pytorch might solve the issue. Activate the environment
However, for the latest pytorch version 2.3.0, it says in the website that python 3.8 is required, and you seem to be running python 3.7, so I don't know if it will work. I think @mihirk284 is runnig python 3.7 so perhaps he can tell us what pytorch version he is running? |
@EtorArza I have tried this command, see the result: mjavadzallaghi@mjavadzallaghi-Legion-Pro-5-16IRX9:~/aerial_gym_ws/aerial_gym_reference_governor/aerial_gym/scripts$ conda activate rlgpu
(rlgpu) mjavadzallaghi@mjavadzallaghi-Legion-Pro-5-16IRX9:~/aerial_gym_ws/aerial_gym_reference_governor/aerial_gym/scripts$ conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
Retrieving notices: ...working... done
Channels:
- pytorch
- nvidia
- defaults
Platform: linux-64
Collecting package metadata (repodata.json): done
Solving environment: done
# All requested packages already installed.
(rlgpu) mjavadzallaghi@mjavadzallaghi-Legion-Pro-5-16IRX9:~/aerial_gym_ws/aerial_gym_reference_governor/aerial_gym/scripts$ As you mentioned, python version of |
@MJavadZallaghi I'm on python 3.8, which is why I can install the latest pytorch version. Im in 22.04, so I get python 3.8 by default. It should be possible to install python 3.8 in 20.04 as well. You need to create the rlgpu environment from scratch. You can change the isaacgym/python/rlgpu_conda_env.yml file before running
After you edit this file and run |
@EtorArza Thank you! I have solved the issue with your help. For the next person who face the issue, I did these steps:
name: rlgpu
channels:
- pytorch
- conda-forge
- defaults
dependencies:
- python=3.8 #3.7
#- pytorch=2.3.0 #1.8.1
#- torchvision=0.9.1
#- cudatoolkit=12.1 #11.1
- pyyaml>=5.3.1
- scipy>=1.5.0
- tensorboard>=2.2.1
For more information:
(rlgpu) mjavadzallaghi@mjavadzallaghi-Legion-Pro-5-16IRX9:~/aerial_gym_ws/aerial_gym_reference_governor/aerial_gym/scripts$ nvidia-smi
Wed May 8 12:01:30 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.171.04 Driver Version: 535.171.04 CUDA Version: 12.2 |
|-----------------------------------------+----------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 NVIDIA GeForce RTX 4060 ... Off | 00000000:01:00.0 Off | N/A |
| N/A 37C P4 N/A / 50W | 14MiB / 8188MiB | 0% Default |
| | | N/A |
+-----------------------------------------+----------------------+----------------------+
+---------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 1162 G /usr/lib/xorg/Xorg 4MiB |
| 0 N/A N/A 1958 G /usr/lib/xorg/Xorg 4MiB |
+---------------------------------------------------------------------------------------+
# packages in environment at /home/mjavadzallaghi/anaconda3/envs/rlgpu:
#
# Name Version Build Channel
_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 2_gnu conda-forge
absl-py 2.1.0 pyhd8ed1ab_0 conda-forge
aerial-gym 1.0.0 dev_0 <develop>
brotli-python 1.1.0 py38h17151c0_1 conda-forge
bzip2 1.0.8 hd590300_5 conda-forge
c-ares 1.28.1 hd590300_0 conda-forge
ca-certificates 2024.2.2 hbcca054_0 conda-forge
certifi 2024.2.2 pyhd8ed1ab_0 conda-forge
charset-normalizer 3.3.2 pyhd8ed1ab_0 conda-forge
colorama 0.4.6 pyhd8ed1ab_0 conda-forge
contourpy 1.1.1 pypi_0 pypi
cycler 0.12.1 pypi_0 pypi
filelock 3.14.0 pypi_0 pypi
fonttools 4.51.0 pypi_0 pypi
freetype 2.12.1 h267a509_2 conda-forge
fsspec 2024.3.1 pypi_0 pypi
fvcore 0.1.5.post20210915 py38 fvcore
grpcio 1.62.2 py38h94a1851_0 conda-forge
idna 3.7 pyhd8ed1ab_0 conda-forge
imageio 2.34.1 pypi_0 pypi
importlib-metadata 7.1.0 pyha770c72_0 conda-forge
importlib-resources 6.4.0 pypi_0 pypi
iopath 0.1.9 py38 iopath
isaacgym 1.0rc4 dev_0 <develop>
jinja2 3.1.4 pypi_0 pypi
kiwisolver 1.4.5 pypi_0 pypi
lcms2 2.16 hb7c19ff_0 conda-forge
ld_impl_linux-64 2.40 h55db66e_0 conda-forge
lerc 4.0.0 h27087fc_0 conda-forge
libabseil 20240116.2 cxx17_h59595ed_0 conda-forge
libblas 3.9.0 20_linux64_openblas conda-forge
libcblas 3.9.0 20_linux64_openblas conda-forge
libdeflate 1.20 hd590300_0 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 13.2.0 h77fa898_7 conda-forge
libgfortran-ng 13.2.0 h69a702a_7 conda-forge
libgfortran5 13.2.0 hca663fb_7 conda-forge
libgomp 13.2.0 h77fa898_7 conda-forge
libgrpc 1.62.2 h15f2491_0 conda-forge
libjpeg-turbo 3.0.0 hd590300_1 conda-forge
liblapack 3.9.0 20_linux64_openblas conda-forge
libnsl 2.0.1 hd590300_0 conda-forge
libopenblas 0.3.25 pthreads_h413a1c8_0 conda-forge
libpng 1.6.43 h2797004_0 conda-forge
libprotobuf 4.25.3 h08a7969_0 conda-forge
libre2-11 2023.09.01 h5a48ba9_2 conda-forge
libsqlite 3.45.3 h2797004_0 conda-forge
libstdcxx-ng 13.2.0 hc0a3c3a_7 conda-forge
libtiff 4.6.0 h1dd3fc0_3 conda-forge
libuuid 2.38.1 h0b41bf4_0 conda-forge
libwebp-base 1.4.0 hd590300_0 conda-forge
libxcb 1.15 h0b41bf4_0 conda-forge
libxcrypt 4.4.36 hd590300_1 conda-forge
libzlib 1.2.13 hd590300_5 conda-forge
markdown 3.6 pyhd8ed1ab_0 conda-forge
markupsafe 2.1.5 py38h01eb140_0 conda-forge
matplotlib 3.7.5 pypi_0 pypi
mpmath 1.3.0 pypi_0 pypi
ncurses 6.4.20240210 h59595ed_0 conda-forge
networkx 3.1 pypi_0 pypi
ninja 1.11.1.1 pypi_0 pypi
numpy 1.24.4 py38h59b608b_0 conda-forge
nvidia-cublas-cu12 12.1.3.1 pypi_0 pypi
nvidia-cuda-cupti-cu12 12.1.105 pypi_0 pypi
nvidia-cuda-nvrtc-cu12 12.1.105 pypi_0 pypi
nvidia-cuda-runtime-cu12 12.1.105 pypi_0 pypi
nvidia-cudnn-cu12 8.9.2.26 pypi_0 pypi
nvidia-cufft-cu12 11.0.2.54 pypi_0 pypi
nvidia-curand-cu12 10.3.2.106 pypi_0 pypi
nvidia-cusolver-cu12 11.4.5.107 pypi_0 pypi
nvidia-cusparse-cu12 12.1.0.106 pypi_0 pypi
nvidia-nccl-cu12 2.20.5 pypi_0 pypi
nvidia-nvjitlink-cu12 12.4.127 pypi_0 pypi
nvidia-nvtx-cu12 12.1.105 pypi_0 pypi
openjpeg 2.5.2 h488ebb8_0 conda-forge
openssl 3.3.0 hd590300_0 conda-forge
packaging 24.0 pyhd8ed1ab_0 conda-forge
pillow 10.3.0 py38h9e66945_0 conda-forge
pip 24.0 pyhd8ed1ab_0 conda-forge
platformdirs 4.2.1 pyhd8ed1ab_0 conda-forge
pooch 1.8.1 pyhd8ed1ab_0 conda-forge
portalocker 2.8.2 py38h578d9bd_1 conda-forge
protobuf 4.25.3 py38hb5c7596_0 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
pyparsing 3.1.2 pypi_0 pypi
pysocks 1.7.1 pyha2e5f31_6 conda-forge
python 3.8.19 hd12c33a_0_cpython conda-forge
python-dateutil 2.9.0.post0 pypi_0 pypi
python_abi 3.8 4_cp38 conda-forge
pytorch3d 0.3.0 pypi_0 pypi
pyyaml 6.0.1 py38h01eb140_1 conda-forge
re2 2023.09.01 h7f4b329_2 conda-forge
readline 8.2 h8228510_1 conda-forge
requests 2.31.0 pyhd8ed1ab_0 conda-forge
scipy 1.10.1 py38h59b608b_3 conda-forge
setuptools 69.5.1 pyhd8ed1ab_0 conda-forge
six 1.16.0 pyh6c4a22f_0 conda-forge
sympy 1.12 pypi_0 pypi
tabulate 0.9.0 pyhd8ed1ab_1 conda-forge
tensorboard 2.16.2 pyhd8ed1ab_0 conda-forge
tensorboard-data-server 0.7.0 py38hcdda232_1 conda-forge
termcolor 2.4.0 pyhd8ed1ab_0 conda-forge
tk 8.6.13 noxft_h4845f30_101 conda-forge
torch 2.3.0 pypi_0 pypi
torchvision 0.18.0 pypi_0 pypi
tqdm 4.66.4 pyhd8ed1ab_0 conda-forge
transitions 0.9.0 pypi_0 pypi
triton 2.3.0 pypi_0 pypi
typing-extensions 4.11.0 pypi_0 pypi
urllib3 2.2.1 pyhd8ed1ab_0 conda-forge
werkzeug 3.0.3 pyhd8ed1ab_0 conda-forge
wheel 0.43.0 pyhd8ed1ab_1 conda-forge
xorg-libxau 1.0.11 hd590300_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xz 5.2.6 h166bdaf_0 conda-forge
yacs 0.1.8 pyhd8ed1ab_0 conda-forge
yaml 0.2.5 h7f98852_2 conda-forge
zipp 3.17.0 pyhd8ed1ab_0 conda-forge
zstd 1.5.6 ha6fb4c9_0 conda-forge
|
Hi, I followed the steps commented by @MJavadZallaghi but now I received this error:
For more information
|
Solved changing the python version on |
Hello,
I am a researcher from IMEC involved in the SPEAR project. I'm eager to explore your simulation tool before the kickoff meeting so that I can formulate questions and concerns for WP2. However, I encountered an error while trying to install the 'aerial_gym_simulator,' as shown in the image below:
I have attempted to find a solution by searching the Nvidia forums and discovered that others have experienced a similar issue related to the RTX 4090 GPU, as documented here. I've tried both conda and docker, but the problem persists.
Could you please provide your suggestions on how to proceed with resolving this issue?
Maybe one solution is changing the driver version to 525?
Thanks
The text was updated successfully, but these errors were encountered: