Skip to content

terminate called after throwing an instance of 'at::Error' what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36) #77

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
JinQiangWang2021 opened this issue Nov 8, 2021 · 3 comments

Comments

@JinQiangWang2021
Copy link

JinQiangWang2021 commented Nov 8, 2021

/home/account/anaconda3/envs/RL17/bin/python /home/account/Documents/Deep_RL_Implementations/results/Cart_Pole.py
/home/account/anaconda3/envs/RL17/lib/python3.7/site-packages/gym/envs/registration.py:14: PkgResourcesDeprecationWarning: Parameters to load are deprecated. Call .resolve and .require separately.
result = entry_point.load(False)

AGENT NAME: A3C
1.1: A3C
TITLE CartPole
layer info [20, 10, [2, 1]]
layer info [20, 10, [2, 1]]
{'learning_rate': 0.005, 'linear_hidden_units': [20, 10], 'final_layer_activation': ['SOFTMAX', None], 'gradient_clipping_norm': 5.0, 'discount_rate': 0.99, 'epsilon_decay_rate_denominator': 1.0, 'normalise_rewards': True, 'exploration_worker_difference': 2.0, 'clip_rewards': False, 'Actor': {'learning_rate': 0.0003, 'linear_hidden_units': [64, 64], 'final_layer_activation': 'Softmax', 'batch_norm': False, 'tau': 0.005, 'gradient_clipping_norm': 5, 'initialiser': 'Xavier'}, 'Critic': {'learning_rate': 0.0003, 'linear_hidden_units': [64, 64], 'final_layer_activation': None, 'batch_norm': False, 'buffer_size': 1000000, 'tau': 0.005, 'gradient_clipping_norm': 5, 'initialiser': 'Xavier'}, 'min_steps_before_learning': 400, 'batch_size': 256, 'mu': 0.0, 'theta': 0.15, 'sigma': 0.25, 'action_noise_std': 0.2, 'action_noise_clipping_range': 0.5, 'update_every_n_steps': 1, 'learning_updates_per_learning_session': 1, 'automatically_tune_entropy_hyperparameter': True, 'entropy_term_weight': None, 'add_extra_noise': False, 'do_evaluation_iterations': True, 'output_activation': None, 'hidden_activations': 'relu', 'dropout': 0.0, 'initialiser': 'default', 'batch_norm': False, 'columns_of_data_to_be_embedded': [], 'embedding_dimensions': [], 'y_range': ()}
RANDOM SEED 2741982467
terminate called after throwing an instance of 'at::Error'
what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36)
frame #0: torch::autograd::Engine::thread_init(int) + 0x34 (0x7f4a94844264 in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #1: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7f4a94873a5a in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #2: + 0xc92bd (0x7f4a6872a2bd in /home/jqw/anaconda3/envs/RL17/bin/../lib/libstdc++.so.6)
frame #3: + 0x76ba (0x7f4aa8e396ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x6d (0x7f4aa8b6f51d in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'at::Error'
what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36)
frame #0: torch::autograd::Engine::thread_init(int) + 0x34 (0x7f4a94844264 in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #1: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7f4a94873a5a in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #2: + 0xc92bd (0x7f4a6872a2bd in /home/jqw/anaconda3/envs/RL17/bin/../lib/libstdc++.so.6)
frame #3: + 0x76ba (0x7f4aa8e396ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x6d (0x7f4aa8b6f51d in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'at::Error'
what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36)
frame #0: torch::autograd::Engine::thread_init(int) + 0x34 (0x7f4a94844264 in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #1: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7f4a94873a5a in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #2: + 0xc92bd (0x7f4a6872a2bd in /home/jqw/anaconda3/envs/RL17/bin/../lib/libstdc++.so.6)
frame #3: + 0x76ba (0x7f4aa8e396ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x6d (0x7f4aa8b6f51d in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'at::Error'
what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36)
frame #0: torch::autograd::Engine::thread_init(int) + 0x34 (0x7f4a94844264 in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #1: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7f4a94873a5a in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #2: + 0xc92bd (0x7f4a6872a2bd in /home/jqw/anaconda3/envs/RL17/bin/../lib/libstdc++.so.6)
frame #3: + 0x76ba (0x7f4aa8e396ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x6d (0x7f4aa8b6f51d in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'at::Error'
what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36)
frame #0: torch::autograd::Engine::thread_init(int) + 0x34 (0x7f4a94844264 in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #1: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7f4a94873a5a in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #2: + 0xc92bd (0x7f4a6872a2bd in /home/jqw/anaconda3/envs/RL17/bin/../lib/libstdc++.so.6)
frame #3: + 0x76ba (0x7f4aa8e396ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x6d (0x7f4aa8b6f51d in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'at::Error'
what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36)
frame #0: torch::autograd::Engine::thread_init(int) + 0x34 (0x7f4a94844264 in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #1: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7f4a94873a5a in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #2: + 0xc92bd (0x7f4a6872a2bd in /home/jqw/anaconda3/envs/RL17/bin/../lib/libstdc++.so.6)
frame #3: + 0x76ba (0x7f4aa8e396ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x6d (0x7f4aa8b6f51d in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'at::Error'
what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36)
frame #0: torch::autograd::Engine::thread_init(int) + 0x34 (0x7f4a94844264 in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #1: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7f4a94873a5a in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #2: + 0xc92bd (0x7f4a6872a2bd in /home/jqw/anaconda3/envs/RL17/bin/../lib/libstdc++.so.6)
frame #3: + 0x76ba (0x7f4aa8e396ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x6d (0x7f4aa8b6f51d in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'at::Error'
what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36)
frame #0: torch::autograd::Engine::thread_init(int) + 0x34 (0x7f4a94844264 in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #1: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7f4a94873a5a in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #2: + 0xc92bd (0x7f4a6872a2bd in /home/jqw/anaconda3/envs/RL17/bin/../lib/libstdc++.so.6)
frame #3: + 0x76ba (0x7f4aa8e396ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x6d (0x7f4aa8b6f51d in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'at::Error'
what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36)
frame #0: torch::autograd::Engine::thread_init(int) + 0x34 (0x7f4a94844264 in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #1: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7f4a94873a5a in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #2: + 0xc92bd (0x7f4a6872a2bd in /home/jqw/anaconda3/envs/RL17/bin/../lib/libstdc++.so.6)
frame #3: + 0x76ba (0x7f4aa8e396ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x6d (0x7f4aa8b6f51d in /lib/x86_64-linux-gnu/libc.so.6)

terminate called after throwing an instance of 'at::Error'
what(): CUDA error (3): initialization error (check_status at /pytorch/aten/src/ATen/cuda/detail/CUDAHooks.cpp:36)
frame #0: torch::autograd::Engine::thread_init(int) + 0x34 (0x7f4a94844264 in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #1: torch::autograd::python::PythonEngine::thread_init(int) + 0x2a (0x7f4a94873a5a in /home/jqw/anaconda3/envs/RL17/lib/python3.7/site-packages/torch/_C.cpython-37m-x86_64-linux-gnu.so)
frame #2: + 0xc92bd (0x7f4a6872a2bd in /home/jqw/anaconda3/envs/RL17/bin/../lib/libstdc++.so.6)
frame #3: + 0x76ba (0x7f4aa8e396ba in /lib/x86_64-linux-gnu/libpthread.so.0)
frame #4: clone + 0x6d (0x7f4aa8b6f51d in /lib/x86_64-linux-gnu/libc.so.6)

@JinQiangWang2021
Copy link
Author

I have Solved this question:

import os 

os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]=""

@MISTCARRYYOU
Copy link

MISTCARRYYOU commented Dec 15, 2021

nice

I have Solved this question:

import os 

os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBLE_DEVICES"]=""

@XiaoTailong
Copy link

XiaoTailong commented Dec 15, 2021 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants