Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED #136

Open
EJDU21 opened this issue Dec 27, 2022 · 1 comment
Open

failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED #136

EJDU21 opened this issue Dec 27, 2022 · 1 comment

Comments

@EJDU21
Copy link

EJDU21 commented Dec 27, 2022

How can I save this problem? I use 3080ti /tensorflow 1.15 / python 3.7.10 / cuda 10.1

2022-12-27 15:27:14.672772: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
ERROR:tensorflow:Error recorded from training_loop: 2 root error(s) found.
(0) Internal: Blas GEMM launch failed : a.shape=(4096, 2), b.shape=(2, 128), m=4096, n=128, k=2
[[node electra/embeddings_1/MatMul (defined at /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[add_10/_9743]]
(1) Internal: Blas GEMM launch failed : a.shape=(4096, 2), b.shape=(2, 128), m=4096, n=128, k=2
[[node electra/embeddings_1/MatMul (defined at /opt/conda/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
0 successful operations.
0 derived errors ignored.

@martinpolacek
Copy link

Hey, you have to use NVIDIA version of TF, which supports training on RTX cards.

https://github.com/NVIDIA/tensorflow#install

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants