Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test_gpu fails on RTX 3090 with CUDA 11.1 #89

Open
mdabek opened this issue Mar 24, 2021 · 2 comments
Open

test_gpu fails on RTX 3090 with CUDA 11.1 #89

mdabek opened this issue Mar 24, 2021 · 2 comments

Comments

@mdabek
Copy link

mdabek commented Mar 24, 2021

Thank you for the warp-transducer, I was successfully using it with previous CUDA versions and on older card (RTX 2080) and with previous Ubuntu 18.04. I am currently trying to change the setup and move to the following configuration:
HW: RTX 3090Ti
OS: Ubuntu 20.04
CUDA: 11.1

The first change I needed to make was to remove the compute_30, since it is not supported by Cuda 11.1. The code can be compiled and I am able to run test_gpu. However, the option_test in test_gpu fails.
Here is the output:

Running gpu tests
finish small_test 0
grad mismatch in options_test expected grad: -0.186844 calculated score: 0 !(-0.186944 < 0 < -0.186744)
grad mismatch in options_test expected grad: -0.062555 calculated score: 0 !(-0.062655 < 0 < -0.062455)
grad mismatch in options_test expected grad: 0.249399 calculated score: 0 !(0.249299 < 0 < 0.249499)
grad mismatch in options_test expected grad: -0.203377 calculated score: 0 !(-0.203477 < 0 < -0.203277)
grad mismatch in options_test expected grad: 0.202399 calculated score: 0 !(0.202299 < 0 < 0.202499)
grad mismatch in options_test expected grad: 0.000977 calculated score: 0 !(0.000877 < 0 < 0.001077)
grad mismatch in options_test expected grad: -0.141016 calculated score: 0 !(-0.141116 < 0 < -0.140916)
grad mismatch in options_test expected grad: 0.079123 calculated score: 0 !(0.079023 < 0 < 0.079223)
grad mismatch in options_test expected grad: 0.061893 calculated score: 0 !(0.061793 < 0 < 0.061993)
grad mismatch in options_test expected grad: -0.011552 calculated score: 0 !(-0.011652 < 0 < -0.011452)
grad mismatch in options_test expected grad: -0.08128 calculated score: 0 !(-0.08138 < 0 < -0.08118)
grad mismatch in options_test expected grad: 0.092832 calculated score: 0 !(0.092732 < 0 < 0.092932)
grad mismatch in options_test expected grad: -0.154257 calculated score: 0 !(-0.154357 < 0 < -0.154157)
grad mismatch in options_test expected grad: 0.229433 calculated score: 0 !(0.229333 < 0 < 0.229533)
grad mismatch in options_test expected grad: -0.075176 calculated score: 0 !(-0.075276 < 0 < -0.075076)
....
score mismatch in options_test expected score: 4.28065 calculated score: -0 !(4.28055 < -0 < 4.28075)
score mismatch in options_test expected score: 3.93844 calculated score: -0 !(3.93834 < -0 < 3.93854)
finish options_test 0
finish inf_test 0
finished 0
Some or all tests fail

All the grads calculated by option_test are 0.
Is this known issue? How can it be debugged and solved?

Thank you!

@RossSong
Copy link

RossSong commented Apr 12, 2021

Maybe you should use "compute_86" for RTX 3090

https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/

@flp1990
Copy link

flp1990 commented Nov 19, 2021

Maybe you should use "compute_86" for RTX 3090

https://arnon.dk/matching-sm-architectures-arch-and-gencode-for-various-nvidia-cards/

you are right!!!
thank you very much,
i use RTX 3080, install and uninstall nvcc and NVIDAI driver many times......

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants