Tensorflow 2.3 doesn't recognize GeForce MX130 GPU

[elishafer]

System information

• OS Platform and Distribution: Linux Ubuntu 18.04.4
• TensorFlow installed from: binary as per instructions on tf website
• TensorFlow version: 2.3
• Python version: 3.6
• Installed using : pip
• CUDA/cuDNN version: CUDA 10.1, cuDNN 7
• GPU model and memory: GeForce MX130, 2GB

Problem Description:
Tensorflow 2.3 doesn't recognize my GPU.

Provide the exact sequence of commands / steps that you executed before running into the problem
After installation I run the following:

python3 -c "import tensorflow as tf;print(tf.reduce_sum(tf.random.normal([1000, 1000])))"

I get this error message:

tensorflow.python.framework.errors_impl.InternalError: CUDA runtime implicit initialization on GPU:0 failed. Status: device kernel image is invalid

On Tensorflow 2.2, I don't get the error.

[sedol1339]

Tensorflow 2.3 also does not work with TPU: #41542

[ravikyram]

@elishafer

Can you please check if tensorflow sees your GPU by running below code.

pip install tensorflow-gpu
print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))

Thanks!

[shaprann]

I get the same error on GeForce 840M, compute capability 5.0
Downgrading to TF 2.2 solves it, but it would be nice to understand why 2.3 fails though

[sanjoy]

CC @chsigg @sandeepngupta @manivaradarajan

This is likely because starting TF 2.3 we don't ship PTX for older compute capabilities to reduce the size of the TF pip package.

The simplest solution for you is to build the TF pip from source (this will take a while but it is a one-time cost) with the compute capabilities you need included.

[elishafer]

@ravikyram
I ran what you requested:

pip install tensorflow-gpu

>>> print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
2020-08-02 09:48:16.786563: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-08-02 09:48:16.838980: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 09:48:16.839502: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce MX130 computeCapability: 5.0
coreClock: 1.189GHz coreCount: 3 deviceMemorySize: 1.96GiB deviceMemoryBandwidth: 37.33GiB/s
2020-08-02 09:48:16.839577: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-08-02 09:48:16.844172: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-08-02 09:48:16.847447: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-08-02 09:48:16.849151: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-08-02 09:48:16.854795: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-08-02 09:48:16.858097: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-08-02 09:48:16.869744: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-08-02 09:48:16.870083: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 09:48:16.870725: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 09:48:16.871176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
Num GPUs Available:  1

BTW, maybe the error Status: device kernel image is invalid should be mentioned in the installation documentation because I found very little mention of this error anywhere and it took me several hours to figure out the cause and solution.

[sanjoy]

BTW, maybe the error Status: device kernel image is invalid should be mentioned in the installation documentation

That's a good idea. @chsigg Can you please add this error message to https://www.tensorflow.org/install/gpu?

[manivaradarajan]

Please see tensorflow/docs#1642 which updates GPU.md.

[elishafer]

I'm closing this issue, since my question has been answered.
I'm impressed by your speedy response and work.
Cheers!

[google-ml-butler]

Are you satisfied with the resolution of your issue?
Yes
No

[lgariv]

@ravikyram
I ran what you requested:

pip install tensorflow-gpu

>>> print("Num GPUs Available: ", len(tf.config.experimental.list_physical_devices('GPU')))
2020-08-02 09:48:16.786563: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcuda.so.1
2020-08-02 09:48:16.838980: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 09:48:16.839502: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1716] Found device 0 with properties: 
pciBusID: 0000:01:00.0 name: GeForce MX130 computeCapability: 5.0
coreClock: 1.189GHz coreCount: 3 deviceMemorySize: 1.96GiB deviceMemoryBandwidth: 37.33GiB/s
2020-08-02 09:48:16.839577: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-08-02 09:48:16.844172: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcublas.so.10
2020-08-02 09:48:16.847447: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcufft.so.10
2020-08-02 09:48:16.849151: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcurand.so.10
2020-08-02 09:48:16.854795: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusolver.so.10
2020-08-02 09:48:16.858097: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcusparse.so.10
2020-08-02 09:48:16.869744: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudnn.so.7
2020-08-02 09:48:16.870083: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 09:48:16.870725: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:982] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2020-08-02 09:48:16.871176: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1858] Adding visible gpu devices: 0
Num GPUs Available:  1

BTW, maybe the error Status: device kernel image is invalid should be mentioned in the installation documentation because I found very little mention of this error anywhere and it took me several hours to figure out the cause and solution.

Hey, I'm using the TensorFlow 2.2.0 from pip, and I get Num GPUs Available: 0. Did you have to do anything special to make it work?

[manivaradarajan]

Please open a new issue instead of piggybacking on a closed issue.