RuntimeError: No such operator torchvision::nms

[davidsteinar]

🐛 Bug

Ran into this issue when trying to use Faster R-CCC example on a Windows machine.
RuntimeError: No such operator torchvision::nms

To Reproduce

Steps to reproduce the behavior:

conda create -n pytorch
conda activate pytorch
conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
python

and then

>>> import torch
>>> torch.ops.torchvision.nms
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\david\Miniconda3\envs\pytorch\lib\site-packages\torch\_ops.py", line 61, in __getattr__
    op = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator torchvision::nms

Expected behavior

To be able to use torchvision.nms ...

Environment

Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: GeForce GTX 1070
Nvidia driver version: 436.48
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.17.4
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.0                      166
[conda] mkl-service               2.3.0            py38hb782905_0
[conda] mkl_fft                   1.0.15           py38h14836fe_0
[conda] mkl_random                1.1.0            py38hf9181ef_0
[conda] pytorch                   1.4.0           py3.8_cuda101_cudnn7_0    pytorch
[conda] torchvision               0.5.0                py38_cu101    pytorch

and

nvidia-smi
Tue Feb 25 15:39:48 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 436.48       Driver Version: 436.48       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+

[fmassa]

Hi,

Thanks for the bug report!

There seems to be an issue with our binaries, at least for Windows.

@peterjc123 do you know if this is a known problem?

[davidsteinar]

I updated my CUDA Toolkit to 10.1 update2 and matching cuDNN to v7.6.5

Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: GeForce GTX 1070
Nvidia driver version: 436.48
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.17.4
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.0                      166
[conda] mkl-service               2.3.0            py38hb782905_0
[conda] mkl_fft                   1.0.15           py38h14836fe_0
[conda] mkl_random                1.1.0            py38hf9181ef_0
[conda] pytorch                   1.4.0           py3.8_cuda101_cudnn7_0    pytorch
[conda] torchvision               0.5.0                py38_cu101    pytorch

but I still get the same error.

[peterjc123]

Sorry, I could not reproduce this error with torch==1.4.0 and torchvision==0.5.0 with python=3.7 and cudatoolkit=10.1.

Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchvision
>>> import torch
>>> torch.ops.torchvision.nms
<built-in method nms of PyCapsule object at 0x0000029BF9F74150>

My env:

Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 10 Pro
GCC version: (Rev2, Built by MSYS2 project) 7.1.0
CMake version: version 3.16.4

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration: GPU 0: GeForce GTX 1070
Nvidia driver version: 442.19
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.17.3
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] blas                      1.0                         mkl    defaults
[conda] mkl                       2019.4                      245    defaults
[conda] mkl-service               2.3.0            py37hb782905_0    defaults
[conda] mkl_fft                   1.0.14           py37h14836fe_0    defaults
[conda] mkl_random                1.1.0            py37h675688f_0    defaults
[conda] pytorch                   1.4.0           py3.7_cuda101_cudnn7_0    pytorch
[conda] torchfile                 0.1.0                    pypi_0    pypi
[conda] torchvision               0.5.0                py37_cu101    pytorch

[davidsteinar]

It seems that when I do

conda create -n torch
conda activate torch
python -V

I get Python 3.6.8

Then when I do

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
python -V

I get Python 3.8.1. This environment produces the RuntimeError: No such operator torchvision::nms

But

if I specifically create a python 3.7 environment

conda create -n torch37 python=3.7
conda activate torch37
python -V

I get Python 3.7.6

Then

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
python -V

I also get Python 3.7.6. In this environment I get no error

import torch
import torchvision
torch.ops.torchvision.nms
>>> <built-in method nms of PyCapsule object at 0x00000201B0AC9FC0>

So the issue seems to be with conda install pytorch torchvision cudatoolkit=10.1 -c pytorch choosing python 3.8 and causing this issue...

[peterjc123]

You mean it's Python 3.8 only issue, right? I'll check later.

[davidsteinar]

Yes it seems so.

[amrotork]

I have the same issue. I'll try @davidsteinar solution. I'll remove the environment and create a new one for that.

Thanks @davidsteinar and @peterjc123

[peterjc123]

I tried and it is actually a DLL loading problem for Python 3.8. The workground is to add the directory of cuda before importing torchvision using the following code.

import os
import sys
py_dll_path = os.path.join(sys.exec_prefix, 'Library', 'bin')
os.add_dll_directory(py_dll_path)