RuntimeError: No such operator torchvision::nms #1916

davidsteinar · 2020-02-25T15:40:53Z

🐛 Bug

Ran into this issue when trying to use Faster R-CCC example on a Windows machine.
RuntimeError: No such operator torchvision::nms

To Reproduce

Steps to reproduce the behavior:

conda create -n pytorch
conda activate pytorch
conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
python

and then

>>> import torch
>>> torch.ops.torchvision.nms
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\david\Miniconda3\envs\pytorch\lib\site-packages\torch\_ops.py", line 61, in __getattr__
    op = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator torchvision::nms

Expected behavior

To be able to use torchvision.nms ...

Environment

Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: GeForce GTX 1070
Nvidia driver version: 436.48
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.17.4
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.0                      166
[conda] mkl-service               2.3.0            py38hb782905_0
[conda] mkl_fft                   1.0.15           py38h14836fe_0
[conda] mkl_random                1.1.0            py38hf9181ef_0
[conda] pytorch                   1.4.0           py3.8_cuda101_cudnn7_0    pytorch
[conda] torchvision               0.5.0                py38_cu101    pytorch

and

nvidia-smi
Tue Feb 25 15:39:48 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 436.48       Driver Version: 436.48       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+

The text was updated successfully, but these errors were encountered:

fmassa · 2020-02-25T16:01:41Z

Hi,

Thanks for the bug report!

There seems to be an issue with our binaries, at least for Windows.

@peterjc123 do you know if this is a known problem?

davidsteinar · 2020-02-25T16:52:08Z

I updated my CUDA Toolkit to 10.1 update2 and matching cuDNN to v7.6.5

Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: GeForce GTX 1070
Nvidia driver version: 436.48
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.17.4
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.0                      166
[conda] mkl-service               2.3.0            py38hb782905_0
[conda] mkl_fft                   1.0.15           py38h14836fe_0
[conda] mkl_random                1.1.0            py38hf9181ef_0
[conda] pytorch                   1.4.0           py3.8_cuda101_cudnn7_0    pytorch
[conda] torchvision               0.5.0                py38_cu101    pytorch

but I still get the same error.

peterjc123 · 2020-02-25T17:07:13Z

Sorry, I could not reproduce this error with torch==1.4.0 and torchvision==0.5.0 with python=3.7 and cudatoolkit=10.1.

Python 3.7.4 (default, Aug  9 2019, 18:34:13) [MSC v.1915 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torchvision
>>> import torch
>>> torch.ops.torchvision.nms
<built-in method nms of PyCapsule object at 0x0000029BF9F74150>

My env:

Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 10 Pro
GCC version: (Rev2, Built by MSYS2 project) 7.1.0
CMake version: version 3.16.4

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 9.0.176
GPU models and configuration: GPU 0: GeForce GTX 1070
Nvidia driver version: 442.19
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v9.0\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.17.3
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] blas                      1.0                         mkl    defaults
[conda] mkl                       2019.4                      245    defaults
[conda] mkl-service               2.3.0            py37hb782905_0    defaults
[conda] mkl_fft                   1.0.14           py37h14836fe_0    defaults
[conda] mkl_random                1.1.0            py37h675688f_0    defaults
[conda] pytorch                   1.4.0           py3.7_cuda101_cudnn7_0    pytorch
[conda] torchfile                 0.1.0                    pypi_0    pypi
[conda] torchvision               0.5.0                py37_cu101    pytorch

davidsteinar · 2020-02-25T17:28:34Z

It seems that when I do

conda create -n torch
conda activate torch
python -V

I get Python 3.6.8

Then when I do

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
python -V

I get Python 3.8.1. This environment produces the RuntimeError: No such operator torchvision::nms

But

if I specifically create a python 3.7 environment

conda create -n torch37 python=3.7
conda activate torch37
python -V

I get Python 3.7.6

Then

conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
python -V

I also get Python 3.7.6. In this environment I get no error

import torch
import torchvision
torch.ops.torchvision.nms
>>> <built-in method nms of PyCapsule object at 0x00000201B0AC9FC0>

So the issue seems to be with conda install pytorch torchvision cudatoolkit=10.1 -c pytorch choosing python 3.8 and causing this issue...

peterjc123 · 2020-02-25T17:31:02Z

You mean it's Python 3.8 only issue, right? I'll check later.

davidsteinar · 2020-02-25T17:33:16Z

Yes it seems so.

amrotork · 2020-02-26T21:33:09Z

I have the same issue. I'll try @davidsteinar solution. I'll remove the environment and create a new one for that.

Thanks @davidsteinar and @peterjc123

peterjc123 · 2020-02-27T03:49:09Z

I tried and it is actually a DLL loading problem for Python 3.8. The workground is to add the directory of cuda before importing torchvision using the following code.

import os
import sys
py_dll_path = os.path.join(sys.exec_prefix, 'Library', 'bin')
os.add_dll_directory(py_dll_path)

Summary: The way it works on the Anaconda distribution of Python 3.8 is a bit different. Loading DLLs explicitly (e.g. `ctype.CDLL`) relies on paths appended by `os.add_dll_directory`. But if you try to load DLLs implicitly (e.g. `from torch._C import *`), it will rely on `PATH`. Fixes pytorch/vision#1916. Pull Request resolved: #33856 Differential Revision: D20150080 Pulled By: soumith fbshipit-source-id: cdbe76c138ea259ef7414c6634d4f7e0b1871af3

Summary: The way it works on the Anaconda distribution of Python 3.8 is a bit different. Loading DLLs explicitly (e.g. `ctype.CDLL`) relies on paths appended by `os.add_dll_directory`. But if you try to load DLLs implicitly (e.g. `from torch._C import *`), it will rely on `PATH`. Fixes pytorch/vision#1916. Pull Request resolved: pytorch#33856 Differential Revision: D20150080 Pulled By: soumith fbshipit-source-id: cdbe76c138ea259ef7414c6634d4f7e0b1871af3

nicolas-gervais · 2020-03-29T18:56:11Z

In Python 3.6 the solution will give the error:

AttributeError: module 'os' has no attribute 'add_dll_directory'

So I solved my issue adding this before importing torch and torchvision:

import os
import sys
py_dll_path = os.path.join(sys.exec_prefix, 'Library', 'bin')
os.environ['PATH'] += py_dll_path

peterjc123 · 2020-03-29T23:35:08Z

@nicolas-gervais This shouldn't be needed for Python 3.6. Could you please tell me something about your env, like how did you install python, pytorch and torchvision?

ckqsars · 2020-04-14T07:56:00Z

I updated my CUDA Toolkit to 10.1 update2 and matching cuDNN to v7.6.5

Collecting environment information...
PyTorch version: 1.4.0
Is debug build: No
CUDA used to build PyTorch: 10.1

OS: Microsoft Windows 10 Pro
GCC version: Could not collect
CMake version: Could not collect

Python version: 3.8
Is CUDA available: Yes
CUDA runtime version: 10.1.243
GPU models and configuration: GPU 0: GeForce GTX 1070
Nvidia driver version: 436.48
cuDNN version: C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\bin\cudnn64_7.dll

Versions of relevant libraries:
[pip3] numpy==1.17.4
[pip3] torch==1.4.0
[pip3] torchvision==0.5.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2020.0                      166
[conda] mkl-service               2.3.0            py38hb782905_0
[conda] mkl_fft                   1.0.15           py38h14836fe_0
[conda] mkl_random                1.1.0            py38hf9181ef_0
[conda] pytorch                   1.4.0           py3.8_cuda101_cudnn7_0    pytorch
[conda] torchvision               0.5.0                py38_cu101    pytorch

but I still get the same error.

How to show the Environment?

peterjc123 · 2020-04-14T09:01:46Z

python -m torch.utils.collect_env

linghu8812 · 2020-04-22T09:22:39Z

Hello, I also encountered this issue with python3.6, here is my environment:
Collecting environment information...
PyTorch version: 1.4.0+cu100
Is debug build: No
CUDA used to build PyTorch: 10.0

OS: Ubuntu 18.04.4 LTS
GCC version: (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
CMake version: Could not collect

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: Could not collect
GPU models and configuration:
GPU 0: Tesla P100-PCIE-16GB

Nvidia driver version: 384.130
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.6.3

Versions of relevant libraries:
[pip3] numpy==1.18.2
[pip3] torch==1.4.0+cu100
[pip3] torchvision==0.5.0
[conda] Could not collect

plumtus · 2020-04-23T14:55:47Z

I encountered the same issue in pytorch 1.5
I installed pytorch from source (tag v1.5.0), torchvison form source(master), python version is 3.6

torch.ops.torchvision.nms
op = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator torchvision::nms

yaw91 · 2020-04-23T15:13:53Z

I also encountered the same error with PyTorch-1.5.0 and torchvision-0.5.0. Below are the details of my environment.

Collecting environment information...
PyTorch version: 1.5.0
Is debug build: No
CUDA used to build PyTorch: 10.2

OS: CentOS Linux release 7.7.1908 (Core)
GCC version: (GCC) 7.3.0
CMake version: version 3.8.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration: GPU 0: Tesla P100-PCIE-12GB
Nvidia driver version: 440.33.01
cuDNN version: Could not collect

Versions of relevant libraries:
[pip3] numpy==1.17.4
[pip3] torch==1.5.0
[pip3] torchvision==0.5.0
[conda] blas 1.0 mkl
[conda] cuda80 1.0 h205658b_0 pytorch
[conda] mkl 2019.4 243
[conda] mkl-service 2.3.0 py36he904b0f_0
[conda] mkl_fft 1.0.1 py36h3010b51_0
[conda] mkl_random 1.0.1 py36h629b387_0
[conda] pytorch 0.4.1 py36_py35_py27__9.0.176_7.1.2_2 pytorch
[conda] torchvision 0.2.1 py_2 pytorch

fmassa · 2020-04-27T11:51:41Z

@yaw91 @plumtus you need to use PyTorch 1.5 together with torchvision 0.6.0, otherwise this won't work.

mthrok · 2020-04-27T16:40:45Z

I am also having the same issue on PyTorch nightly build.

Collecting environment information...
PyTorch version: 1.6.0.dev20200424
Is debug build: No
CUDA used to build PyTorch: 9.2

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.7
Is CUDA available: Yes
CUDA runtime version: 9.2.88
GPU models and configuration:
GPU 0: Quadro GP100
GPU 1: Quadro GP100

Nvidia driver version: 418.116.00
cuDNN version: Could not collect

Versions of relevant libraries:
[pip] numpy==1.18.1
[pip] torch==1.6.0.dev20200424
[pip] torchvision==0.7.0a0+c031e28
[conda] blas                      1.0                         mkl
[conda] cudatoolkit               9.2                           0
[conda] mkl                       2020.0                      166
[conda] mkl-service               2.3.0            py37he904b0f_0
[conda] mkl_fft                   1.0.15           py37ha843d7b_0
[conda] mkl_random                1.1.0            py37hd6b4f25_0
[conda] numpy                     1.18.1           py37h4f9e942_0
[conda] numpy-base                1.18.1           py37hde5b4d6_1
[conda] pytorch                   1.6.0.dev20200424 py3.7_cuda9.2.148_cudnn7.6.3_0    pytorch-nightly
[conda] torchvision               0.7.0a0+c031e28           <pip>

mthrok · 2020-04-27T17:17:01Z

Solved. In my case the error occurred because I used different machines with different GPUs for building torchvision and running training.

Specifying CUDA compute capability such as TORCH_CUDA_ARCH_LIST="6.0;6.1;7.0" when building torchvision resolved this.

ebrahim31 · 2020-06-18T06:26:37Z

Could you explain how do we can Specifying CUDA compute capability such as TORCH_CUDA_ARCH_LIST="6.0;6.1;7.0" when building torchvision resolved this?

fmassa · 2020-07-07T14:47:09Z

@ebrahim31 just pass

TORCH_CUDA_ARCH_LIST="6.0;6.1;7.0" python setup.py install

when compiling torchvision.

yustiks · 2020-10-30T15:43:45Z

I tried and it is actually a DLL loading problem for Python 3.8. The workground is to add the directory of cuda before importing torchvision using the following code.
import os
import sys
py_dll_path = os.path.join(sys.exec_prefix, 'Library', 'bin')
os.add_dll_directory(py_dll_path)

I get another error that add_dll_directory is not an implemented function of os module.

fmassa · 2020-10-30T16:09:21Z

@yustiks add_dll_directory is only present in Python 3.8+.

In torchvision we have a guard for this see

vision/torchvision/extension.py

Lines 27 to 34 in b93d5ee

    
           if sys.version_info >= (3, 8): 
        
               os.add_dll_directory(lib_dir) 
        
           elif with_load_library_flags: 
        
               res = kernel32.AddDllDirectory(lib_dir) 
        
               if res is None: 
        
                   err = ctypes.WinError(ctypes.get_last_error()) 
        
                   err.strerror += f' Error adding "{lib_dir}" to the DLL directories.' 
        
                   raise err

Please open a new issue if you hit an issue with torchvision

xysong1201 · 2021-01-12T05:49:12Z

I also encounter the same issue. In my case, nvidia-smi: CUDA 11.0 and I installed pytorch using torchtoolkit 10.1.
So in order to solve, use this command:
conda install pytorch torchvision torchaudio cudatoolkit=11.0 -c pytorch
Hope this helps

Guemann-ui · 2022-01-29T16:13:34Z

Install Python 3.7 instead of 3.8 will solve the problem

fmassa added bug help wanted topic: binaries windows labels Feb 25, 2020

peterjc123 mentioned this issue Feb 27, 2020

Improve dll loading logic on Windows pytorch/pytorch#33856

Closed

facebook-github-bot closed this as completed in pytorch/pytorch@c18cb1e Feb 28, 2020

dsteinar mentioned this issue Mar 4, 2020

No such operator torchvision::nms #1405

Closed

fmassa mentioned this issue Apr 27, 2020

Improve error message when incompatible torchvision / PyTorch are used #2148

Closed

java-abhinav07 mentioned this issue Jun 20, 2020

non zero depreciated fix #2314

Closed

fmassa mentioned this issue Jul 30, 2020

setup.py dynamically changes the pytorch version dependency. #2519

Closed

Borda mentioned this issue Sep 14, 2020

ref: fix fastrcnn test Lightning-Universe/lightning-bolts#218

Merged

Yongxuanzhang mentioned this issue Mar 11, 2021

object search doesn't work in python 3.8 jina-ai/examples#425

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

RuntimeError: No such operator torchvision::nms #1916

RuntimeError: No such operator torchvision::nms #1916

davidsteinar commented Feb 25, 2020

fmassa commented Feb 25, 2020

davidsteinar commented Feb 25, 2020

peterjc123 commented Feb 25, 2020 •

edited

Loading

davidsteinar commented Feb 25, 2020

peterjc123 commented Feb 25, 2020

davidsteinar commented Feb 25, 2020

amrotork commented Feb 26, 2020 •

edited

Loading

peterjc123 commented Feb 27, 2020

nicolas-gervais commented Mar 29, 2020 •

edited

Loading

peterjc123 commented Mar 29, 2020

ckqsars commented Apr 14, 2020

peterjc123 commented Apr 14, 2020

linghu8812 commented Apr 22, 2020

plumtus commented Apr 23, 2020

yaw91 commented Apr 23, 2020

fmassa commented Apr 27, 2020

mthrok commented Apr 27, 2020

mthrok commented Apr 27, 2020

ebrahim31 commented Jun 18, 2020

fmassa commented Jul 7, 2020

yustiks commented Oct 30, 2020

fmassa commented Oct 30, 2020

xysong1201 commented Jan 12, 2021

Guemann-ui commented Jan 29, 2022 •

edited

Loading

RuntimeError: No such operator torchvision::nms #1916

RuntimeError: No such operator torchvision::nms #1916

Comments

davidsteinar commented Feb 25, 2020

🐛 Bug

To Reproduce

Expected behavior

Environment

fmassa commented Feb 25, 2020

davidsteinar commented Feb 25, 2020

peterjc123 commented Feb 25, 2020 • edited Loading

davidsteinar commented Feb 25, 2020

peterjc123 commented Feb 25, 2020

davidsteinar commented Feb 25, 2020

amrotork commented Feb 26, 2020 • edited Loading

peterjc123 commented Feb 27, 2020

nicolas-gervais commented Mar 29, 2020 • edited Loading

peterjc123 commented Mar 29, 2020

ckqsars commented Apr 14, 2020

peterjc123 commented Apr 14, 2020

linghu8812 commented Apr 22, 2020

plumtus commented Apr 23, 2020

yaw91 commented Apr 23, 2020

fmassa commented Apr 27, 2020

mthrok commented Apr 27, 2020

mthrok commented Apr 27, 2020

ebrahim31 commented Jun 18, 2020

fmassa commented Jul 7, 2020

yustiks commented Oct 30, 2020

fmassa commented Oct 30, 2020

xysong1201 commented Jan 12, 2021

Guemann-ui commented Jan 29, 2022 • edited Loading

peterjc123 commented Feb 25, 2020 •

edited

Loading

amrotork commented Feb 26, 2020 •

edited

Loading

nicolas-gervais commented Mar 29, 2020 •

edited

Loading

Guemann-ui commented Jan 29, 2022 •

edited

Loading