No such operator torchvision::nms #1405

soldierofhell · 2019-10-02T08:45:28Z

Hi,

I'm get an error when I try to use new torch.ops, e.g.

import torch
torch.ops.torchvision.nms

results in

/usr/local/lib/python3.6/dist-packages/torch/_ops.py in __getattr__(self, op_name)
---> 61         op = torch._C._jit_get_operation(qualified_op_name)

RuntimeError: No such operator torchvision::nms`

No matter if I use nightly build or master clone, e.g.
pip install --pre torch torchvision -f https://download.pytorch.org/whl/nightly/cu100/torch_nightly.html

How to debug setup.py to track where comilation/loading fails?

The text was updated successfully, but these errors were encountered:

fmassa · 2019-10-02T11:07:09Z

Hi,

There is currently a problem with the torchvision wheels for the nightlies, they haven't been updated in more than 10 days.

The error you are getting indicates that you haven't compiled the new torchvision extensions.
For that, you should clone the repo, uninstall torchvision, remove any potential pre-built version (via python setup.py clean), and then re-install it with python setup.py install.

This should work I believe.

cc @soumith for the problem pushing nightlies for torchvision

soldierofhell · 2019-10-02T15:36:18Z

@fmassa, unfortunatelly it still doesn't work even on clean simple install path

mkvirtualenv pt1
pip install --pre torch -f https://download.pytorch.org/whl/nightly/cu100/torch_nightly.html
pip install numpy six
git clone https://github.com/pytorch/vision.git
cd vision
python setup.py install

I don't see any signs of errors during compilation process

soldierofhell · 2019-10-02T15:48:05Z

Ok, I see a problem in

vision/torchvision/ops/boxes.py

Line 31 in bf85957

return torch.ops.torchvision.nms(boxes, scores, iou_threshold)

Should there be return torchvision.ops.nms(boxes, scores, iou_threshold)?

soldierofhell · 2019-10-03T07:30:30Z

Sorry, I messed up sematics. torch.ops.torchvision.nms is internal for loaded op and torchvision.ops.nms is external for python wrapper

macimovic · 2019-10-18T07:25:59Z

Hi I am getting the same error. Can you please help me?

soldierofhell · 2019-10-18T08:25:29Z

I must say I didn't managed to fix it (didn't had enough time to dig into torchscript custom ops naming rules). I think there's indeed a problem and I think the tests should fail on this issue. @fmassa as this is observed by other people, please take a look at it

fmassa · 2019-10-18T08:29:20Z

Hi,

Can you print torch.__version__ and torchvision.__version__?

What is a bit confusing now is that torchvision nightly requires torch 1.3.0, and NOT a torch nightly, so this might be the reason

fmassa · 2019-10-18T08:33:34Z

As I've mentioned in #1489, I'm going to be changing the torchvision nightlies to depend on PyTorch nightly.

macimovic · 2019-10-18T08:37:29Z

torch: 1.4.0.dev20191016+cu100
torchvision: 0.5.0.dev20191017+cu100
I am using CUDA 10.0

soldierofhell · 2019-10-18T08:38:02Z

I'll check 1.3.0, because I used torch nightly, but I reported this error before 1.3.0 release

macimovic · 2019-10-18T08:48:33Z

Hi, I managed to find a compatible (working) pair of torchvision and torch.
torch: 1.3.0
torchvision: 0.4.1

fmassa · 2019-10-18T08:50:35Z

The error indicates that there is an uncompatibility between the version of PyTorch that was used to build torchvision and the version of PyTorch that you are using (it's not clear from the message unfortunately)

I'm fixing things now so that torchvision nightly depend on PyTorch nightly, and I'll regenerate the binaries.

fmassa · 2019-10-18T08:55:50Z

Should be fixed with #1492

fmassa · 2019-10-21T10:56:23Z

Fixed via #1492

davidsteinar · 2020-02-18T15:13:56Z

Hi @fmassa , I am facing this issue on Windows using the latest stable build.

This is how I installed:

conda create -n pytorch
conda activate pytorch
conda install pytorch torchvision cudatoolkit=10.1 -c pytorch
conda list

Name	Version	Build	Channel
blas	1.0	mkl
ca-certificates	2020.1.1	0
certifi	2019.11.28	py38_0
cudatoolkit	10.1.243	h74a9793_0
freetype	2.9.1	ha9979f8_1
icc_rt	2019.0.0	h0cc432a_1
intel-openmp	2020.0	166
jpeg	9b	hb83a4c4_2
libpng	1.6.37	h2a8f88b_0
libtiff	4.1.0	h56a325e_0
mkl	2020.0	166
mkl-service	2.3.0	py38hb782905_0
mkl_fft	1.0.15	py38h14836fe_0
mkl_random	1.1.0	py38hf9181ef_0
ninja	1.9.0	py38h74a9793_0
numpy	1.18.1	py38h93ca92e_0
numpy-base	1.18.1	py38hc3f5095_1
olefile	0.46	py_0
openssl	1.1.1d	he774522_4
pillow	7.0.0	py38hcc1f983_0
pip	20.0.2	py38_1
python	3.8.1	h5fd99cc_1
pytorch	1.4.0	py3.8_cuda101_cudnn7_0	pytorch
setuptools	45.2.0	py38_0
six	1.14.0	py38_0
sqlite	3.31.1	he774522_0
tk	8.6.8	hfa6e2cd_0
torchvision	0.5.0	py38_cu101	pytorch
vc	14.1	h0510ff6_4
vs2015_runtime	14.16.27012	hf0eaf9b_1
wheel	0.34.2	py38_0
wincertstore	0.2	py38_0
xz	5.2.4	h2fa13f4_4
zlib	1.2.11	h62dcd97_3
zstd	1.3.7	h508b16e_0

nvidia-smi
Tue Feb 18 15:08:03 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 436.48       Driver Version: 436.48       CUDA Version: 10.1     |
|-------------------------------+----------------------+----------------------+

And here is the error message:

Python 3.8.1 (default, Jan  8 2020, 15:55:49) [MSC v.1916 64 bit (AMD64)] :: Anaconda, Inc. on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
>>> torch.ops.torchvision.nms
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\david\Miniconda3\envs\pytorch\lib\site-packages\torch\_ops.py", line 61, in __getattr__
    op = torch._C._jit_get_operation(qualified_op_name)
RuntimeError: No such operator torchvision::nms
>>>

What can I do ?

cpuhrsch · 2020-02-24T19:50:22Z

@davidsteinar - if you're still encountering this please create a new issue to gain visibility

FrancescoSaverioZuppichini · 2020-03-04T14:41:02Z

same problem here

dsteinar · 2020-03-04T14:43:13Z

@FrancescoSaverioZuppichini see #1916

FrancescoSaverioZuppichini · 2020-03-04T16:53:10Z

Oh thank you, fixed :)

LiRWZ · 2020-04-16T01:27:55Z

I met the same erro:No such operator torchvision::nms.So I change the version of pytoch and torchvision to 1.2.0 and 0.4.0,but the erro is still here.
update(2020/4/21)
I hava solved this issue by changing the version of numpy from 1.18.1 to 1.17.0.

chiboreache · 2020-07-29T12:22:10Z

same on ArchLinux np is 1.19

torch.version: 1.6.0rc7
torchvision.version: 0.6.0a0

fmassa · 2020-07-30T12:19:23Z

@chiboreache PyTorch 1.6.0 requires torchvision 0.7.0

chiboreache · 2020-07-30T13:29:15Z

@chiboreache PyTorch 1.6.0 requires torchvision 0.7.0

that really strange

in aur we only has 0.6.1-1

chiboreache · 2020-07-30T13:50:23Z

@fmassa

git version actually was 0.8.0a0+f5bf9d5

it finally works again, thank you!

but i getting warning

/usr/lib/python3.8/site-packages/detectron2/layers/wrappers.py:226: UserWarning: This overload of nonzero is deprecated:
        nonzero()
Consider using one of the following signatures instead:
        nonzero(*, bool as_tuple) (Triggered internally at  ../torch/csrc/utils/python_arg_parser.cpp:766.)
  return x.nonzero().unbind(1)

is it torch or detektron2 issue?

fmassa · 2020-07-30T15:25:33Z

this warning is expected, and was introduced in the latest release of PyTorch. detectron2 should update its implementation to use torch.nonzero(x, as_tuple=False).unbind(1)

GotFusion · 2020-09-13T11:37:32Z

@chiboreache PyTorch 1.6.0 requires torchvision 0.7
The problem didn't fixed even though I met the requirement

fmassa · 2020-09-14T12:41:37Z

@GotFusion I believe the issue is still incompatibility of the versions -- you might be using a different python environment for running the code than the one you used for checking the versions

GotFusion · 2020-09-14T12:45:39Z

@GotFusion I believe the issue is still incompatibility of the versions -- you might be using a different python environment for running the code than the one you used for checking the versions

Thank you for your advice. I have changed the python version to 3.7.9, and it works now

yustiks · 2020-10-30T15:42:55Z

torchvision 0.7.0

yes, I changed to 0.7.0, and it worked

feiyuhuahuo · 2020-11-20T02:43:38Z

Hey guys, I meet the same problem: RuntimeError: No such operator torchvision::nms with torch1.6 with torchvision0.7, and torch1.7 with torchvision0.8.1.
Then I modifty the code to such a way and it works:
from
torch.ops.torchvision.nms(boxes, scores, iou_thres)
to
import torchvision
torchvision.ops.nms(boxes, scores, iou_thres)
Hope this is helpful.

fmassa · 2020-11-20T09:16:11Z

@feiyuhuahuo can you share where in the codebase / tutorials the torch.ops.torchvision.nms(boxes, scores, iou_thres version is used? The only version that we encourage users to use is torchvision.ops.nms(boxes, scores, iou_thres)

feiyuhuahuo · 2020-11-20T09:33:12Z

@fmassa I get the code from the official yolov5 project.
https://github.com/ultralytics/yolov5/blob/199c9c787427ece5723d5309e1c7c524a99bc59d/utils/general.py#L326

fmassa · 2020-11-20T09:39:33Z

@feiyuhuahuo they should change their code, as this is not supported.

For torch.ops.torchvision to be available, we first need to import torchvision

fmassa · 2020-11-20T09:41:51Z

@feiyuhuahuo I've commented in ultralytics/yolov5#1272 (comment)

GraceKafuu · 2020-12-01T03:32:12Z

@chiboreache PyTorch 1.6.0 requires torchvision 0.7.0

Thanks!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

vasyllyashkevych · 2021-01-01T15:54:00Z

I did like was noticed here but the new issues appeared later:
if expr.id == "True":
RecursionError: maximum recursion depth exceeded in comparison

Also, tried to downgrade version like:
pip3 install torch==1.4.0 torchvision==0.5.0 -f https://download.pytorch.org/whl/torch_stable.html -i https://pypi.tuna.tsinghua.edu.cn/simple

Alas, unsuccessfully

KastanDay · 2021-07-14T23:32:33Z

Edit: I created a new issue with a detailed report: #4180

Same problem here. The primary issue is resolve with a simple naming change (below, thanks to @feiyuhuahuo) but this creates the RecursionError: maximum recursion depth exceeded in comparison issue referenced by @vasyllyashkevych.

# This fix creates RecursionError
File: local/lib/python3.6/dist-packages/torchvision-0.7.0a0+78ed10c-py3.6-linux-aarch64.egg/torchvision/ops/boxes.py

OLD (bad):
torch.ops.torchvision.nms(boxes, scores, iou_thres)
NEW (better):
import torchvision # top of file
torchvision.ops.nms(boxes, scores, iou_thres)

Did anyone manage to resolve the maximum recursion depth error?

System details:

Host machine: Nvidia Jetson Xaiver (arm architecture, not x64)
Python: python3.6
Detectron2: installed from source on Github (July 14, 2021)
torch version: 1.9.0
torchvision version: 0.7.0 (a0)
Cuda version: 10.2

Full stack trace:

$ sudo python3 demo.py --config-file ../configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml \
  --video-input IMG_3578.MOV \
  --opts MODEL.WEIGHTS detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl
  
[07/14 17:23:49 detectron2]: Arguments: Namespace(confidence_threshold=0.5, config_file='../configs/COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml', input=None, opts=['MODEL.WEIGHTS', 'detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl'], output=None, video_input='IMG_3578.MOV', webcam=False)
[07/14 17:24:00 fvcore.common.checkpoint]: [Checkpointer] Loading from detectron2://COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x/137849600/model_final_f10217.pkl ...
[07/14 17:24:01 fvcore.common.checkpoint]: Reading a file from 'Detectron2 Model Zoo'
WARNING [07/14 17:24:01 fvcore.common.checkpoint]: The checkpoint state_dict contains keys that are not used by the model:
  proposal_generator.anchor_generator.cell_anchors.{0, 1, 2, 3, 4}
  0%|                                                                                   | 0/221 [00:04<?, ?it/s]
Traceback (most recent call last):
  File "demo.py", line 176, in <module>
    for vis_frame in tqdm.tqdm(demo.run_on_video(video), total=num_frames):
  File "/usr/local/lib/python3.6/dist-packages/tqdm/std.py", line 1185, in __iter__
    for obj in iterable:
  File "/home/zion/detectron2/demo/predictor.py", line 129, in run_on_video
    yield process_predictions(frame, self.predictor(frame))
  File "/usr/local/lib/python3.6/dist-packages/detectron2/engine/defaults.py", line 320, in __call__
    predictions = self.model([inputs])[0]
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/meta_arch/rcnn.py", line 146, in forward
    return self.inference(batched_inputs)
  File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/meta_arch/rcnn.py", line 204, in inference
    proposals, _ = self.proposal_generator(images, features, None)
  File "/usr/local/lib/python3.6/dist-packages/torch/nn/modules/module.py", line 889, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/proposal_generator/rpn.py", line 478, in forward
    anchors, pred_objectness_logits, pred_anchor_deltas, images.image_sizes
  File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/proposal_generator/rpn.py", line 511, in predict_proposals
    self.training,
  File "/usr/local/lib/python3.6/dist-packages/detectron2/modeling/proposal_generator/proposal_utils.py", line 116, in find_top_rpn_proposals
    keep = batched_nms(boxes.tensor, scores_per_img, lvl, nms_thresh)
  File "/usr/local/lib/python3.6/dist-packages/detectron2/layers/nms.py", line 21, in batched_nms
    return box_ops.batched_nms(boxes.float(), scores, idxs, iou_threshold)
  File "/usr/local/lib/python3.6/dist-packages/torch/jit/_trace.py", line 1091, in wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/torchvision-0.7.0a0+78ed10c-py3.6-linux-aarch64.egg/torchvision/ops/boxes.py", line 85, in batched_nms
    keep = nms(boxes_for_nms, scores, iou_threshold)
  File "/usr/local/lib/python3.6/dist-packages/torchvision-0.7.0a0+78ed10c-py3.6-linux-aarch64.egg/torchvision/ops/boxes.py", line 43, in nms
    return torchvision.ops.nms(boxes, scores, iou_threshold)
  File "/usr/local/lib/python3.6/dist-packages/torchvision-0.7.0a0+78ed10c-py3.6-linux-aarch64.egg/torchvision/ops/boxes.py", line 43, in nms
    return torchvision.ops.nms(boxes, scores, iou_threshold)
  File "/usr/local/lib/python3.6/dist-packages/torchvision-0.7.0a0+78ed10c-py3.6-linux-aarch64.egg/torchvision/ops/boxes.py", line 43, in nms
    return torchvision.ops.nms(boxes, scores, iou_threshold)
  [Previous line repeated 969 more times]
RecursionError: maximum recursion depth exceeded

fmassa added the topic: binaries label Oct 2, 2019

soldierofhell closed this as completed Oct 3, 2019

ppwwyyxx mentioned this issue Oct 17, 2019

RuntimeError: No such operator torchvision::nms #1489

Closed

soldierofhell reopened this Oct 18, 2019

fmassa mentioned this issue Oct 18, 2019

Make CI use PyTorch nightly for CI and nightlies #1492

Merged

fmassa closed this as completed Oct 21, 2019

chengjiun mentioned this issue Jul 15, 2020

YoloV5 RuntimeError: No such operator torchvision:nms ultralytics/yolov5#352

Closed

burhanmudassar mentioned this issue Jul 21, 2020

RuntimeError: No such operator torchvision::deform_conv2d #2498

Closed

fmassa mentioned this issue Nov 20, 2020

RuntimeError: No such operator torchvision::nms ultralytics/yolov5#1272

Closed

rokm mentioned this issue Nov 23, 2020

[5939] Failed to execute script train pyinstaller/pyinstaller#5331

Closed

MessikommerNico mentioned this issue Mar 25, 2021

Error while running Object Detection uzh-rpg/rpg_asynet#24

Closed

KastanDay mentioned this issue Jul 14, 2021

[Detectron2] RuntimeError: No such operator torchvision::nms and RecursionError: maximum recursion depth exceeded #4180

Closed

ydaiming mentioned this issue Sep 28, 2021

Issue with torchvision::nms using custom Pytorch and TorchVision aws/sagemaker-pytorch-training-toolkit#181

Closed

No such operator torchvision::nms #1405

No such operator torchvision::nms #1405

Comments

soldierofhell commented Oct 2, 2019

fmassa commented Oct 2, 2019

soldierofhell commented Oct 2, 2019

soldierofhell commented Oct 2, 2019

soldierofhell commented Oct 3, 2019

macimovic commented Oct 18, 2019

soldierofhell commented Oct 18, 2019

fmassa commented Oct 18, 2019

fmassa commented Oct 18, 2019

macimovic commented Oct 18, 2019

soldierofhell commented Oct 18, 2019

macimovic commented Oct 18, 2019

fmassa commented Oct 18, 2019

fmassa commented Oct 18, 2019

fmassa commented Oct 21, 2019

davidsteinar commented Feb 18, 2020

cpuhrsch commented Feb 24, 2020

FrancescoSaverioZuppichini commented Mar 4, 2020

dsteinar commented Mar 4, 2020

FrancescoSaverioZuppichini commented Mar 4, 2020

LiRWZ commented Apr 16, 2020 • edited Loading

chiboreache commented Jul 29, 2020

fmassa commented Jul 30, 2020

chiboreache commented Jul 30, 2020

chiboreache commented Jul 30, 2020

fmassa commented Jul 30, 2020

GotFusion commented Sep 13, 2020

fmassa commented Sep 14, 2020

GotFusion commented Sep 14, 2020 • edited Loading

yustiks commented Oct 30, 2020

feiyuhuahuo commented Nov 20, 2020

fmassa commented Nov 20, 2020

feiyuhuahuo commented Nov 20, 2020

fmassa commented Nov 20, 2020

fmassa commented Nov 20, 2020

GraceKafuu commented Dec 1, 2020

vasyllyashkevych commented Jan 1, 2021 • edited Loading

KastanDay commented Jul 14, 2021 • edited Loading

Did anyone manage to resolve the maximum recursion depth error?

LiRWZ commented Apr 16, 2020 •

edited

Loading

GotFusion commented Sep 14, 2020 •

edited

Loading

vasyllyashkevych commented Jan 1, 2021 •

edited

Loading

KastanDay commented Jul 14, 2021 •

edited

Loading