Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RuntimeError: nms is not compiled with GPU support #4075

Closed
tianxianhao opened this issue Nov 6, 2020 · 4 comments
Closed

RuntimeError: nms is not compiled with GPU support #4075

tianxianhao opened this issue Nov 6, 2020 · 4 comments
Assignees

Comments

@tianxianhao
Copy link

Thanks for your error report and we appreciate it a lot.

Checklist

  1. I have searched related issues but cannot get the expected help.
  2. The bug has not been fixed in the latest version.

Describe the bug
RuntimeError: nms is not compiled with GPU support

Reproduction

  1. What command or script did you run?
    import torch
    import torchvision
    import os
    print(torch.version,torch.cuda.is_available())
    os.environ['CUDA_VISIBLE_DEVICES']="4,5"
    import mmdet
    print(mmdet.version)

from mmcv.ops import get_compiling_cuda_version, get_compiler_version
print(get_compiler_version())
print(get_compiling_cuda_version())

from mmdet.apis import init_detector, inference_detector
root_path='/home/tianxianhao/Project/mmdetection/'
config_file = root_path+'configs/faster_rcnn/faster_rcnn_r50_fpn_1x_coco.py'
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

init a detector

model = init_detector(config_file, device=device)

inference the demo image

inference_detector(model, root_path+'demo/demo.jpg')
A placeholder for the command.

2. Did you make any modifications on the code or config? Did you understand what you have modified?
3. What dataset did you use?

**Environment**

1. Please run `python mmdet/utils/collect_env.py` to collect necessary environment information and paste it here.
sys.platform: linux
Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0]
CUDA available: True
GPU 0,1,2,3,4,5,6,7: Tesla P100-PCIE-16GB
CUDA_HOME: /home/tianxianhao/cuda/cuda-10.1
NVCC: Cuda compilation tools, release 10.1, V10.1.105
GCC: gcc (GCC) 6.1.0
PyTorch: 1.7.0
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 10.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_37,code=compute_37
  - CuDNN 7.6.3
  - Magma 2.5.2
  - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, 

TorchVision: 0.8.1
OpenCV: 4.4.0
MMCV: 1.1.6
MMCV Compiler: GCC 6.1
MMCV CUDA Compiler: not available
MMDetection: 2.6.0+d3cf38d

2. You may add addition that may be helpful for locating the problem, such as
    - How you installed PyTorch [e.g., pip, conda, source]
    conda install --use-local
    - Other environment variables that may be related (such as `$PATH`, `$LD_LIBRARY_PATH`, `$PYTHONPATH`, etc.)
     no such problem in environment variable setting
**Error traceback**
If applicable, paste the error trackback here.

Python 3.7.9 (default, Aug 31 2020, 12:42:55)
Type "copyright", "credits" or "license" for more information.

IPython 7.19.0 -- An enhanced Interactive Python.

runfile('/home/tianxianhao/.config/spyder-py3/temp.py', wdir='/home/tianxianhao/.config/spyder-py3')
1.7.0 True
2.6.0
GCC 6.1
not available
Traceback (most recent call last):

File "/home/tianxianhao/.config/spyder-py3/temp.py", line 26, in
inference_detector(model, root_path+'demo/demo.jpg')

File "/home/tianxianhao/Project/mmdetection/mmdet/apis/inference.py", line 119, in inference_detector
result = model(return_loss=False, rescale=True, **data)[0]

File "/home/tianxianhao/anaconda3/envs/mmdt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 727, in _call_impl
result = self.forward(*input, **kwargs)

File "/home/tianxianhao/anaconda3/envs/mmdt/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 84, in new_func
return old_func(*args, **kwargs)

File "/home/tianxianhao/Project/mmdetection/mmdet/models/detectors/base.py", line 170, in forward
return self.forward_test(img, img_metas, **kwargs)

File "/home/tianxianhao/Project/mmdetection/mmdet/models/detectors/base.py", line 147, in forward_test
return self.simple_test(imgs[0], img_metas[0], **kwargs)

File "/home/tianxianhao/Project/mmdetection/mmdet/models/detectors/two_stage.py", line 194, in simple_test
proposal_list = self.rpn_head.simple_test_rpn(x, img_metas)

File "/home/tianxianhao/Project/mmdetection/mmdet/models/dense_heads/rpn_test_mixin.py", line 36, in simple_test_rpn
proposal_list = self.get_bboxes(*rpn_outs, img_metas)

File "/home/tianxianhao/anaconda3/envs/mmdt/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 164, in new_func
return old_func(*args, **kwargs)

File "/home/tianxianhao/Project/mmdetection/mmdet/models/dense_heads/anchor_head.py", line 570, in get_bboxes
scale_factor, cfg, rescale)

File "/home/tianxianhao/Project/mmdetection/mmdet/models/dense_heads/rpn_head.py", line 167, in _get_bboxes_single
dets, keep = batched_nms(proposals, scores, ids, nms_cfg)

File "/home/tianxianhao/anaconda3/envs/mmdt/lib/python3.7/site-packages/mmcv/ops/nms.py", line 259, in batched_nms
dets, keep = nms_op(boxes_for_nms, scores, **nms_cfg_)

File "/home/tianxianhao/anaconda3/envs/mmdt/lib/python3.7/site-packages/mmcv/utils/misc.py", line 310, in new_func
output = old_func(*args, **kwargs)

File "/home/tianxianhao/anaconda3/envs/mmdt/lib/python3.7/site-packages/mmcv/ops/nms.py", line 113, in nms
inds = NMSop.apply(boxes, scores, iou_threshold, offset)

File "/home/tianxianhao/anaconda3/envs/mmdt/lib/python3.7/site-packages/mmcv/ops/nms.py", line 18, in forward
bboxes, scores, iou_threshold=float(iou_threshold), offset=offset)

RuntimeError: nms is not compiled with GPU support
A placeholder for trackback.


**Bug fix**
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!
@tianxianhao
Copy link
Author

Do not install the pytorch and torchvision in local pkgs way by conda!

@frederikschubert
Copy link

I also run into this error. I installed pytorch, torchvision and mmcv-full using the recommended way:

pip install torch==1.7.0+cu110 torchvision==0.8.1+cu110 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
pip install -f https://download.openmmlab.com/mmcv/dist/index.html mmcv-full==1.2.0+torch1.7.0+cu110

Environment

sys.platform: linux
Python: 3.8.0 (default, Nov  6 2019, 21:49:08) [GCC 7.3.0]
CUDA available: True
GPU 0: GeForce RTX 2080 Ti
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.0_bu.TC445_37.28845127_0
GCC: gcc (SUSE Linux) 7.5.0
PyTorch: 1.7.0+cu110
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.0
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80
  - CuDNN 8.0.4
  - Magma 2.5.2
  - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.8.1+cu110
OpenCV: 4.4.0
MMCV: 1.2.0
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: not available
MMDetection: 2.6.0+b03ff9f

@tianxianhao
Copy link
Author

I also run into this error. I installed pytorch, torchvision and mmcv-full using the recommended way:

pip install torch==1.7.0+cu110 torchvision==0.8.1+cu110 torchaudio===0.7.0 -f https://download.pytorch.org/whl/torch_stable.html
pip install -f https://download.openmmlab.com/mmcv/dist/index.html mmcv-full==1.2.0+torch1.7.0+cu110

Environment

sys.platform: linux
Python: 3.8.0 (default, Nov  6 2019, 21:49:08) [GCC 7.3.0]
CUDA available: True
GPU 0: GeForce RTX 2080 Ti
CUDA_HOME: /usr/local/cuda
NVCC: Build cuda_11.0_bu.TC445_37.28845127_0
GCC: gcc (SUSE Linux) 7.5.0
PyTorch: 1.7.0+cu110
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.6.0 (Git Hash 5ef631a030a6f73131c77892041042805a06064f)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 11.0
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80
  - CuDNN 8.0.4
  - Magma 2.5.2
  - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON,

TorchVision: 0.8.1+cu110
OpenCV: 4.4.0
MMCV: 1.2.0
MMCV Compiler: GCC 7.3
MMCV CUDA Compiler: not available
MMDetection: 2.6.0+b03ff9f

hello, I sovle this problem by installing pytorch1.6.0 and corresponding mmcv-full version. Maybe the mmcv not support pytorch1.7.0 yet.

@frederikschubert
Copy link

They should support pytorch 1.7.0 since the 1.2.0 release, but it looks like you are right.

I got my setup running after installing pytorch 1.6.0 with CUDA 10.1:

pip install torch==1.6.0+cu101 torchvision==0.7.0+cu101 -f https://download.pytorch.org/whl/torch_stable.html
pip install -f https://download.openmmlab.com/mmcv/dist/index.html mmcv-full==1.2.0+torch1.6.0+cu101

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants