-
Notifications
You must be signed in to change notification settings - Fork 9.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
error in ms_deformable_col2im_cuda: an illegal memory access was encountered #7186
Comments
Experienced the same issue and tested a few environments and GPU models. Fixed by using an earlier version of mmcv-full. 1.4.2 is the latest version of mmcv-full that worked for me.
|
Exactly same error:
I'm on:
P.S.: downgrading mmcv-full to 1.4.2 solved the problem as @PeterVennerstrom mentioned above. |
i met the same question, and in my environment, there is 8 gpus, if i use 0, it will not happened and if i use other gpus, it will occured |
Hello, i met the same question as you. Have you solved this question? |
@xuqingyu26 a workaround is downgrading mmcv-full to 1.4.2 which solved the problem in my case, as mentioned in my comment. |
hi, any update on this? |
It was fixed. Here's a link to the issue with a link to the PR. |
Describe the bug
I'm getting the following error when trying to run deformable_detr
Reproduction
I tried to train the config file below
https://github.com/open-mmlab/mmdetection/blob/7a9bc498d5cc972171ec4f7332afcd70bb50e60e/configs/deformable_detr/deformable_detr_r50_16x2_50e_coco.py
No I did not modify
Environment
sys.platform: linux
Python: 3.8.12 (default, Oct 12 2021, 13:49:34) [GCC 7.5.0]
CUDA available: True
GPU 0: TITAN RTX
CUDA_HOME: /usr/local/cuda-11.0
NVCC: Build cuda_11.0_bu.TC445_37.28845127_0
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.7.0
PyTorch compiling details: PyTorch built with:
TorchVision: 0.8.0
OpenCV: 4.5.5
MMCV: 1.4.4
MMCV Compiler: GCC 7.5
MMCV CUDA Compiler: 11.0
MMDetection: 2.20.0+
torch is installed by pip
Error traceback
Bug fix
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!
The text was updated successfully, but these errors were encountered: