[bug]When h and w are not divisible by 4, Correlation calculations are unreliable

[wangjiangben-hw]

Thanks for reporting the unexpected results and we appreciate it a lot.

Checklist

1. I have searched related issues but cannot get the expected help.
2. I have read the FAQ documentation but cannot get the expected help.
3. The unexpected results still exist in the latest version.

Describe the Issue
When h and w are not divisible by 4, Correlation calculations are unreliable

Reproduction

1. What command, code, or script did you run?

import torch
from mmcv.ops import Correlation

shape = (1,32,11,11)
idx=0

while True:
    torch.manual_seed(0)
    feat1 = torch.randn(*shape)
    feat2 = torch.randn(*shape)
    layer = Correlation(max_displacement=1, padding=0)
    out_cuda1 = layer(feat1.cuda(), feat2.cuda())

    torch.manual_seed(0)
    feat1 = torch.randn(*shape)
    feat2 = torch.randn(*shape)
    layer = Correlation(max_displacement=1, padding=0)
    out_cuda2 = layer(feat1.cuda(), feat2.cuda())

    assert ((out_cuda1 - out_cuda2).abs() > 1e-5).sum() == 0
    idx += 1
    print(idx)

2. Did you make any modifications on the code? Did you understand what you have modified?

**cuda:11.1, torch:1.8.1+cu111, MMCV:1.6.1, **

1. Please run python -c "from mmcv.utils import collect_env; print(collect_env())" to collect necessary environment information and paste it here.
2. You may add addition that may be helpful for locating the problem, such as
   • How you installed PyTorch [e.g., pip, conda, source]
   • Other environment variables that may be related (such as $PATH, $LD_LIBRARY_PATH, $PYTHONPATH, etc.)

Error traceback
If applicable, paste the error traceback here.

AseertionError

No errors are reported when both h and w in the shape can be divisible by 4.

Bug fix
If you have already identified the reason, you can provide the information here. If you are willing to create a PR to fix it, please also leave a comment here and that would be much appreciated!