Evaluation on coco is not consistent when batch size is not 1

[Cold-Winter]

If you do not know the root cause of the problem / bug, and wish someone to help you, please
include:

To Reproduce

1. detectron2/data/build.py in build_detection_test_loader function: change from
   'batch_sampler = torch.utils.data.sampler.BatchSampler(sampler, 1, drop_last=False)' to 'batch_sampler = torch.utils.data.sampler.BatchSampler(sampler, 4, drop_last=False)'
2. Command line:
   'CUDA_VISIBLE_DEVICES=0,1,2,3 python tools/plain_train_net.py --num-gpus=4 --config-file configs/COCO-Detection/faster_rcnn_R_101_FPN_3x.yaml --eval-only --dist-url='tcp://127.0.0.1:62222' MODEL.WEIGHTS checkpoints/model_final_f6e8b1.pkl'
3. The box ap is not consistent.
   Before change: 'copypaste: 42.0358,62.4804,45.8763,25.2248,45.5545,54.5923'
   After change: 'copypaste: 42.0413,62.4595,45.9625,25.2514,45.5270,54.5579'

Expected behavior

If there are no obvious error in "what you observed" provided above,
please tell us the expected behavior.

I am wondering the batch_norm layer may have some problems.

I expect the evaluation results from same model on 'coco' with different batchsize should be same.

Environment

---

sys.platform linux
Python 3.7.4 (default, Aug 13 2019, 20:35:49) [GCC 7.3.0]
Numpy 1.17.2
Detectron2 Compiler GCC 6.3
Detectron2 CUDA Compiler 10.0
DETECTRON2_ENV_MODULE
PyTorch 1.3.0+cu100
PyTorch Debug Build False
torchvision 0.4.1+cu100
CUDA available True
GPU 0,1,2,3,4,5,6,7 Tesla V100-SXM2-16GB
CUDA_HOME /usr/local/cuda
NVCC Cuda compilation tools, release 10.0, V10.0.130
Pillow 6.2.0
cv2 4.1.1

---

PyTorch built with:

• GCC 7.3
• Intel(R) Math Kernel Library Version 2019.0.4 Product Build 20190411 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v0.20.5 (Git Hash 0125f28c61c1f822fd48570b4c1066f96fcb9b2e)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• NNPACK is enabled
• CUDA Runtime 10.0
• NVCC architecture flags: -gencode;arch=compute_35,code=sm_35;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_61,code=sm_61;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_50,code=compute_50
• CuDNN 7.6.3
• Magma 2.5.1
• Build settings: BLAS=MKL, BUILD_NAMEDTENSOR=OFF, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -fopenmp -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -O2 -fPIC -Wno-narrowing -Wall -Wextra -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Wno-stringop-overflow, DISABLE_NUMA=1, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=True, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF,

[ppwwyyxx]

This is expected as mentioned in #33 because it adds padding to images. That's why we use batch size of 1.