ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 512, 1, 1]).

[jajajajaja121]

Thanks for your error report and we appreciate it a lot.

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.

Describe the bug
When I train with deeplabplus， no matter how I change my input size, there is always an error ad below, I have tried different version of pytorch ,different version of mmsegmentation, and my data is made by my self as the format of Cityscape, I don't know what is wrong, please help me, thanks a lot.

Reproduction

1. What command or script did you run?

   python tools/train.py configs/deeplabv3plus/deeplabv3plus_r101-d8_769x769_80k_cityscapes.py --gpus 1
   

I have change the data root in cityscape.py and I change syncBN to BN in configs/model/deeplabv3_r50-d8.py, and my dataset is made by myself, and the format is same as cityscape, int the left8bintimage is my train image, and in gtfine is my label which has one channel and label every pixel in 1-19 here is one of my label:

and if you map the label in different color it will look like this:

Environment

sys.platform: linux
Python: 3.7.9 (default, Aug 31 2020, 12:42:55) [GCC 7.3.0]
CUDA available: True
GPU 0,1,2: GeForce RTX 2080 Ti
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 10.1, V10.1.105
GCC: gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0
PyTorch: 1.6.0+cu101
PyTorch compiling details: PyTorch built with:
  - GCC 7.3
  - C++ Version: 201402
  - Intel(R) Math Kernel Library Version 2020.0.2 Product Build 20200624 for Intel(R) 64 architecture applications
  - Intel(R) MKL-DNN v1.5.0 (Git Hash e2ac1fac44c5078ca927cb9b90e1b3066a0b2ed0)
  - OpenMP 201511 (a.k.a. OpenMP 4.5)
  - NNPACK is enabled
  - CPU capability usage: AVX2
  - CUDA Runtime 10.1
  - NVCC architecture flags: -gencode;arch=compute_37,code=sm_37;-gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75
  - CuDNN 7.6.3
  - Magma 2.5.2
  - Build settings: BLAS=MKL, BUILD_TYPE=Release, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DUSE_VULKAN_WRAPPER -O2 -fPIC -Wno-narrowing -Wall -Wextra -Werror=return-type -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-sign-compare -Wno-unused-parameter -Wno-unused-variable -Wno-unused-function -Wno-unused-result -Wno-unused-local-typedefs -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, USE_CUDA=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_STATIC_DISPATCH=OFF, 

TorchVision: 0.7.0
OpenCV: 4.4.0
MMCV: 1.2.0
MMCV Compiler: GCC 7.5
MMCV CUDA Compiler: not available
MMSegmentation: 0.8.0+15cca88

Error traceback

Traceback (most recent call last):
  File "tools/train.py", line 164, in <module>
    main()
  File "tools/train.py", line 160, in main
    meta=meta)
  File "/home/xiongpan/code/mmsegmentation-0.8/mmseg/apis/train.py", line 118, in train_segmentor
    runner.run(data_loaders, cfg.workflow)
  File "/opt/conda/envs/mms/lib/python3.7/site-packages/mmcv/runner/iter_based_runner.py", line 130, in run
    iter_runner(iter_loaders[i], **kwargs)
  File "/opt/conda/envs/mms/lib/python3.7/site-packages/mmcv/runner/iter_based_runner.py", line 60, in train
    outputs = self.model.train_step(data_batch, self.optimizer, **kwargs)
  File "/opt/conda/envs/mms/lib/python3.7/site-packages/mmcv/parallel/data_parallel.py", line 67, in train_step
    return self.module.train_step(*inputs[0], **kwargs[0])
  File "/home/xiongpan/code/mmsegmentation-0.8/mmseg/models/segmentors/base.py", line 156, in train_step
    losses = self(**data_batch)
  File "/opt/conda/envs/mms/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/envs/mms/lib/python3.7/site-packages/mmcv/runner/fp16_utils.py", line 84, in new_func
    return old_func(*args, **kwargs)
  File "/home/xiongpan/code/mmsegmentation-0.8/mmseg/models/segmentors/base.py", line 122, in forward
    return self.forward_train(img, img_metas, **kwargs)
  File "/home/xiongpan/code/mmsegmentation-0.8/mmseg/models/segmentors/encoder_decoder.py", line 160, in forward_train
    gt_semantic_seg)
  File "/home/xiongpan/code/mmsegmentation-0.8/mmseg/models/segmentors/encoder_decoder.py", line 104, in _decode_head_forward_train
    self.train_cfg)
  File "/home/xiongpan/code/mmsegmentation-0.8/mmseg/models/decode_heads/decode_head.py", line 188, in forward_train
    seg_logits = self.forward(inputs)
  File "/home/xiongpan/code/mmsegmentation-0.8/mmseg/models/decode_heads/sep_aspp_head.py", line 83, in forward
    self.image_pool(x),
  File "/opt/conda/envs/mms/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/envs/mms/lib/python3.7/site-packages/torch/nn/modules/container.py", line 117, in forward
    input = module(input)
  File "/opt/conda/envs/mms/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/envs/mms/lib/python3.7/site-packages/mmcv/cnn/bricks/conv_module.py", line 187, in forward
    x = self.norm(x)
  File "/opt/conda/envs/mms/lib/python3.7/site-packages/torch/nn/modules/module.py", line 722, in _call_impl
    result = self.forward(*input, **kwargs)
  File "/opt/conda/envs/mms/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py", line 136, in forward
    self.weight, self.bias, bn_training, exponential_average_factor, self.eps)
  File "/opt/conda/envs/mms/lib/python3.7/site-packages/torch/nn/functional.py", line 2012, in batch_norm
    _verify_batch_size(input.size())
  File "/opt/conda/envs/mms/lib/python3.7/site-packages/torch/nn/functional.py", line 1995, in _verify_batch_size
    raise ValueError('Expected more than 1 value per channel when training, got input size {}'.format(size))
ValueError: Expected more than 1 value per channel when training, got input size torch.Size([1, 512, 1, 1])

[xvjiarui]

Hi @jajajajaja121
You should set batch size > 1 by setting data.samples_per_gpu=2.

[YtengTan]

Same, did you fix it?

[androbaza]

@YtengTan Hey, just encountered this too. have you been able to overcome this?

[farouk09]

@YtengTan @androbaza @jajajajaja121 Hey, just encountered the same problem, any ideas on how to solve this problem?