[Bug] Error while evaluating TPVFormer

[bqm1111]

Prerequisite

• I have searched Issues and Discussions but cannot get the expected help.
• I have read the FAQ documentation but cannot get the expected help.
• The bug has not been fixed in the latest version (dev-1.x) or latest version (dev-1.0).

Task

I'm using the official example scripts/configs for the officially supported tasks/models/datasets.

Branch

main branch https://github.com/open-mmlab/mmdetection3d

Environment

sys.platform: linux
Python: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0]
CUDA available: True
numpy_random_seed: 2147483648
GPU 0: NVIDIA GeForce RTX 4080 Laptop GPU
CUDA_HOME: /usr/local/cuda
NVCC: Cuda compilation tools, release 11.1, V11.1.74
GCC: gcc (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
PyTorch: 2.1.0+cu121
PyTorch compiling details: PyTorch built with:

• GCC 9.3
• C++ Version: 201703
• Intel(R) oneAPI Math Kernel Library Version 2022.2-Product Build 20220804 for Intel(R) 64 architecture applications
• Intel(R) MKL-DNN v3.1.1 (Git Hash 64f6bcbcbab628e96f33a62c3e975f8535a7bde4)
• OpenMP 201511 (a.k.a. OpenMP 4.5)
• LAPACK is enabled (usually provided by MKL)
• NNPACK is enabled
• CPU capability usage: AVX2
• CUDA Runtime 12.1
• NVCC architecture flags: -gencode;arch=compute_50,code=sm_50;-gencode;arch=compute_60,code=sm_60;-gencode;arch=compute_70,code=sm_70;-gencode;arch=compute_75,code=sm_75;-gencode;arch=compute_80,code=sm_80;-gencode;arch=compute_86,code=sm_86;-gencode;arch=compute_90,code=sm_90
• CuDNN 8.9.2
• Magma 2.6.1
• Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=12.1, CUDNN_VERSION=8.9.2, CXX_COMPILER=/opt/rh/devtoolset-9/root/usr/bin/c++, CXX_FLAGS= -D_GLIBCXX_USE_CXX11_ABI=0 -fabi-version=11 -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Werror=bool-operation -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=old-style-cast -Wno-invalid-partial-specialization -Wno-unused-private-field -Wno-aligned-allocation-unavailable -Wno-missing-braces -fdiagnostics-color=always -faligned-new -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Werror=cast-function-type -Wno-stringop-overflow, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.1.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=OFF, USE_NCCL=1, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF,

TorchVision: 0.16.0+cu121
OpenCV: 4.8.1
MMEngine: 0.10.1
MMDetection: 3.3.0
MMDetection3D: 1.4.0+fe25f7a
spconv2.0: False

Reproduces the problem - code sample

python tools/test.py projects/TPVFormer/configs/tpvformer_8xb1-2x_nus-seg.py checkpoints/tpvformer_pretrained_fcos3d_r101_dcn.pth

Reproduces the problem - command or script

python tools/test.py projects/TPVFormer/configs/tpvformer_8xb1-2x_nus-seg.py checkpoints/tpvformer_pretrained_fcos3d_r101_dcn.pth

Reproduces the problem - error message

Traceback (most recent call last):
  File "/home/sherlock/workspace/learning/mmdet3d/tools/test.py", line 149, in <module>
    main()
  File "/home/sherlock/workspace/learning/mmdet3d/tools/test.py", line 145, in main
    runner.test()
  File "/home/sherlock/anaconda3/envs/sherlock/lib/python3.10/site-packages/mmengine/runner/runner.py", line 1823, in test
    metrics = self.test_loop.run()  # type: ignore
  File "/home/sherlock/anaconda3/envs/sherlock/lib/python3.10/site-packages/mmengine/runner/loops.py", line 438, in run
    metrics = self.evaluator.evaluate(len(self.dataloader.dataset))
  File "/home/sherlock/anaconda3/envs/sherlock/lib/python3.10/site-packages/mmengine/evaluator/evaluator.py", line 79, in evaluate
    _results = metric.evaluate(size)
  File "/home/sherlock/anaconda3/envs/sherlock/lib/python3.10/site-packages/mmengine/evaluator/metric.py", line 133, in evaluate
    _metrics = self.compute_metrics(results)  # type: ignore
  File "/home/sherlock/anaconda3/envs/sherlock/lib/python3.10/site-packages/mmdet3d/evaluation/metrics/seg_metric.py", line 130, in compute_metrics
    ret_dict = seg_eval(
  File "/home/sherlock/anaconda3/envs/sherlock/lib/python3.10/site-packages/mmdet3d/evaluation/functional/seg_eval.py", line 100, in seg_eval
    pred_seg[gt_seg == ignore_index] = -1
IndexError: too many indices for array: array is 0-dimensional, but 1 were indexed

Additional information

I added indices=2 in val_dataloader in config file to debug.

val_dataloader = dict(
    batch_size=1,
    num_workers=4,
    persistent_workers=True,
    drop_last=False,
    sampler=dict(type='DefaultSampler', shuffle=False),
    dataset=dict(
        type=dataset_type,
        data_root=data_root,
        data_prefix=data_prefix,
        ann_file='nuscenes_infos_val.pkl',
        pipeline=val_pipeline,
        test_mode=True, indices=2))

When I look into seg_eval() function to print out some info, I got the following information

seg_pred shape = [array(1391), array(1391)]
gt_labels shape = [array([11, 11, 11, ..., 15, 15, 15], dtype=uint8), array([11, 11, 11, ..., 15, 15, 15], dtype=uint8)]
pred_seg = 1391
gt_seg = [11 11 11 ... 15 15 15]

It seems that the output of TPVFormer does not generate the expected segmentation mask. How can I fix this now?

[cuge1995]

The same issue, did you solved it?

[cuge1995]

Here are more output

array(3089), array(3089), array(3057), array(3089), array(3057), array(3121
), array(2865), array(2865), array(2480), array(13), array(1296), array(591), array(271), array(15), arr
ay(14), array(14), array(13), array(13), array(14), array(14), array(17), array(17), array(19), array(19
), array(16), array(15), array(16), array(18), array(18), array(19), array(20), array(19), array(8), arr
ay(10), array(12), array(13), array(19), array(21), array(22), array(534), array(14), array(18), array(1
5), array(21), array(16), array(19), array(19), array(18), array(19), array(16), array(16), array(8), ar
ray(11), array(16), array(13), array(5), array(9), array(6), array(6), array(3), array(2), array(3), arr
ay(4), array(6), array(4), array(6), array(8), array(8), array(6), array(6), array(7), array(10), array(
7), array(10), array(10), array(10), array(7), array(5), array(6), array(15), array(19), array(20), arra
y(21), array(22), array(22), array(22), array(22), array(123), array(23), array(23), array(24), array(16
), array(14), array(11), array(7), array(5), array(6), array(4), array(0), array(0), array(0), array(7),
 array(3), array(14), array(6), array(6), array(20), array(23), array(25), array(9), array(23), array(9)
, array(28), array(12), array(12), array(9), array(9), array(8), array(26), array(28), array(22), array(
22), array(22), array(22), array(22), array(22), array(22), array(22), array(22), array(22), array(25),
array(25), array(22), array(24), array(24), array(22), array(23), array(21), array(21), array(21), array
(21), array(22), array(22), array(22), array(23), array(25), array(16), array(23), array(983), array(22)
, array(93), array(1240), array(250), array(24), array(25), array(27), array(19), array(18), array(21),
array(18), array(15), array(21), array(13), array(14), array(23), array(23), array(23), array(13), array
(11), array(10), array(11), array(12), array(12), array(0), array(11), array(11), array(11), array(12),
array(14), array(16), array(16), array(16), array(16), array(16), array(15), array(18), array(17), array
(18), array(12), array(18), array(19), array(12), array(18), array(20), array(20), array(21), array(22),
 array(20), array(11), array(11), array(3416), array(3453), array(3443), array(3507), array(3827), array
(4497), array(104), array(127), array(3604), array(1414), array(2973), array(2119), array(4299), array(1
3), array(35), array(448), array(21), array(2742), array(306), array(22), array(18), array(18), array(16
), array(30), array(23), array(19), array(14), array(5), array(10), array(6), array(6), array(5), array(
5), array(9), array(5), array(8), array(9), array(9), array(8), array(9), array(5055), array(223), array
(637), array(5373), array(4765), array(2396), array(858), array(347), array(951), array(636), array(247)
, array(18), array(5), array(8), array(14), array(1234), array(17), array(17), array(17), array(17), arr
ay(17), array(17), array(18), array(17), array(17), array(17), array(17), array(17), array(18), array(17
), array(18), array(20), array(19), array(19), array(19), array(20), array(20), array(20), array(20), ar
ray(20), array(20), array(350), array(30), array(30), array(29), array(30), array(279), array(23), array
(20), array(20), array(20), array(21), array(20), array(20), array(20), array(23), array(24), array(21),
 array(19), array(19), array(19), array(1273), array(21), array(2943), array(1919), array(863), array(95
), array(30), array(29), array(29), array(30), array(20), array(16), array(20), array(4046), array(1227)

[cuge1995]

update

        for i in range(len(seg_logits_list)):
            seg_logits = seg_logits_list[i]
            seg_pred = seg_logits
            # print(seg_logits.size())
            # seg_pred = seg_logits.argmax(dim=0)
            # print('the seg 2222pred is',seg_pred)
            batch_data_samples[i].set_data({
                'pts_seg_logits':
                PointData(**{'pts_seg_logits': seg_logits}),
                'pred_pts_seg':
                PointData(**{'pts_semantic_mask': seg_pred})
            })
        return batch_data_samples

It works!

[russellyq]

Hi, @cuge1995

Can you provide full code on seg_eval.py