Is there a problem with the mmsegmentation Miou calculation?

[daigang896]

Hello. Train my own dataset and use the same 500 images and labels as the verification set. After training, the val Miou (self written calculation) is quite different from the Miou evaluated during training. The Miou evaluated during training is 5.94% higher. I don't know what's going on?Is there a problem with the mmsegmentation Miou calculation?

[MengzhangLI]

Hi, in my opinion, the mIoU metric MMSegmentation adopted is standard with no problem, the same as many other framework such as Paddleseg (You can checkout its corresponding paper.)

Besides, please check the test mode: whole or slides, ss or ms or msf and so on.

Best,

[daigang896]

img_scale_test = (1280, 1280)

img_scale_test = (3384, 2710)
cfg.test_pipeline = [
dict(type='LoadImageFromFile'),
dict(
type='MultiScaleFlipAug',
img_scale=img_scale_test,
# img_ratios=[0.5, 0.75, 1.0, 1.25, 1.5, 1.75],
flip=False,
transforms=[
dict(type='Resize', keep_ratio=True),
# dict(type='RandomFlip'),
dict(type='Normalize', **cfg.img_norm_cfg),
dict(type='ImageToTensor', keys=['img']),
dict(type='Collect', keys=['img']),
])
]。thank you. The size of the original image is basically the same as that of my test, and the test Miou result of (1280,1280) is 5% higher. I don't know why.

[MengzhangLI]

Hi, why img_scale_test has two different value (1280, 1280) and (3384, 2710)?

[PangXuejiao]

I have a similar mistake with you. Have you solved it？

[MengzhangLI]

I have a similar mistake with you. Have you solved it？

Hi, @PangXuejiao and @daigang896 :

Can you check the image for mIoU calculation is whole image or just cropped size? I am not sure whether it is caused by different input.

Best,

[MengzhangLI]

Related issue: #794

Please also check whether the background is also been counted for calculation, which can really "improve" results.

Best,

[daigang896]

Hi, why img_scale_test has two different value (1280, 1280) and (3384, 2710)?

(3384, 2710) is the size of the original image. (1280, 1280) was also tested. The results are quite different

[MengzhangLI]

Hi, why img_scale_test has two different value (1280, 1280) and (3384, 2710)?

(3384, 2710) is the size of the original image. (1280, 1280) was also tested. The results are quite different

It is normal that different image sizes makes metric different, and we usually use metric of original image size.

[daigang896]

I see. Now I use the original size to measure.
I just don't quite understand how computing Miou with 500 images will cost more than 50G of CPU memory?

[wwjwy]

Hi, in my opinion, the mIoU metric MMSegmentation adopted is standard with no problem, the same as many other framework such as Paddleseg (You can checkout its corresponding paper.)

Besides, please check the test mode: whole or slides, ss or ms or msf and so on.

Best,

hi, can you explain in detail the meaning of different test modes(whole, slides, ms, ss, msf)? Thanks!

[MengzhangLI]

Hi, @wwjwy
It means in inference, whole or slide (part) of image is the input of network.

ss is single scale, i.e., one image.

ms is multi scale, i.e., several images (which is generated by single image)

msf is multi scale + flip operation, i.e., several images with several flip (certain data augmentaion, it likes test time augmentation).

In short, basical result is ss with whole image, and the results :

whole < slide (in many cases)

ss < ms < msf

best,

[wwjwy]

Hi, @wwjwy
It means in inference, whole or slide (part) of image is the input of network.

ss is single scale, i.e., one image.

ms is multi scale, i.e., several images (which is generated by single image)

msf is multi scale + flip operation, i.e., several images with several flip (certain data augmentaion, it likes test time augmentation).

In short, basical result is ss with whole image, and the results :

whole < slide (in many cases)

ss < ms < msf

best,

THANKS

[MengzhangLI]

I see. Now I use the original size to measure.
I just don't quite understand how computing Miou with 500 images will cost more than 50G of CPU memory?

Try to change batch size will change GPU memory. But I do not test the CPU memory, could you give me more details about CPU memory?

[daigang896]

Hello. During the training, the memory increased seriously during the full image verification Miou, and only 500 images cost more than 50 g of memory

[MengzhangLI]

Hello. During the training, the memory increased seriously during the full image verification Miou, and only 500 images cost more than 50 g of memory

Hi, I think it is caused by old version of MMSegmentation, please check this related PR: #709

In the old version, each test image sample would keep many useless things in memory such as its prediction mask, you can upgrade your MMSeg and check out whether it could be fixed.

Best,

[daigang896]

Thanks. I'll update MMSeg and try again.