decode.acc_seg and aux.acc_seg are high but mIoU is very low

[PangXuejiao]

When I train my own dataset, the decode.acc_seg and aux.acc_seg can reach more than 80%, but the mIoU is only about 10%. Does anyone know why? And how to calculation the decode.acc_seg and aux.acc_seg？which file I can find the functions in?

[MengzhangLI]

Hi, @PangXuejiao

I guess it is because the decode.acc_seg is calculated with both foreground and background. We all know many background pixels can be correctly classified as background label. And when calculating mIoU, only foreground counts!

For example, see this config and its log:

This phenomenon is same as your claim because the foreground pixels (i.e., vessel of medical image) is way too less than background.

Best,

[PangXuejiao]

Hi, @PangXuejiao

I guess it is because the decode.acc_seg is calculated with both foreground and background. We all know many background pixels can be correctly classified as background label. And when calculating mIoU, only foreground counts!

For example, see this config and its log:

This phenomenon is same as your claim because the foreground pixels (i.e., vessel of medical image) is way too less than background.

Best,

Thank you for your timely reply. However, I used "fcn_unet_s5-d16_128x128_40k_chase_db1.py" to train my dataset. My dataset contains three categories, no background category, and the proportion of the three categories is similar. And how does the code know where the background is? Your answer inspired me. I'll see the code and try again

[MengzhangLI]

Do you mean your customized dataset does not have label 0? I mean your three kinds labels should be set 0, 1 and 2, respectively. It still has problem because the calculation in log mixing all pixels while in test mode or evaluation only certain foreground counts.

[PangXuejiao]

Do you mean your customized dataset does not have label 0? I mean your three kinds labels should be set 0, 1 and 2, respectively. It still has problem because the calculation in log mixing all pixels while in test mode or evaluation only certain foreground counts.

My three kinds labels have be set 0, 1 and 2, respectively. But my dataset is like ade, specifically，the '0' label also represents a specific category rather than a background. So, the '0' label is also need to be calculated when calculating mIoU. Meanwhile, how to calculate the mIoU of the adk and cityscapes dataset (the datasets have no background)

[MengzhangLI]

Just debug whether various kinds labels make mIoU metric different first.

[PangXuejiao]

Just debug whether various kinds labels make mIoU metric different first.

Thank you very much. I use 'labelme' to label my dataset, and then I delete the 'background' . According to your suggection, I keep the background. And then, the 'mIoU' returns to normal.

[MengzhangLI]

Great to hear that!

Now I will close this issue but welcome to create the issue or pull request in the future!

Best,