[Fix] Make accuracy take into account ignore_index

[HJoonKwon]

Motivation

Your repository has been so helpful for my projects and I deeply appreciate it.
My opinion can be wrong, but please look into it and I hope it can be helpful a bit at least.

I tried to apply

reduce_zero_label=True 

to ignore the background label, and it turned out that the value of acc_seg from the decode_head was weirdly low while other values such as loss and IoU were all looking good. Looking into the decode_head.py,

    @force_fp32(apply_to=('seg_logit', ))
    def losses(self, seg_logit, seg_label):
        """Compute segmentation loss."""
        loss = dict()
        seg_logit = resize(
            input=seg_logit,
            size=seg_label.shape[2:],
            mode='bilinear',
            align_corners=self.align_corners)
        if self.sampler is not None:
            seg_weight = self.sampler.sample(seg_logit, seg_label)
        else:
            seg_weight = None
        seg_label = seg_label.squeeze(1)

        if not isinstance(self.loss_decode, nn.ModuleList):
            losses_decode = [self.loss_decode]
        else:
            losses_decode = self.loss_decode
        for loss_decode in losses_decode:
            if loss_decode.loss_name not in loss:
                loss[loss_decode.loss_name] = loss_decode(
                    seg_logit,
                    seg_label,
                    weight=seg_weight,
                    ignore_index=self.ignore_index)
            else:
                loss[loss_decode.loss_name] += loss_decode(
                    seg_logit,
                    seg_label,
                    weight=seg_weight,
                    ignore_index=self.ignore_index)

        loss['acc_seg'] = accuracy(seg_logit, seg_label)
        return loss

the loss_decode receives self.ignore_index as an input argument, while the accuracy doesn't. I think it explains why the accuracy seemed so different from other metrics. Thus, I think the accuracy function should receive the ignore index as an argument as well to take into account the ignore_index to calculate the accuracy of the decode head.

And, I found that the point_head.py also uses the same function without receiving the ignore_index. So I guess both of them should be fixed.

Modification

Firstly, the accuracy function can be defined like this

def accuracy(pred, target, ignore_index=255, topk=1, thresh=None): 

and inside of the function, the accuracy can be calculated like this to exclude the ignored index,

res.append(correct_k.mul_(100.0 / target[target!=ignore_index].numel())) 

Secondly, in decode_head.py, we can simply put self.ignore_index into one of the arguments like this,

loss['acc_seg'] = accuracy(seg_logit, seg_label, ignore_index=self.ignore_index) 

Finally, the same strategy can be applied to point_head.py like this,

loss['acc_point'] = accuracy(point_logits, point_label, ignore_index=self.ignore_index)

BC-breaking (Optional)

Use cases (Optional)

Checklist

1. Pre-commit or other linting tools are used to fix the potential lint issues.
2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMDet3D.
4. The documentation has been modified accordingly, like docstring or example tutorials.

[CLAassistant]

All committers have signed the CLA.

[codecov]

Codecov Report

Merging #1259 (dd02ad8) into master (b163101) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master    #1259   +/-   ##
=======================================
  Coverage   90.22%   90.22%           
=======================================
  Files         130      130           
  Lines        7558     7560    +2     
  Branches     1258     1258           
=======================================
+ Hits         6819     6821    +2     
  Misses        531      531           
  Partials      208      208           

Flag	Coverage Δ
unittests	90.22% <100.00%> (+<0.01%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmseg/models/decode_heads/decode_head.py	92.63% <100.00%> (ø)
mmseg/models/decode_heads/point_head.py	93.54% <100.00%> (ø)
mmseg/models/losses/accuracy.py	100.00% <100.00%> (ø)

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 0934a57...dd02ad8. Read the comment docs.

[MeowZheng]

please add UT to test this feature in

mmsegmentation/tests/test_models/test_losses/test_utils.py

Line 44 in b4314f9

def test_accuracy():

[MeowZheng]

I think it would be better to set None as default value of ignore_index

[HJoonKwon]

Sure, I will do it with adding UT by tonight(KST). Thank you for the comment :)

[MeowZheng]

Suggested change

	ignore_index (int | None): The label index to be ignored. Default: 255
	ignore_index (int | None): The label index to be ignored.
	If it is None, any label will be taken into account. Default: None

[Junjun2016]

Hi, thanks for your contribution.
Please fix the lint error and add unittests.

[HJoonKwon]

I found a bug while making unit tests for it, and it is fixed now. I checked the pre-commit and all passed fine :)

[Junjun2016]

I found a bug while making unit tests for it, and it is fixed now. I checked the pre-commit and all passed fine :)

Great.

[HJoonKwon]

Hi! I have a question. Should I do pull-rebase based on the most recent commit on the master branch? or Am I done here? :)

[MeowZheng]

Yes, you should

[HJoonKwon]

@MeowZheng Okay, I did pull-rebase based on the current master branch and pushed again!