Accuracy metric computing doesn't ignore "ignore index"

[jiehuang165]

Describe the bug
Shouldn't "ignore_index" be ignored when computing accuracy?

Model training will calculate "acc_seg" when computing loss.

mmsegmentation/mmseg/models/decode_heads/decode_head.py

Line 264 in ba52d50

loss['acc_seg'] = accuracy(seg_logit, seg_label)

It calls accuracy funciton

mmsegmentation/mmseg/models/losses/accuracy.py

Line 5 in ba52d50

def accuracy(pred, target, topk=1, thresh=None):

and the accuracy metric is calculated as

mmsegmentation/mmseg/models/losses/accuracy.py

Line 49 in ba52d50

res.append(correct_k.mul_(100.0 / target.numel()))

However, the denominator target.numel() includes areas with ignore_index. I mean accuracy=(num of correct) / (all num without ignore). Ignore samples should not present in both numerator and denominator, right? The above implementation causes the accuracy metric to be lower than expected. I think maybe the correct codes are:

loss['acc_seg'] = accuracy(seg_logit, seg_label, ignore_index=self.ignore_index)

def accuracy(pred, target, topk=1, thresh=None, ignore_index=-100):
    ...
    res.append(correct_k.mul_(100.0 / target[target != ignore_index].numel()))
    ...

[HJoonKwon]

@jiehuang165 I have the same issue and agree with you. In the function 'losses' in the decode_head.py, other losses like loss_decode receives the ignore_index as input, but accuracy does not.

        if not isinstance(self.loss_decode, nn.ModuleList):
            losses_decode = [self.loss_decode]
        else:
            losses_decode = self.loss_decode
        for loss_decode in losses_decode:
            if loss_decode.loss_name not in loss:
                loss[loss_decode.loss_name] = loss_decode(
                    seg_logit,
                    seg_label,
                    weight=seg_weight,
                    ignore_index=self.ignore_index)
            else:
                loss[loss_decode.loss_name] += loss_decode(
                    seg_logit,
                    seg_label,
                    weight=seg_weight,
                    ignore_index=self.ignore_index)

        loss['acc_seg'] = accuracy(seg_logit, seg_label)
        return loss

[muyuuuu]

I have puzzled by this problem...

  File "/home/20031211375/DFC/mmsegmentation/mmseg/models/losses/accuracy.py", line 52, in accuracy
    correct_k.mul_(100.0 / target[target != ignore_index].numel()))
ZeroDivisionError: float division by zero

[HJoonKwon]

@muyuuuu #1336 proposes a solution to this problem

[muyuuuu]

thank you.

[MeowZheng]

solved