some error in calc_IoU_sets function

[niuniupower]

def calc_IoU_Sets(truth, pred, c=1, **kwargs):
# Obtain sets with associated class
gt = np.equal(truth, c)
pd = np.equal(pred, c)
# Calculate IoU
if (pd.sum() + gt.sum() - np.logical_and(pd, gt).sum()) != 0:
iou = np.logical_and(pd, gt).sum() /
(pd.sum() + gt.sum() - np.logical_and(pd, gt).sum())
else : iou = 0.0
# Return computed IoU
return iou

In this function, you should reduce the FN, but you reduce the FP.

[muellerdo]

Hello @niuniupower,

mhm, I re-checked the correctness of the two implementations (set-wise and confusion-matrix-wise) of the IoU/Jaccard.
They should be fine.

Where exactly did you find an error?

For the formulas, I would refer to Wikipedia: https://en.wikipedia.org/wiki/Jaccard_index

Set-wise:

def calc_IoU_Sets(truth, pred, c=1, **kwargs):
    # Obtain sets with associated class
    gt = np.equal(truth, c)
    pd = np.equal(pred, c)
    # Calculate IoU
    if  (pd.sum() + gt.sum() - np.logical_and(pd, gt).sum()) != 0:
        iou = np.logical_and(pd, gt).sum() / \
              (pd.sum() + gt.sum() - np.logical_and(pd, gt).sum())
    else : iou = 0.0
    # Return computed IoU
    return iou

Confusion-matrix-wise:

def calc_IoU_CM(truth, pred, c=1, **kwargs):
    # Obtain confusion mat
    tp, tn, fp, fn = calc_ConfusionMatrix(truth, pred, c)
    # Calculate IoU
    if (tp + fp + fn) != 0 : iou = tp / (tp + fp + fn)
    else : iou = 0.0
    # Return computed IoU
    return iou

Cheers,
Dominik

UPDATE: Mhm, I see that there is a definition difference in the confusion-matrix-based formulas.

Tensorflow, Keras and key publications like Taha & Hanbury (https://bmcmedimaging.biomedcentral.com/track/pdf/10.1186/s12880-015-0068-x.pdf) define it with False-Negative instead of True-Negative as Wikipedia.

• https://www.tensorflow.org/api_docs/python/tf/keras/metrics/BinaryIoU
• https://keras.io/api/metrics/segmentation_metrics/

However, the reference in Wikipedia is also to the key paper of Taha & Hanbury, which indicates that we have found one of these rare mistakes in Wikipedia :o

UPDATE#2:
Updated the incorrect confusion-matrix based formula in the Wikipedia page for Jaccard: https://en.wikipedia.org/wiki/Jaccard_index