Hausdorff and IoU are stopped changing while dice metric is decreasing

[al3ms]

Hello everyone, and thanks for such a great framework.

I need help using Hausdorff and IoU, where they stop learning when reaching a specific value! While the loss and dice metric keeps changing. surface_distance also has a problem since it is always equal to "inf".

These are three measurements (it is the same for the following 30 epochs):

********************Testing********************
Test_loss: 0.1801
Test_Metric: 0.6199
hausdorff:  24.8
surface_distance:  inf
IoU: 0.4916495382785797
current epoch: 1 current mean dice: 0.6199
best mean dice: 0.6199 at epoch: 1

********************Testing********************
Test_loss: 0.1544
Test_Metric: 0.6456
hausdorff:  24.8
surface_distance:  inf
IoU: 0.4916495382785797
current epoch: 2 current mean dice: 0.6456
best mean dice: 0.6456 at epoch: 2

********************Testing********************
Test_loss: 0.1111
Test_Metric: 0.6889
hausdorff:  24.8
surface_distance:  inf
IoU: 0.4916495382785797
current epoch: 3 current mean dice: 0.6889
best mean dice: 0.6889 at epoch: 3

This is my code:

meanIoU = MeanIoU(reduction="mean") 
surfaceDistanceMetric = SurfaceDistanceMetric(include_background=True, symmetric=True, reduction="mean")
hausdorffDistanceMetric = HausdorffDistanceMetric(include_background=True, percentile=95, reduction="mean") 

post_pred = Compose([AsDiscrete(argmax=True, to_onehot=2)])
post_label = Compose([AsDiscrete(to_onehot=2)])

Inside testing part in each epoch:
for t_data in test_loader:
test_step += 1
# print("Now I will loop within Task ",i+1, " testing set:")
test_volume = t_data["vol"]
test_label = t_data["seg"]
test_volume, test_label = (test_volume.to(device), test_label.to(device),)

  optimizer.zero_grad()

  test_outputs = model(test_volume)
  test_loss = loss_function(test_outputs, test_label)      

  test_outputs2 = [post_pred(i) for i in decollate_batch(test_outputs)]
  test_label2 = [post_label(i) for i in decollate_batch(test_label)]

  surfaceDistanceMetric(test_outputs2, test_label2)
  hausdorffDistanceMetric(test_outputs2, test_label2)
  meanIoU(test_outputs2,test_label2)

hd = hausdorffDistanceMetric.aggregate().item() 
sd = surfaceDistanceMetric.aggregate().item() 
iou = meanIoU.aggregate().item() 

hausdorffDistanceMetric.reset() 
surfaceDistanceMetric.reset() 
meanIoU.reset() 

print('hausdorff: ' , hd)
print('surface_distance: ' , sd)
print(f'IoU:',iou)

Thanks

[al3ms]

This warning is related to the surface distance problem. Could anyone suggest a solution:

/usr/local/lib/python3.10/dist-packages/monai/metrics/surface_distance.py:163: UserWarning: the prediction of class 1 is all 0, this may result in nan/inf distance.
warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/usr/local/lib/python3.10/dist-packages/monai/metrics/hausdorff_distance.py:170: UserWarning: the prediction of class 1 is all 0, this may result in nan/inf distance.
warnings.warn(f"the prediction of class {c} is all 0, this may result in nan/inf distance.")
/usr/local/lib/python3.10/dist-packages/numpy/lib/function_base.py:4486: RuntimeWarning: invalid value encountered in subtract
diff_b_a = subtract(b, a)

[yiheng-wang-nv]

Hi @al3ms , it can happen since there may have multiple samples.
You set:

surfaceDistanceMetric = SurfaceDistanceMetric(include_background=True, symmetric=True, reduction="mean")

and as we know, "(inf + 0.5)/2=inf", thus the output of surfaceDistanceMetric will always be inf, no matter how accurate other predictions are.

The following is an example:

from monai.metrics import DiceMetric, SurfaceDistanceMetric
import torch

sd_metric = SurfaceDistanceMetric(include_background=True, symmetric=True, reduction="mean")
dice_metric = DiceMetric(include_background=True)

# 1st epoch data

label_1 = torch.tensor([[[0, 1], [0, 0]], [[1, 0], [1, 1]]])
label_2 = torch.tensor([[[1, 1], [0, 1]], [[0, 0], [1, 0]]])
label_3 = torch.tensor([[[1, 0], [0, 1]], [[0, 1], [1, 0]]])
label_4 = torch.tensor([[[0, 0], [0, 1]], [[1, 1], [1, 0]]])
labels = torch.stack([label_1, label_2, label_3, label_4])

pred_1 = torch.tensor([[[1, 1], [1, 1]], [[0, 0], [0, 0]]])  # no foreground pred
pred_2 = torch.tensor([[[1, 0], [0, 1]], [[0, 1], [1, 0]]])  # only one pixel's pred is wrong
preds = torch.stack([pred_1, pred_2, label_3, label_4])

# 2nd epoch data
preds_2nd = torch.stack([pred_1, label_2, label_3, label_4])  # in this iteration, pred_2 becomes all correct

# 1st epoch
sd_metric(preds[:2], labels[:2])
dice_metric(preds[:2], labels[:2])

sd_metric(preds[2:], labels[2:])
dice_metric(preds[2:], labels[2:])

sd = sd_metric.aggregate().item()
dice = dice_metric.aggregate().item()

sd_metric.reset()
dice_metric.reset()
print("sd: ", sd)
print("dice: ", dice)

# 2nd epoch
sd_metric(preds_2nd[:2], labels[:2])
dice_metric(preds_2nd[:2], labels[:2])

sd_metric(preds_2nd[2:], labels[2:])
dice_metric(preds_2nd[2:], labels[2:])

sd = sd_metric.aggregate().item()
dice = dice_metric.aggregate().item()

sd_metric.reset()
dice_metric.reset()
print("sd: ", sd)
print("dice: ", dice)

Runnning the code above, you will see in the first epoch, dice is 0.7333333492279053, and in the second epoch, dice is 0.800000011920929. Whereas, sd is always inf.

[yiheng-wang-nv]

BTW, as @KumoLiu just mentioned, according to the code you posted, the mean dice related code is not existing. Therefore, could you please also post those part of the code, thus it could be more easier to help to figure out the issue?

[al3ms]

I calculated the dice from dice loss (1-dice loss), which makes sense why it is not stopped. Now I made some changes to code and the IoU works fine by adding the following:
torch.sigmoid(test_outputs)
While the hausdorffDistanceMetric is zero.

The final code:

post_label = Compose([EnsureType(),AsDiscrete(to_onehot=2)])
      test_volume = t_data["vol"]
      test_label = t_data["seg"]
      test_label = test_label != 0
      test_volume, test_label = (test_volume.to(device), test_label.to(device))
                
      optimizer.zero_grad()
      test_outputs = model(test_volume)
      test_loss = loss_function(test_outputs, test_label)      

      test_label2 = [post_label(i) for i in decollate_batch(test_label)]
      test_outputs2 = torch.sigmoid(test_outputs)

      dice_metric(y_pred=test_outputs2, y=test_label2)
      meanIoU(test_outputs2,test_label2)
      hausdorffDistanceMetric(test_outputs2, test_label2)

avg_test_loss += test_loss.item()

cd = dice_metric.aggregate().item() 
iou = meanIoU.aggregate().item() 
hd = hausdorffDistanceMetric.aggregate().item() 

dice_metric.reset() 
meanIoU.reset() 
hausdorffDistanceMetric.reset() 

print(f'Dice:',cd)   
print('hausdorff: ' , hd)
print(f'IoU:',iou)

[yiheng-wang-nv]

Hi @al3ms , Hausdorff Distance should be calculated on one-hot format predictions. In your case, after doing sigmoid activation, it may also need to use a threshold to do binarization on the predictions.
Can also check here:
https://github.com/Project-MONAI/MONAI/blob/30915adf6da70ac0dceacd9fc4afba6d0dce04a1/monai/metrics/hausdorff_distance.py#L154