Doubting DiceLoss and DiceMetric are not correct

[al3ms]

I am working on medical image segmentation and trying different networks from Monai.
I am using exactly the same data, code, loss, and metric settings. I get these results:

1- AHNet:
Training epoch_loss: 0.4314
test_loss_epoch: 0.4469
Dice: 0.5642064809799194
IoU: 0.48765531182289124
current epoch: 71
best mean dice: 0.5642 at epoch: 71

2-Basic U-Net
Training epoch_loss: 0.5231
test_loss_epoch: 0.5310
Dice: 0.8365685343742371
IoU: 0.7675291299819946
current epoch: 17
best mean dice: 0.8366 at epoch: 17

3-UNETR
Training epoch_loss: 0.4261
test_loss_epoch: 0.4492
Dice: 0.8440054059028625
IoU: 0.7712767720222473
current epoch: 49
best mean dice: 0.8440 at epoch: 49

4-UNet
Training epoch_loss: 0.2848
test_loss_epoch: 0.3179
Dice: 0.8705800175666809
IoU: 0.8008243441581726
current epoch: 172
best mean dice: 0.8706 at epoch: 172

5-Pre trained UNet :
Training epoch_loss: 0.0968
test_loss_epoch: 0.1447
Dice: 0.8927300572395325
IoU: 0.8281022906303406
current epoch: 16
best mean dice: 0.8927 at epoch: 16

Are the results correct? Since it does not make sense to have one network with a high loss and a high metric value where another has a high metric value and low loss.

I searched the internet but no clear example where it used both dice metric and loss at the same time except one who calculated the loss using the built in function and calculates the dice through equation: loss = 1 - dice. Also. I tried changing many sittings but nothing work.

This is the code: (I tried to make lines between the relevant code)

##############################
loss_function = DiceLoss(to_onehot_y=True, softmax=True)
optimizer = torch.optim.Adam(model.parameters(), 1e-4, weight_decay=1e-5, amsgrad=True)
post_pred = Compose([EnsureType(),AsDiscrete(argmax=True, to_onehot=2)])
post_label = Compose([EnsureType(),AsDiscrete(to_onehot=2)])
##############################

def train(model, data_in, loss, optim, max_epochs, model_dir, test_interval=1 , device=torch.device("cuda:0")):
    best_metric = -1
    best_metric_epoch = -1
    save_loss_train = []
    save_loss_test = []
    save_metric_train = []
    save_metric_test = []
    train_loader, test_loader = data_in
    
##############################
    meanIoU = MeanIoU(reduction="mean") 
    dice_metric = DiceMetric(include_background=True, reduction="mean") 
##############################
    for epoch in range(max_epochs):
        print("-" * 10)
        print(f"epoch {epoch + 1}/{max_epochs}")
        model.train()
        train_epoch_loss = 0
        train_step = 0
        epoch_metric_train = 0

        for batch_data in train_loader:
            train_step += 1
            volume = batch_data["vol"]
            label = batch_data["seg"]
            label = label != 0
            volume, label = (volume.to(device), label.to(device))
            outputs = model(volume)
	    optim.zero_grad()
            train_loss = loss(outputs, label)
            train_loss.backward()
            optim.step()
            train_epoch_loss += train_loss.item()
            print(
                f"{train_step}/{len(train_loader) // train_loader.batch_size}, "
                f"Train_loss: {train_loss.item():.4f}")

        print('-'*20)
        train_epoch_loss /= train_step
        print(f'Training epoch_loss: {train_epoch_loss:.4f}')
        save_loss_train.append(train_epoch_loss)
        np.save(os.path.join(model_dir, 'loss_train.npy'), save_loss_train)
        save_metric_train.append(epoch_metric_train)
        np.save(os.path.join(model_dir, 'metric_train.npy'), save_metric_train)

        if (epoch + 1) % test_interval == 0:
            model.eval()
            with torch.no_grad():
                test_epoch_loss = 0
                test_metric = 0
                epoch_metric_test = 0
                test_step = 0
                IoU=0

                for test_data in test_loader:
                    test_step += 1
                    test_volume = test_data["vol"]
                    test_label = test_data["seg"]
                    test_label = test_label != 0
                    test_volume, test_label = (test_volume.to(device), test_label.to(device),)
                    test_outputs = model(test_volume) 
##############################
                    test_loss = loss(test_outputs, test_label)
                    test_epoch_loss += test_loss.item()

                    test_outputs2 = [post_pred(i) for i in decollate_batch(test_outputs)]
                    test_label2 = [post_label(i) for i in decollate_batch(test_label)]

                    dice_metric(y_pred=test_outputs2, y=test_label2)
                    meanIoU(test_outputs2,test_label2)

                cd = dice_metric.aggregate().item() 
                iou = meanIoU.aggregate().item() 

                dice_metric.reset() 
                meanIoU.reset() 
##############################

                test_epoch_loss /= test_step
                print(f'test_loss_epoch: {test_epoch_loss:.4f}')
                save_loss_test.append(test_epoch_loss)
                np.save(os.path.join(model_dir, 'loss_test.npy'), save_loss_test)
                print(f'Dice:',cd)   
                save_metric_test.append(cd)
                np.save(os.path.join(model_dir, 'metric_test.npy'), save_metric_test)
                print(f'IoU:',iou)
                if cd > best_metric:
                    best_metric = cd
                    best_metric_epoch = epoch + 1
                    torch.save(model.state_dict(), os.path.join(
                        model_dir, "best_metric_model.pth"))

                print(
                    f"current epoch: {epoch + 1} current mean dice: {test_metric:.4f}"
                    f"\nbest mean dice: {best_metric:.4f} "
                    f"at epoch: {best_metric_epoch}"
                )  

    print(
        f"train completed, best_metric: {best_metric:.4f} "
        f"at epoch: {best_metric_epoch}")

train(model, data_in, loss_function, optimizer, 600, model_dir)

Thanks

[wyli]

thanks for the question, maybe it's because of the different final blocks for different networks, for example, the basic_unet returns the logit directly https://github.com/Project-MONAI/MONAI/blob/e6ec945e4b87b90835cdad29ea64b1f27b8accda/monai/networks/nets/basic_unet.py#L251

unet by default returns the result after a relu https://github.com/Project-MONAI/MONAI/blob/e6ec945e4b87b90835cdad29ea64b1f27b8accda/monai/networks/nets/unet.py#L122

also dice_metric is not 1 - DiceLoss because dice_metric is computed after post_pred which is argmax in this case, but DiceLoss is based on softmax=True in your example.

[al3ms]

Thanks @wyli for clarification. So what is the right way of calculating the Dice Metric? Is it after post_pred using argmax? If so, is that mean the calculations in my work are correct?

[wyli]

yes, it looks good to me...