Swin unetr btcv segmentation's DSC seems not consistent with the original paper

[HJS-design]

Hello,

I find a pre-trained Swin UNETR model (named Swin unetr btcv segmentation) in the MONAI Model Zoo (https://monai.io/model-zoo.html). It seems that this is the final result of this tutorial 3d_segmentation/swin_unetr_brats21_segmentation_3d.ipynb

However, what stands out in your final model weight is the following mean DSC curve because it appears to slightly lower than originally expected. Because in the original paper (https://arxiv.org/pdf/2111.14791), Table 1 shows all DSC are significantly higher than 0.8, and even achieves 0.95 on some organs. By contrast, your DSC curve below only shows a mean DSC about 0.8.

In addition, Table 1 in their original paper shows that Swin UNETR achieves DSC of 0.875 on label 5 and 0.953 on label 7. But when I download your weight, I found none of them achieves DSC of 0.8 on average in the 6 validation scans you used.

I am confused about the difference between the DSC of yours and in the original paper. Could you kindly please give me an explanation for this?

[KumoLiu]

Hi @HJS-design, the reason behind this discrepancy could be multifactorial and may include differences in the pre-training data, the fine-tuning process, hyperparameter settings. Maybe you can find more details from the config file.
https://github.com/Project-MONAI/model-zoo/blob/5219004057db016a6fdf656943d982737cc81777/models/swin_unetr_btcv_segmentation/configs/train.json#L35

cc @tangy5 @dongyang0122, in case you have more information want to share here.

Thanks.

[HJS-design]

Hi KumoLiu. I conjecture the crucial reason is that the paper use an ensemble of 5 models obtained from the five-fold cross validation to predict (as mentioned in their experiment); By contrast, you only use the first fold to train the model. Another reason is that the paper report their result on the test dataset (which we can not download from the website) and your result is shown on the validation dataset. But I do not know my conjecture is correct. So I just want to obtain a verification from you.

[KumoLiu]

Hi @tangy5, @dongyang0122,

Could you please share more information here?

Thanks in advance!