What is the purpose of SUPPORTED_DROPOUT_MODE in MLPBlock

[clamcy]

Hello,

I have a question regarding the differentiation between vit and swin in the SUPPORTED_DROPOUT_MODE for the MLPBlock in MONAI.

Code reference: https://github.com/Project-MONAI/MONAI/blob/dev/monai/networks/blocks/mlp.py

At line 54, we define self.drop1 = nn.Dropout(dropout_rate).

Following this, there is a conditional check:

If dropout_opt is vit, then self.drop2 = nn.Dropout(dropout_rate).
If dropout_opt is swin, then self.drop2 = self.drop1.

It seems that in the latter case, self.drop2 is the same as the former.

Could you please explain the purpose of this differentiation?

Thank you and sorry for any inconvenience caused.

[KumoLiu]

Hi @clamcy, thanks for your interest here.

The difference is described here: https://github.com/Project-MONAI/MONAI/blob/08d572839ede5d087e49d4dfadba033e512528df/monai/networks/blocks/mlp.py#L37C13-L41C95
"vit" mode uses two dropout instances and "swin" will only use one.

[clamcy]

Hello @KumoLiu, thank you for your reply! However, it seems that calling the same dropout instance twice does not differ significantly from calling two separate dropout instances, as torch.nn.Dropout.forward simply calls F.dropout(). You can find a related discussion here: https://discuss.pytorch.org/t/using-same-dropout-object-for-multiple-drop-out-layers/39027