Attention masking bug?

[rayleizhu]

It seems that your attention masking part is wrong, as the image patches from different images should have a different mask, while you use a single mask definition for all patches regardless of where they are from.

MixMIM/models_mixmim.py

Line 130 in 9da3eee

if attn_mask is not None:

BTW, how do you cope with mixing more images? I see only mixing 2 images case in the code.

[rayleizhu]

I've figured out why: complementary masking is implemented in WindowAttention:

MixMIM/models_mixmim.py

Line 80 in 9da3eee

mask_new = mask * mask.transpose(2, 3) + (1 - mask) * (1 - mask).transpose(2, 3)