🐛 Bug in attention head mask for cross-attention module in encoder-decoder models

[stancld]

Currently, encoder-decoder models use either head_mask or decoder_head_mask for masking attention heads in cross-attention modules. Both cases are not perfectly correct. Furthermore, MHA in cross-attention modules shares the parameters with the decoder, i.e. shape = (decoder.num_layers, decoder.num_attention_heads), therefore, the usage of encoder head_mask in the cross-attention module may lead to errors due to the shape mismatch.

---

My contribution: I will take care of this issue this weekend.

---

Reviewers: @patil-suraj @patrickvonplaten

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[patil-suraj]

Unstale

[patrickvonplaten]

Hey @stancld,

Sorry for being so unresponsive here - I'm happy to change the behavior and provide 3 masks