Mask t5 relative position bias then head pruned

[hadaev8]

What does this PR do?

Fixes #17886

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@patrickvonplaten

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[patrickvonplaten]

Thanks for the PR @hadaev8!

Could you add a test for the newly added functionality ? :-)

[hadaev8]

@patrickvonplaten
Never did it, how it should looks like?

[patrickvonplaten]

Hey @hadaev8,

The test should be added to this file here: https://github.com/huggingface/transformers/blob/main/tests/models/t5/test_modeling_t5.py

In this test it would be great if you could do the following for example:
Create a dummy T5 model and run a forward pass with output_attentions=True. The prune a head and run a forward pass again with output_attentions=True. Then you can compare that the attentions returned by the second forward pass will be 0 or just have fewer tensors because the head was pruned

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[hadaev8]

@patrickvonplaten
I spotted another thing.
In encoder-decoder only one broken function to prune. Should I split in two?

[patrickvonplaten]

Looks good to me!

[patrickvonplaten]

@hadaev8 let's maybe do this in another PR :-)

[hadaev8]

Okay, i only will make test

[patrickvonplaten]

Hey @hadaev8,

Sorry last thing - could you maybe remove the accidently added datasets folder? See: https://github.com/huggingface/transformers/pull/17968/files#diff-714284abfa95a1447d7c34554c2d65b16fcfb1af22a44fc15489d13b76e951e5

[LysandreJik]

LGTM! Happy to merge this once the datasets folder has been removed :)

[hadaev8]

@patrickvonplaten
Sorry, still had no time to write the test.
Removed datasets folder.

[LysandreJik]

It seems there is an issue with your CircleCI permissions, the tests won't run.
Could you try refreshing your permissions as shown here?

[hadaev8]

Seems I can't register account at CircleCI because sanctioned country.

[LysandreJik]

Arf that's super annoying, sorry about that @hadaev8. I'll look into triggering it for you.

[LysandreJik]

I pushed the same commits under a different branch: https://github.com/huggingface/transformers/tree/fix_t5_pruning-lysandre
It used my token permissions so it could run. Sorry you're experiencing this, I'll handle the triggers if some tests need to be fixed.

[LysandreJik]

All tests pass, thank you @hadaev8! Merging the PR.

[hadaev8]

@LysandreJik
Cool, thank you.