Extending Encoder Decoder to GPT-2

[satwikkottur]

Adding GPT2 initialization for EncoderDecoder model as pointed out in the issue below.

Currently, only Bert works as a decoder. We might add GPT2 in a couple of weeks. Note that no model has cross-attention layers if it is not already an encoder-decoder model (like Bart or T5) and in this case it does not make sense to use the encoder-decoder wrapper. The model is initialized with random weights for the cross attention layers which will have to be fine-tuned. I agree, that this should be made clearer in the documentation!

Originally posted by @patrickvonplaten in #4517 (comment)

[patrickvonplaten]

It's on the roadmap :-)

[satwikkottur]

Thank you! Look forward to it :)

[djw1809]

Hi - I've actually been working on this myself the past couple days, should I submit a PR when finished?

[patrickvonplaten]

That'd be great!

[djw1809]

Will do - likely sometime this week.

[MaveriQ]

@djw1809 Any update on the PR? :)

[iliemihai]

@patrickvonplaten Hello Patrick, I am watching with much interest EncodeDecoder from transformers :) . Any updates on supporting GPT2 with EncodeDecoder ?

[djw1809]

Got sidetracked with other research - coming back to it in several days, working on my end, just need to play nice with the rest of the repo.

…

On Tue, Jul 7, 2020 at 3:32 PM Mihai Ilie ***@***.***> wrote: @patrickvonplaten <https://github.com/patrickvonplaten> Hello Patrick, I am watching with much interest EncodeDecoder from transformers :) . Any updates on supporting GPT2 with EncodeDecoder ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#4961 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG3PODZXYPBB33F4CBSNZLDR2OO7VANCNFSM4N4QTZQA> .

-- Dylan Weber, Research Assistant | PhD Candidate School of Math and Statistical Sciences WXLR642/BYENG593 Arizona State University

[patrickvonplaten]

@djw1809 - also feel free to already open a PR with unfinished code yet so that I can take a look early on and help you :-)

[patrickvonplaten]

Working on it now. Also linking this PR: #4483

[Squire-tomsk]

@patrickvonplaten Hello Patrick.
As I see from 1d6e71e current cross attention implimentation assume that encoder have same hidden size as GPT-2. I have encoder with hidden size 512 and want to combine it with GPT-2 medium with hidden size 1024. I have done it by Fairseq code and now want to do same by Huggingface. Could you update your solution to support any suitable encoder hidden space size?

[patrickvonplaten]

Hey @Squire-tomsk,

I see what you mean - this would mean to add a new config param for each model that has cross-attention...is this common practice? Would be great if you could open a new issue for that :-)

[Squire-tomsk]

Done #6645

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.