Add XLNetBackbone

[shivance]

Closes #753

Referring XLNetBase, TransformerXL, Two stream Attention

[shivance]

Still WIP.

[shivance]

Hey @mattdangerw & @chenmoneygithub Turns out that addition of XLNet needs MultiHeadRelativeAttention and TwoStreamRelativeAttention apart from relative encoding, transformerXL layers.

Should I break this PR up into multiple ones? I'm not able to make entire model work correctly, just yet.

[mattdangerw]

@shivance timely question! We actually have a contribution guide for models about to land here. #820

Could you try following that and give us feedback?

Re the overall stucture, lets keep everything in a models/xlnet directory for now (don't contribute directly to keras_nlp.layers just yet, though we can follow up with that), and while looking at huggingface or model garden for inspiration is totally fine, try to make sure we stick to the "local" style of KerasNLP wherever possible when writing layers, models etc.

We also should make sure we have the full forward pass matching correctly before we get too deep into review on style, etc. You can usually try making a colab for starters that aligns the forward pass with a reference implementation and share that here!

[abheesht17]

@shivance, let me know when this is ready for review!

[shivance]

@shivance, let me know when this is ready for review!

Really Sorry for slacking off here, am not able to find time cuz of personal reasons. Will follow up soon.

[abheesht17]

No hurry, take your time! Take care!

[susnato]

Hi @shivance can I please continue this PR?

[shivance]

@susnato I have done basic ground work already, however the PR is bug prone, I'm not planning to work on it any sooner though as I've been onto #1052 . So you could probably fork my branch, and continue with my commits (which I made so far). What say?

[susnato]

Hi @shivance thanks for quick reply, I would love to continue your work. How about I fork your branch and push the commits on your repo(to shivance/keras-nlp_branch...) and you merge them, that way we both will share credits? or if you want I can copy the existing code and make a new PR. What do you say?

[shivance]

I think if I add you as collaborator on my fork, and then when you push to my branch, the same PR should get updated without getting closed.

…

On Mon, 5 Jun, 2023, 00:47 Susnato Dhar, ***@***.***> wrote: Hi @shivance <https://github.com/shivance> thanks for quick reply, I would love to continue your work. How about I fork your branch and push the commits on your repo(to shivance/keras-nlp_branch...) and you merge them, that way we both will share credits? or if you want I can copy the existing code and make a new PR. What do you say? — Reply to this email directly, view it on GitHub <#928 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AMK2NO5D7TXLOOIO572OOITXJTNLNANCNFSM6AAAAAAWIKP6E4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[susnato]

then please add me as a collaborator @shivance .

[shivance]

@susnato just gave!

[susnato]

Hi @mattdangerw, I am currently working on this integration. I have one small doubt -
As you said here
"We also should make sure we have the full forward pass matching correctly before we get too deep into review on style"
I am using the HuggingFace Implementation and weights as a reference for forward pass since it is Tensorflow-2. Is it ok or should I switch to another implementation?

[susnato]

Hi @mattdangerw @abheesht17 , In xlnet implementation there are some optional arguments such as mems and perm_mask which are mostly used during pretraining but during inference we mostly leave it as None, but as in keras-nlp we don't define the call method for Backbones(as I noticed in all models), instead we define all the Input layers in __init__ but that makes things complicated as if we define a Input layer for mems, we can't leave it Noneduring the inference otherwise they will give error.

Should I then define the call method in the backbone?(That would make this backbone an exception) or should I do a workaround(eg, instead of passing None we can pass -1e+9)?

[susnato]

Hi @shivance I am sorry but I think it's better to open a new PR. Since you created this PR I need to tag you all time to ask for changing description, to make it ready for review from draft or asking reviews from maintainers, and also you might get bothered about getting tagged all the time, so I am making a new PR, so please close this one also please do not worry, I am going to continue from your commits.