Add LLaMA support

[0amp]

Description

Add support for all 4 LLaMA models using huggingface as a backend. Requires main branch of transformers repo and accelerate (pip install accelerate git+https://github.com/huggingface/transformers).

I added accelerate to setup.py. I'm not sure how to incorporate the main branch of transformers into the package dependencies.

Fixes #201

Type of change

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)

Checklist:

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works (see LLaMA.ipynb demo)
• New and existing unit tests pass locally with my changes
• I have not rewritten tests relating to key interfaces which would affect backward compatibility

[neelnanda-io]

Seems like a very thorough PR, thanks a lot! Gated MLPs as a separate class seems correct to me. Do you know what's up with LLaMA being on hugging face, from someone who is NOT Meta? I wasn't aware of this and am concerned that's against Meta's terms and will be taken down (and thus would be a bad idea for TransformerLens to get involved), since my understanding is that Meta is still acting like they didn't open source the models(?)

…

On Tue, 4 Apr 2023, 8:40 pm 0amp, ***@***.***> wrote: Description Add support for all 4 LLaMA models using huggingface as a backend. Requires main branch of transformers repo and accelerate (pip install accelerate git+https://github.com/huggingface/transformers). Fixes #201 <#201> Type of change Please delete options that are not relevant. - New feature (non-breaking change which adds functionality) Checklist: - I have commented my code, particularly in hard-to-understand areas - I have made corresponding changes to the documentation - My changes generate no new warnings - I have added tests that prove my fix is effective or that my feature works (see LLaMA.ipynb demo) - New and existing unit tests pass locally with my changes - I have not rewritten tests relating to key interfaces which would affect backward compatibility ------------------------------ You can view, comment on, or merge this pull request online at: #234 Commit Summary - 03afd6b <03afd6b> add LLaMA support - 4ab9af2 <4ab9af2> Merge branch 'main' of https://github.com/neelnanda-io/TransformerLens into main - 9fe49d7 <9fe49d7> add LLaMA support - e960519 <e960519> add comments - 0effa5e <0effa5e> fix gpt norm bug File Changes (6 files <https://github.com/neelnanda-io/TransformerLens/pull/234/files>) - *A* demos/LLaMA.ipynb <https://github.com/neelnanda-io/TransformerLens/pull/234/files#diff-9aca493749aa4428eb2660775bc99a951e7243dbef0e0a7b485ea04de83bbb54> (2107) - *M* setup.py <https://github.com/neelnanda-io/TransformerLens/pull/234/files#diff-60f61ab7a8d1910d86d9fda2261620314edcae5894d5aaa236b821c7256badd7> (1) - *M* transformer_lens/HookedTransformer.py <https://github.com/neelnanda-io/TransformerLens/pull/234/files#diff-4bf18c93e88ee8550ce05a5e6debf1a2b5e6912dab5755daf0041309c0e26a2a> (29) - *M* transformer_lens/HookedTransformerConfig.py <https://github.com/neelnanda-io/TransformerLens/pull/234/files#diff-1ed779de36cf80ef997818ce48e8a22649727d2d1ded4ccf5e4a1dc64cf3c7f0> (1) - *M* transformer_lens/components.py <https://github.com/neelnanda-io/TransformerLens/pull/234/files#diff-d3f09bb699c3b05afb6d0cb1102d441eefc4d0f6c2aabdf96ff7d888c43c60aa> (78) - *M* transformer_lens/loading_from_pretrained.py <https://github.com/neelnanda-io/TransformerLens/pull/234/files#diff-49f3eaa073cf950519375c165b2972561dfd6188612e6076923201c52400ac12> (89) Patch Links: - https://github.com/neelnanda-io/TransformerLens/pull/234.patch - https://github.com/neelnanda-io/TransformerLens/pull/234.diff — Reply to this email directly, view it on GitHub <#234>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASRPNKKMBXCF2EH3Z5RRPPTW7R2JVANCNFSM6AAAAAAWTEXGGM> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[0amp]

Yup! I'm not sure what's up with the upload to huggingface. For some reason, the PR was merged into the main transformers branch and there are at least a half dozen other clones (or fine-tuned versions) on huggingface now anyways so I would bet that it probably won't get taken down. My understanding is that Meta has not said anything new about their license.

Also, is there some easy way to fix the dependencies so that the tests pass?

[neelnanda-io]

Hmm, can you raise an issue on the HuggingFace repo and ask about it, and tag the author of the PR? If it's explicitly not allowed by Meta, then I would favour not explicitly having LLaMa in from pretrained and instead having instructions on how to load it from the weights stored locally

…

On Wed, 5 Apr 2023, 01:50 Oam Patel, ***@***.***> wrote: Yup! I'm not sure what's up with the upload to huggingface. For some reason, the PR was merged into the main transformers branch and there are at least a half dozen other clones (or fine-tuned versions) on huggingface now anyways so I would bet that it probably won't get taken down. My understanding is that Meta has not said anything new about their license. Also, is there some easy way to fix the dependencies so that the tests pass? — Reply to this email directly, view it on GitHub <#234 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASRPNKOXGUAC725JY6JAZKTW7S6UHANCNFSM6AAAAAAWTEXGGM> . You are receiving this because you commented.Message ID: ***@***.***>

[0amp]

So there's this huggingface/transformers#21796. Looks like the merged pr is just setting up the LLaMA class and that individual people have uploaded the weights. Seems like the HF team isn't going to do anything about it and Meta hasn't said anything about it yet. Also, there's a script for converting meta weights included in the original PR huggingface/transformers#21796 so I could change this PR to just have people download + convert weights themselves.

[neelnanda-io]

Lol. Thanks for the context. Based on that this is definitely *not *legit, and I'd prefer if TransformerLens didn't explicitly have support to do something that violates Meta's terms (in particular, having `HookedTransformer.from_pretrained("llama-7b")` work, implicitly implies to a normal user that this *is* legitimate use, in my opinion). But I love the idea of having it be supported for anyone who already has access to the weights, that all sounds great and I think there's some interesting questions here. If you could edit the colab/docstring to have clear instructions on how to convert the legitimate weights, that would be amazing.

…

On Wed, 5 Apr 2023 at 23:13, Oam Patel ***@***.***> wrote: So there's this huggingface/transformers#21796 <huggingface/transformers#21796>. Looks like the merged pr is just setting up the LLaMA class and that individual people have uploaded the weights. Seems like the HF team isn't going to do anything about it and Meta hasn't said anything about it yet. Also, there's a script for converting meta weights included in the original PR huggingface/transformers#21796 <huggingface/transformers#21796> so I could change this PR to just have people download + convert weights themselves. — Reply to this email directly, view it on GitHub <#234 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASRPNKN5M3POXAYMQZAIDVTW7XVB3ANCNFSM6AAAAAAWTEXGGM> . You are receiving this because you commented.Message ID: ***@***.***>

[jbloomAus]

@neelnanda-io I think it sounds like @0amp has completed the PR per your requests, and it is documented/expects users to have legal access to the weights. However, due to the nature of the PR, I'll let you merge it, or if you reply here asking me to, I'll merge it.

[neelnanda-io]

Ah, sorry for dropping this! LGTM, thanks again Oam

…

On Tue, 11 Apr 2023, 23:35 Joseph Bloom, ***@***.***> wrote: @neelnanda-io <https://github.com/neelnanda-io> I think it sounds like @0amp <https://github.com/0amp> has completed the PR per your requests, and it is documented/expects users to have legal access to the weights. However, due to the nature of the PR, I'll let you merge it, or if you reply here asking me to, I'll merge it. — Reply to this email directly, view it on GitHub <#234 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ASRPNKKVFCWDUXLGNCIA5LDXAXMDTANCNFSM6AAAAAAWTEXGGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>