Add TokenClassification for Mistral, Mixtral and Qwen2

[josephenguehard]

What does this PR do?

This PR adds token classification for Llama and its extensions, where code from Llama was copied : Gemma, Mistral, Mixtral, Persimmon, Qwen2, Qwen2MoE, StableLm and StarCoder2:

• GemmaForTokenClassification
• LlamaForTokenClassification
• MistralForTokenClassification
• MixtralForTokenClassification
• PersimmonForTokenClassification
• Qwen2ForTokenClassification
• Qwen2MoeForTokenClassification
• StableLmForTokenClassification
• StarCoder2ForTokenClassification

Note: The original PR was intended for Mistral only. Llama has been added as a request from @KoichiYasuoka, and the others models have been added as copying most of Llama setting.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

• text models: @ArthurZucker and @younesbelkada

[ArthurZucker]

Hey! Thanks for the PR, could you explain the motivation behind this? No checkpoints are publicly available, and we usually first oopen a feature request and then integrate it if the community shows interest if there are no official checkpoints!

[KoichiYasuoka]

Thank you @josephenguehard for

• MistralForTokenClassification

since I've already released https://huggingface.co/KoichiYasuoka/Swallow-MS-7b-upos (using trust_remote_code=True). And would you kindly include

• LlamaForTokenClassification

from #26521 for my https://huggingface.co/KoichiYasuoka/Swallow-7b-plus-upos ?

[josephenguehard]

Hi Both!

Thank you for your replies! I will open a separate issue to discuss adding these classes.

@KoichiYasuoka I'd be happy to add LlamaForTokenClassification! I'll try to have this done by the end of the week.

[josephenguehard]

The feature request issue linked to this PR is #29940

[josephenguehard]

@KoichiYasuoka I've added Llama support as well as its extensions (models where code from Llama was copied), and I think the PR is ready for review.

One point where I'd like to have a discussion is on the Dropout. I see in your implementation that you include a Dropout layer, which follows the BertForTokenClassification.
However, I argue that this implementation actually follows BertForSequenceClassification, which also has a Dropout layer. Moreover, LlamaForSequenceClassification does not have such Dropout layer, so I think we should not include it for LlamaForTokenClassification either. But happy to get other's opinions!

[KoichiYasuoka]

Well, Dropout layer... My Implementation follows GPT2ForTokenClassification which was released with #13290 at Transformers v4.10.0. But I'm vague how GPT2 and LLaMA differ at that point...

[josephenguehard]

It's very easy for me to add Dropout if there is a consensus on it.
FYI I'm away and won't have access to my laptop until April 15th, will pick this up then.

[KoichiYasuoka]

Yesterday @SeanLee97 of @WhereIsAI released two models:

• https://huggingface.co/WhereIsAI/billm-llama-7b-conll03-ner
• https://huggingface.co/WhereIsAI/billm-mistral-7b-conll03-ner

They use their own package BiLLM with LlamaForTokenClassification and MistralForTokenClassification.

[SeanLee97]

Yesterday @SeanLee97 of @WhereIsAI released two models:

• https://huggingface.co/WhereIsAI/billm-llama-7b-conll03-ner
• https://huggingface.co/WhereIsAI/billm-mistral-7b-conll03-ner

They use their own package BiLLM with LlamaForTokenClassification and MistralForTokenClassification.

Thank you for mentioning our released models. Happy to see that official 🤗 transformers supports token classification.
However, based on our experiments in the paper (https://arxiv.org/abs/2310.01208), LLMs with the causal mask cannot perform well on token classification tasks. Thus, we convert the causal attention mask from uni- to bi-directional. This change can improve the token classification performance significantly. Hopefully this feature can be added to the official transformers in the future.

[KoichiYasuoka]

Thank you @SeanLee97 but bi-directional seems not causal... IMHO we need causal option in LlamaConfig as used in XLMConfig, where causal=True (default) means uni-directional attention in contrast of causal=False bi-directional.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[KoichiYasuoka]

I'm waiting for your review @ArthurZucker

[ArthurZucker]

Thanks! Let's just add copied from for the test and good to go.
Seems to have decent traction, which is what I was waiting for!

[ArthurZucker]

tests can also be copied from!

[ArthurZucker]

also make sure to resolve the conflics

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[josephenguehard]

@ArthurZucker I've added Copied from for the tests and resolved the conflicts.

[younesbelkada]

Thanks a lot ! LGTM with only one nit - can you propagate these changes to all models with make fix-copies (once the suggestion accepted)

[josephenguehard]

Done @younesbelkada 👍

[younesbelkada]

Clean, thanks !