FastTokenizer for LLaMa

[theblackcat102]

Feature request

FastTokenizer support for LLaMa sentencepiece tokenizer.

Motivation

The offset_mapping is only available in FastTokenizer, it would be useful if there's support for this.

Your contribution

I have tried using existing sentencepiece based model as replacement. However hf conversation code means we are missing the byte fallback support

The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers

Which means out of vocabulary tokens are simply mapped to instead of using the byte mapping inside the vocab.

[sgugger]

Let's maybe wait for the LLaMa PR to be merged first?

[dongs0104]

it is fix on tokenizers

huggingface/tokenizers#1183

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.