Inconsistent behavior between tokenizer and fast tokenizer

[xuzhenqi]

System Info

• transformers version: 4.36.2
• Platform: Linux-4.18.0-193.6.3.el8_2.v1.4.x86_64-x86_64-with-glibc2.29
• Python version: 3.8.10
• Huggingface_hub version: 0.19.4
• Safetensors version: 0.4.0
• Accelerate version: 0.25.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.0.1+cu118 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Using GPU in script?: No
• Using distributed or parallel set-up in script?: No

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("codellama/CodeLlama-7b-hf", trust_remote_code=True, use_fast=False)
fast_tokenizer = AutoTokenizer.from_pretrained("codellama/CodeLlama-7b-hf", trust_remote_code=True, use_fast=True)
prompt = "▁<PRE>//"
inputs = tokenizer(prompt, return_tensors="pt")
print(f"tokenizer ids: {inputs.input_ids}")
inputs = fast_tokenizer(prompt, return_tensors="pt")
print(f"fast tokenizer ids: {inputs.input_ids}")

This scripts will output:

tokenizer ids: tensor([[    1, 32007,   458]])
fast tokenizer ids: tensor([[    1, 32007,   849]])

In the tokenizer.json from the model folder, we can see:

"//": 458,
"▁//": 849,

Fast tokenizer probably ignores the <PRE> token, is it a correct behavior?

Expected behavior

Fast tokenizer should be consistent with normal tokenizer.

[ArthurZucker]

Thanks for reporting! This is pretty much a known bug but will be fixed by the likes of #26678 (when propagated to Llama)

[ArthurZucker]

#28881 will fix this issue!