Add support for non-rust implemented tokenization for __getitem__ method.

[jacklanda]

Overview

This PR is going to add a support for the usage scenario of "getting a slice from the batch-tokenized sequences".

Without this PR, it seems to raise KeyError with the following message KeyError: 'Indexing with integers (to access backend Encoding for a given batch index) is not available when using Python based tokenizers'

P.S. The above scenario could be reproduced by using some models new uploaded but not support to Rust-implemented tokenization, such as fnlp/moss-moon-003-sft. Also we can run the following examplar script for reproducing this issue:

# test script `/home/workspace/test.py` for this PR. 
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("fnlp/moss-moon-003-sft", trust_remote_code=True)
tok.add_special_tokens({"pad_token": "[PAD]"})

texts = ["Today is a good day!", "It's a good idea!", "How's going?"]
batch_tok = tok(texts, padding=True)
print(batch_tok[0:3])  # report `KeyError` here

Error Message

Traceback (most recent call last):
  File "/home/workspace/test.py", line 8, in <module>
    print(batch_tok[0:3])  # report `KeyError` here
  File "/home/app/anaconda3/envs/test/lib/python3.9/site-packages/transformers/tokenization_utils_base.py", line 242, in __getitem__
    raise KeyError(
KeyError: 'Indexing with integers (to access backend Encoding for a given batch index) is not available when using Python based tokenizers'

All in all, I think it seems useful to implement getitem method behind it in Python side :)

Note that this PR is associative with the previous closed one.
#23645

[jacklanda]

It seems that failed in workflow due to Read Timeout on T5-relevant testing.
How can I rerun for this?

[ArthurZucker]

We can re-run that for you 😉

[ArthurZucker]

I am in favor of these changes as I can confirm that fast tokenizer in python support this indexing while slow ones don't. This is narrowing further the gap between the two.
Thanks for adding this, let's just wait for the tests to pass + can you update the error message since this will allow indexing with integers.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[jacklanda]

Request for review :)

[amyeroberts]

Thanks for adding this!

[amyeroberts]

@jacklanda Could you update the error message as requested by @ArthurZucker?

[jacklanda]

@jacklanda Could you update the error message as requested by @ArthurZucker?

@amyeroberts Have updated the mentioned error messages by @ArthurZucker
Thanks.

[jacklanda]

Ask for final review :)

[amyeroberts]

Thanks again for adding!