[Whisper Tokenizer] Make decoding faster after adding timestamps

[sanchit-gandhi]

What does this PR do?

Following the update to the Whisper tokenizer to handle encoding/decoding timestamps (#26054), there is one line in the decoding which takes extremely long:

transformers/src/transformers/models/whisper/tokenization_whisper.py

Line 614 in f94c9b3

token_ids = [token for token in token_ids if token not in timestamp_ids]

Here we do an order N * M operation to filter out all the timestamp tokens, where N is the length of the token ids, and M the number of timestamp tokens (for each token, check whether it’s in the timestamp token list).

In practice, this is causing decoding to take extremely long for typical validation sets, e.g. LibriSpeech test clean took ~30 mins for the tokenizer to decode on a TPU v3 (which has lots of CPU power to run this operation).

This PR switches the timestamp filtering to a regex string operation, which in a toy benchmark was a factor of > 2000 faster. Would love to hear from @ArthurZucker whether we're happy to sacrifice a bit of readability for this speed-up!

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[ArthurZucker]

Is this faster than storing all the timestamp tokens as a property, and use it to filter the tokens?

[sanchit-gandhi]

Yep it was significantly faster once scaled to large datasets and long sequence lengths (>5k samples with 256 seq len)

[ArthurZucker]

I see! Thanks for fixing this then 😉