Prompt length (244 characters or tokens?)

[RafalCer]

Hi,

Whisper's prompt is an awesome feature that allows for accurate handling of tricky words, such as proper nouns denoting company names (e.g. Whatagraph). However, it is a bit unclear from the documentation what the maximum length of the prompt is. As you can see in the screenshot below, the documentation first states that "Whisper only considers the first 244 tokens of the prompt."; however, the next paragraph mentions that "this technique is limited to only 244 characters".

Has anyone explored whether the prompt is limited to the first 244 characters (letters) or tokens (units determined by the tokenizer)?

Thank you!

[glangford]

Some evidence points to the limit being 224 tokens but I don't think this has been discussed in the past. Here is some more conflicting information.

FYI from the Whisper prompting guide

In addition, the prompt is limited to only 224 tokens. If the prompt is longer than 224 tokens, only the final 224 tokens of the prompt will be considered; all prior tokens will be silently ignored. The tokenizer used is the multilingual Whisper tokenizer.

https://cookbook.openai.com/examples/whisper_prompting_guide

That said, this space is for the open source repo of whisper and it appears you are using the online API. So I would consider asking this question over at the community developer forum for the online version of whisper.

[RafalCer]

Thank you for the response! I'm actually using the open source version, but the aforementioned documentation was the only (official) piece of information regarding prompting I could find.

Also, thank you for pointing me to the prompting guide. It consistently mentions that the limit is 244 tokens rather than characters. I also had a closer look at the source code and it seems like the prompt is created and truncated only after the prompt input is encoded, so I'm pretty sure that the limit should apply for tokens.

I experimented a bit with Whisper prompting by transcribing an English file with 2 minutes of speech. At around 01:00, a last name is pronounced which Whisper is unable to transcribe properly on its own, but does perfectly well when it is prompted with the spelling of the last name. In my experiments I tried cluttering the prompt with some sentences from Wikipedia (~700 characters, ~140 tokens), and inserting the problematic last name in the beginning, middle, end, and some random positions of the prompt. Interestingly, the prompt only helped if I placed the last name as one of the last ~5 words in the prompt; in other cases, it would continuously misspell the last name even though it was clearly within the non-truncated range. I also tried cutting down the entire prompt to just 1 sentence of ~80 characters, but the last name would still be spelled incorrectly unless I placed it at the end of my prompt.

I know that there is quite a bit of randomness in Whisper and especially its prompting feature, but I'm curious if anyone has observed or managed to crack how to use the prompting properly.

[JoaquinKeller]

https://platform.openai.com/docs/guides/speech-to-text/prompting in the API doc they say:

• «The model will only consider the final 224 tokens of the prompt and ignore anything earlier»
• «It's important to keep in mind that Whisper only considers the first 244 tokens of the prompt»

So is it the final 224 tokens or the first 244 tokens of the prompt that are taken into account?

[ClaireCJS]

224 isn't enough. It should at least be enough to contain song lyrics for songs with lots of words. Some of us are transcribing lyrics to songs into TIMED lyrics, and whisper is awful at accuracy and needs to be primed with the official lyrics of the song.

Songs have more than 224 words in them. Especially rap, which I don't really listen to, but which should be considered.

These files aren't going to be "chunked up"

[EtienneAb3d]

@ClaireCJS
I see how to build a process:

1. transcribe to get a first timed chunking, even if bad words
2. sync the bad transcription with the not perfect official lyrics to get the vocab using https://github.com/EtienneAb3d/WhisperTimeSync
3. split the audio according to text chunks
4. transcribe chunks using prompts fed with the corresponding vocab
5. merge the result to get the final timed transcription

[RafalCer]

@ClaireCJS

I agree that 224 is quite limiting, especially since 224 tokens is definitely less than 224 words. But note that Whisper does not transcribe any duration of audio in one go. Under the hood it chunks the audio into 30 second chunks and processes those. If you play around a bit with the prompt used at inference of every chunk, you could probably fit all the relevant words for that chunk and prime the transcriber properly. Needless to say, some estimation of which words to use in which chunk is needed, and then it's not a given that the priming will lead to a correct transcription.

[ClaireCJS]

@ClaireCJS I see how to build a process:

1. transcribe to get a first timed chunking, even if bad words
2. sync the bad transcription with the not perfect official lyrics to get the vocab using https://github.com/EtienneAb3d/WhisperTimeSync
3. split the audio according to text chunks
4. transcribe chunks using prompts fed with the corresponding vocab
5. merge the result to get the final timed transcription

So, more about this:

Step #2.

WhisperTimeSync absolutely removes lines of spoken text if it can't match them to the TXT file.

Which is exactly what I'm trying to avoid.

And merging is no easy task.

[EtienneAb3d]

The goal of step 2 is not to produce a perfect match for all, but to provide you with the correct vocab as many times as it's possible.

[EtienneAb3d]

@ClaireCJS
For each chunk, if you add in the prompt, for example, the last 150 official words found on step 2, and not only the words for this chunk, you should have a good probability to get all the words you need to process this chunk.

[ClaireCJS]

That sure is a lot of steps compared to just allowing the acceptance of more tokens 😉

But thanks for pointing out WhisperTimeSync. I couldn't remember it's name, couldn't find it yesterday when I was looking for that concept, and wasn't sure if that concept had been implemented by someone, or if I had dreamed that it had been implemented (version control is hard on your own mind! lol). Now I know it's real, it's name, and will check it out 🎉