How to solve the issue of hot words?

[maowandong]

How to solve the issue of hot words with whisper? The cost of individually training each time hot words are added is somewhat high.

[ryanheise]

You can use the --initial_prompt " My prompt..." option to prompt it with a sentence containing your hot words. The idea of the prompt is to set up Whisper so that it thinks it has just heard that text prior to time zero, and so the next audio it hears will now be primed in a certain way to expect certain words as more likely based on what came before it. If you know what kind of speech you expect to pass into Whisper, try to think of a prompt that could plausibly come before it that would set it up with the appropriate context to predict hot words in what comes next.

[maowandong]

@ryanheise Thanks for your answer. To add some information: I am fine-tuning the recognition performance for Chinese. When recognizing Chinese song titles (Some recent hit songs ), names, and other proper nouns, it always misidentifies. "It's difficult to ascertain the scenario and intent of the language in advance."

[ryanheise]

It's difficult to ascertain the scenario and intent of the language in advance.

There is nothing at all that is characteristic about your use case? Are you actually transcribing purely random types of content? E.g. Sometimes you are transcribing short audio commands via a mobile's built-in microphone like "Open email", other times you are transcribing a lecture, other times you are transcribing a list of random words, other times you are transcribing phone calls, etc. etc. It's unlikely that you know nothing in advance about the context of this speech.

If you don't want to perform any costly training, prompts can still be effective. The prompt doesn't need to be perfect so just try a sentence that includes a list of proper nouns that you want to improve the accuracy of. Once you exhaust the effectiveness of that, then you could turn to training / fine turning.

[maowandong]

@ryanheise Thanks, I will try prompt option to test the performace. In fact, my scenario is somewhat similar to Alexa, there are many skilles, e.g Play music, control home appliances, play movies and TV shows, and so on.

[zachx121]

@maowandong hi, did this prompt method work out? I hvae just come into the same situation.