Broken (repeating) subtitles

[cupiditys]

On each video I upload, the subtitles repeat for about 4 minutes before coming back. I tried with LargeV2 and V3 models.

[umlx5h]

This is a problem I have been aware of for some time.
What runtime are you using, CPU, Vulkan or CUDA?

Re-enabling ASR will recreate the subtitles from the playback position, so that may help.
(Subtitles after the start position will be deleted, but before will remain.)

I reproduced it even when I fed the WAV files directly to whisper.net, so the cause may be on the library side (whisper.net or whisper.cpp) .

debug code: output wav

LLPlayer/FlyleafLib/MediaPlayer/SubtitlesASR.cs

Lines 211 to 216 in b82d422

	// Output wav file for debugging
	//using (FileStream fs = new($"subtitlewhisper-{chunkCnt}.wav", FileMode.Create, FileAccess.Write))
	//{
	// waveStream.WriteTo(fs);
	// waveStream.Position = 0;
	//}

I will do a detailed investigation later.

[cupiditys]

I was using CPU in the gif, tried using cuda and it has the same issue (seems worse really, but unsure).

re-enabling does work but just a bit tedious since you have to do it multiple times

[umlx5h]

Thanks, I donn't have a nvidia gpu so that helps.
I have confirmed it reproduces by Vulkan using Radeon as well, I will investigate it later.

[umlx5h]

I have confirmed that when I feed a specific WAV file directly to whisper.net, it does not reproduce on the CPU, but definitely on Vulkan.

Apparently there are two types of repeating problems: the one that occurs on the CPU must be a problem on the whisper.cpp side, which is the problem of the previous subtitle repeating when silent. which should be resolved when the next audio comes in.

The problem that occurs with vulkan and cuda is that the previous subtitle repeats with audio present, which should be fixed.

I will check whether whisper.net or whisper.cpp has the vulkan/cuda problem.

---

I bought an nvidia GPU and have confirmed that I can reproduce this with a combination of CPU/CUDA and LargeV3.

I need to set up an environment where I can run whisper.cpp directly in a Windows environment.

[umlx5h]

It seems to be a problem on the whisper.cpp or original whisper models, so we will have to wait patiently for it to be fixed in an upgraded version.

sandrohanea/whisper.net#337
ggerganov/whisper.cpp#2191

There are several ways to deal with this.

• Change to a different model.
  • More likely to occur with LargeV3 and LargeV3 Turbo; less likely with LargeV2
• Change runtime options
  • Change to Cuda or Vulkan or CPU
• Re-run ASR from the repeat point.
  • If ASR is enabled and you select it again, it will be re-generated from the playback point.
• Set ASR Chunk Size to a smaller size in the settings.
  • There is a possibility to eliminate repeats because the whisper is run again at short intervals.