Transcription starts spamming descriptive tags and is no longer responsive to audio input

Hey, I've been encountering an issue were the model seems to start spamming descriptive tags. This only happens after a few seconds of silence and after some things have already been said. But once it starts going, it doesn't stop and is no longer responsive to audio input. For the whole last part (starting at about 62360), I was talking again. But that did not seem to be picked up anymore.

This happens on the most recent version of the main branch (780e2be, with the supposed fix for incorrectly clearing the audio segment buffer).

I tried messing around with different models and languages, but wasn't able to find a configuration that gets rid of this. I've also tried the same model with the faster-whisper engine in WhisperLiveKit and the issue didn't occur there.

python simulstreaming_whisper_server.py --language en --model_path ./small.pt --task transcribe --host l
ocalhost --port 43001 --log-level WARNING --vad
WARNING Whisper is not warmed up. The first chunk processing may take longer.
1260 1720  And that's it for the
2260 2720  test.
6660 5580  [typing]
8000 8000  [
9340 9340 ty
10340 10340 ping]
11340 11680  So let
12340 12980 's see if it's working at all
13740 13980 .
14740 14980  Oh.
16540 16840  [typing]
22240 21220  [typing]
23660 24080  [typing]
24660 25080  [typing]
28495 31675  [typing]
32075 32695  [typing]
38765 41925  Okay, now it seems to be working.
43535 43995  [typing]
44625 45405  [typing]
47365 48065  At least let's
48365 49065  not running away
49445 50065  with
50445 51495  audio tags
52005 52495 .
53005 53495  [typing
54005 54495 ]
56745 55765  [typing]
58035 57395  [typing]
59265 58395  [typing]
60360 59620  [typing]
61360 61740  [typing]
62360 62740  [typing]
64610 64910  [typing] [typing] [typing] [typing] [typing] [typing]
67520 66540  [ty
68520 70585 ping]
72885 74125  [ty
77180 82560 ping] [
86410 88650 typing] [typing] [typing] [typing] [typing] [typing] [typing] [ty
88650 92900 ping] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [typing] [ty
93910 93900 ping] [typing] [typing] [typing] [typing] [
95640 95640 typing] [typing] [typing] [typing] [typing] [ty
96640 97350 ping] [typing] [typing] [typing] [typing] [
100160 100180 typing] [typing] [typing] [typing] [typing]
101160 101180  [typing] [typing] [typing] [typing] [typing
102160 102180 ] [typing]
103830 103180  [typing] [typing] [typing] [typing] [typing
104830 104325 ] [typing] [typing] [typing] [typing] [ty
106975 105325 ping] [typing] [typing] [typing] [typing] [ty
107975 106325 ping] [typing] [typing] [typing] [typing] [ty
108975 107325 ping] [typing] [typing] [typing] [typing] [ty

Btw, love the project :) The perfomance increase is amazing and I would love to get this working.

[Gldkslfmsd]

hi, try --vac option. It should apply the VAD iterator to skip the non-voice sounds.
The --vad option does nothing, it should be removed. Sorry.

Thanks!

Nice that fixed it for me. I created a quick pull request that removes the --vad argument if you're interested in that.

[Gldkslfmsd]

reopening until I'll merge the PR #9 . Which I will once I'll find time

[dvgenusa]

Hello,
I'm trying this project (thanks for it), and I had the same problem.
I mean, the transcription starts spamming descriptive tags and no response to new audio input.
I corrected it by using the -vac option, and now I can see that it's detecting voice and silences properly.
But the problem I see now is that, when I start speaking, first or some more words are omited.

Eg.: In this case, after been in silence, I said "items 40 41" but "items" is omited and the transcription is "40 41"

INFO    no online update, only VAD. nonvoice
INFO    no online update, only VAD. nonvoice
INFO    no online update, only VAD. nonvoice
INFO    no online update, only VAD. voice
INFO    no online update, only VAD. voice
INFO    no online update, only VAD. voice
INFO    no online update, only VAD. voice
INFO    no online update, only VAD. voice
INFO    Finish
INFO    Trimming context
INFO    Context text:
INFO    Context after trim:  (len: 4)
INFO    Decoding loop starts
INFO    End of decoding loop
INFO    Output:  40 41
39612 42112  40 41
INFO    no online update, only VAD. nonvoice
INFO    no online update, only VAD. nonvoice
INFO    no online update, only VAD. nonvoice
INFO    no online update, only VAD. nonvoice

I tried to play with "--min-chunk-size" argument (I set it from 0.5 to 2 to try to wait for a "group of sounds" to be transcribed) but it seems there is no effect. 
I also played with "--vac-chunk-size", with a low value ("--vac-chunk-size", "0.01") to activate the transcription as soon as possible and not to loose any sound of the first word, without success either.

What would be the correct prameters and/or configuration to reactivate the transcriptions once the VAC detects voice and not to lose any word?

Thanks.

[Gldkslfmsd]

hi, yes, there's a bug that's fixed in PR #13 . I need to review and merge. Can you try to test that PR and let me know? That would be helpful.

Thanks!

[dvgenusa]

I didn't see that PR.
I'll try to test it and compare with current version.

I'll let you know.

Thanks!

[dvgenusa]

I tried that PR and it seems it fix the problem.
I haven't tested it a lot but, I've tried the same (keep in silence few seconds and start speaking again) and the transcription of the first word is now ok.

I also notice that now the output is json format. Ok, not problem.

I think I'll do more test. Anything I'll let you know.

Thanks!

[Gldkslfmsd]

I tried that PR and it seems it fix the problem. I haven't tested it a lot but, I've tried the same (keep in silence few seconds and start speaking again) and the transcription of the first word is now ok.

Thanks!

I also notice that now the output is json format. Ok, not problem.

We're working on it with Marko, it was meant for another PR. I'm thinking about changing the format to json, or have it optional, for backdependency and simpler use of the applications that don't need the structured output. What do you think?

I think I'll do more test. Anything I'll let you know.

Thanks!

Thanks. You can also check with this --logdir parameter that is supposed to save all processed audio, context and generated part in a dir. It should be useful for inspection and spotting the problems with and without the fix. But I think it's in a dev branch translate-server so far, I'm quite behind and busy, sorry

[dvgenusa]

We're working on it with Marko, it was meant for another PR. I'm thinking about changing the format to json, or have it optional, for backdependency and simpler use of the applications that don't need the structured output. What do you think?

Sorry but I couldn't test it more.
The output in json format is nice, but I think that, as you indicated, it would be a good idea to have it as an optional.
I mean, two types of output: the simple one (start timestamp, end timestamp, transcription) and the detailed one (json format with more details in fields.