Any way to use real-time audio input recognition?

[toboil-features]

I tried to use "./build/bin/whisper-stream" as claimed in README.md. The thing is, on latest commit I can't find any binary with "stream" word to be laying in /build/bin folder.
So, how to use it?

[argan719]

me either,

cmake -B build -DWHISPER_SDL2=ON cmake --build build --config Release ./build/bin/whisper-stream -m ./models/ggml-base.en.bin -t 8 --step 500 --length 5000

Output Result

init: found 1 capture devices:
init: - Capture device #0: 'Audio Adapter (Unitek Y-247A) Mono'
init: attempt to open default capture device ...
init: obtained spec for input device (SDL Id = 2):
init: - sample rate: 16000
init: - format: 33056 (required: 33056)
init: - channels: 1 (required: 1)
init: - samples per frame: 1024
whisper_init_from_file_with_params_no_state: loading model from 'models/ggml-base.en.bin'
whisper_init_with_params_no_state: use gpu = 1
whisper_init_with_params_no_state: flash attn = 0
whisper_init_with_params_no_state: gpu_device = 0
whisper_init_with_params_no_state: dtw = 0
whisper_init_with_params_no_state: devices = 1
whisper_init_with_params_no_state: backends = 1
whisper_model_load: loading model
whisper_model_load: n_vocab = 51864
whisper_model_load: n_audio_ctx = 1500
whisper_model_load: n_audio_state = 512
whisper_model_load: n_audio_head = 8
whisper_model_load: n_audio_layer = 6
whisper_model_load: n_text_ctx = 448
whisper_model_load: n_text_state = 512
whisper_model_load: n_text_head = 8
whisper_model_load: n_text_layer = 6
whisper_model_load: n_mels = 80
whisper_model_load: ftype = 1
whisper_model_load: qntvr = 0
whisper_model_load: type = 2 (base)
whisper_model_load: adding 1607 extra tokens
whisper_model_load: n_langs = 99
whisper_model_load: CPU total size = 147.37 MB
whisper_model_load: model size = 57.10 MB
whisper_model_load: ERROR not all tensors loaded from model file - expected 245, got 3
whisper_init_with_params_no_state: failed to load model
Segmentation fault

[danbev]

The thing is, on latest commit I can't find any binary with "stream" word to be laying in /build/bin folder.

Make sure you are building with -DWHISPER_SDL2=ON so that the examples that depend on it actually get built.

[danbev]

@argan719 It looks like something is wrong with the model you are using, as the ggml.base.en model should have 245 tensors and not 3. Can you try removing this model and downloading it again using:

$ rm models/ggml-base.en.bin 
$ ./models/download-ggml-model.sh base.en
Downloading ggml model base.en from 'https://huggingface.co/ggerganov/whisper.cpp' ...

[peardox]

The thing that's really bad from my perspective in the above output is why does wh8isper_init try continuing when obviously it's about to fail

Segmentation fault isn't really good style for handling what is obviously a null pointer... This is a common issue in whisper, null pointers seldom checked for

[danbev]

Segmentation fault isn't really good style for handling what is obviously a null pointer... This is a common issue in whisper, null pointers seldom checked for

This is not checked in the stream example and is a bug. The handling in whisper_init_from_file_with_params does check the ctx for null, but the stream example does not check this like other examples like whisper-cli.

[argan719]

I resolved above problem thanks to you. thank you:)
But I wanna know this is work in raspberrypi4

When I run this on my rasp, and 3.5mm mic. it doesn't work like your demo.
`./build/bin/whisper-stream -m ./models/ggml-base.en.bin -t 8 --step 250 --length 5000 --capture 2'

Is it possible to real-time streaming STT in raspi? I changed step or other option but didn't work.

I turned BBC News, and slow version too, but this is result.

I think this stream code make bottleneck..

[danbev]

Is it possible to real-time streaming STT in raspi?

I don't have access to a raspi to try this out so I've not tried this myself, but there is a discussion thread that show how this can be done (just in case you had not seen it already).

[toboil-features]

After explicitly telling it to compile with SDL, it compiled whisper-stream.

cmake -B build -DWHISPER_SDL2=ON -DGGML_VULKAN=1
cmake --build build -j --config Release