Process exits when language detection is enabled

[katya4oyu]

I added set_language(Some("auto")) to the full_usage example and ran it(whisper-rs v0.10.0, MBP M1 Max, large-v2 model).

    let ctx = WhisperContext::new_with_params(
        &whisper_path.to_string_lossy(),
        WhisperContextParameters::default()
    ).expect("failed to open model");
    let mut state = ctx.create_state().expect("failed to create key");
    let mut params = FullParams::new(SamplingStrategy::default());
    params.set_progress_callback_safe(|progress| println!("Progress callback: {}%", progress));

    params.set_language(Some("auto")); // Add

    let st = std::time::Instant::now();
    state
        .full(params, &samples)
        .expect("failed to convert samples");
    let et = std::time::Instant::now();

The process terminated with the following message:

...
whisper_model_load: model size    = 3117.02 MB
whisper_init_state: kv self size  =  220.20 MB
whisper_init_state: kv cross size =  245.76 MB
whisper_init_state: compute buffer (conv)   =   30.92 MB
whisper_init_state: compute buffer (encode) =  212.36 MB
whisper_init_state: compute buffer (cross)  =    9.32 MB
whisper_init_state: compute buffer (decode) =   99.17 MB

Process finished with exit code 139 (interrupted by signal 11:SIGSEGV)

When I set Some("en") or Some("ja"), it works correctly,
but Some("auto"), None, set_detect_language(true) also finished the process.

[tazz4843]

Yep I noticed this bug last night with an unrelated PR. Don't have time to fix until later today, but it's an easy fix from what I can tell.

[tazz4843]

Turns out I had no clue what it was, I think it's a safe bet this is an upstream bug. I'll do some digging and see.

[tazz4843]

I can narrow the segfault down to this line of code: https://github.com/ggerganov/whisper.cpp/blob/master/whisper.cpp#L3503 but can't find much more since debug symbols are always stripped for some reason.

[Andreybest]

Hey, got the same problem with language detection.
Im no good at CPP, but from what I've tried to debug, issue is supposed to be here: https://github.com/ggerganov/whisper.cpp/blob/d38af151a1ed5378c5a9ae368e767ed22c8ab141/whisper.cpp#L3530C21-L3530C21

Looks like it takes ctx.state (which is null in this context - from my debugger), and state variable is full - as a function name (whisper_decode_with_state) suggests.

From what I see in the latest code of whisper.cpp, is is now targeted directly to state instead of ctx.state:
https://github.com/ggerganov/whisper.cpp/blob/022756a87204cd06c5d58f67b3708b550dcc38b0/whisper.cpp#L3490C20-L3490C20
Specifically: ggerganov/whisper.cpp#1519

So update to latest version of whisper.cpp should be the fix to this issue.

@tazz4843 @katya4oyu

I'll try to update whisper.cpp on sys crate, just to understand how it works and to help you and me :)

[tazz4843]

I ran into this myself, I'll be doing some testing to see if this is resolved or not on my service.

[WenqingZong]

Hi, I got the same problem and google leads me to here.

It seems the fix will be published on 0.10.1, currently I'm using whisper-rs = { git = "https://github.com/tazz4843/whisper-rs", branch = "master", features = ["cuda"]} in my Cargo.toml file, but cannot compile it due to this issue:

root@8a0b02512257:/workspaces/rs-whisper-test# cargo clean && cargo build --release
     Removed 1044 files, 382.1MiB total
   Compiling proc-macro2 v1.0.78
   Compiling glob v0.3.1
   Compiling unicode-ident v1.0.12
   Compiling prettyplease v0.2.16
   Compiling libc v0.2.153
   Compiling rustix v0.38.31
   Compiling bitflags v2.4.2
   Compiling cfg-if v1.0.0
   Compiling regex-syntax v0.8.2
   Compiling either v1.10.0
   Compiling linux-raw-sys v0.4.13
   Compiling minimal-lexical v0.2.1
   Compiling memchr v2.7.1
   Compiling bindgen v0.69.4
   Compiling home v0.5.9
   Compiling rustc-hash v1.1.0
   Compiling shlex v1.3.0
   Compiling lazycell v1.3.0
   Compiling log v0.4.21
   Compiling cc v1.0.89
   Compiling lazy_static v1.4.0
   Compiling fs_extra v1.3.0
   Compiling rs-whisper-test v0.1.0 (/workspaces/rs-whisper-test)
   Compiling hound v3.5.1
   Compiling libloading v0.8.2
   Compiling itertools v0.12.1
   Compiling clang-sys v1.7.0
   Compiling nom v7.1.3
   Compiling cmake v0.1.50
   Compiling quote v1.0.35
   Compiling syn v2.0.52
   Compiling regex-automata v0.4.6
   Compiling which v4.4.2
   Compiling cexpr v0.6.0
   Compiling regex v1.10.3
   Compiling whisper-rs-sys v0.8.1 (https://github.com/tazz4843/whisper-rs?branch=master#9861dfdb)
   Compiling whisper-rs v0.10.1 (https://github.com/tazz4843/whisper-rs?branch=master#9861dfdb)
error: linking with `cc` failed: exit status: 1
  |
  = note: LC_ALL="C" PATH="/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/vscode/vscode-server/bin/linux-x64/019f4d1419fbc8219a181fab7892ebccf7ee29a2/bin/remote-cli:/root/.cargo/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/cloud/3rdparty/ffmpeg/bin" VSLANG="1033" "cc" "-m64" "/tmp/rustcXdkPYN/symbols.o" "/workspaces/rs-whisper-test/target/release/deps/rs_whisper_test-b84891d36382a851.rs_whisper_test.47e03ed10f129cdd-cgu.0.rcgu.o" "/workspaces/rs-whisper-test/target/release/deps/rs_whisper_test-b84891d36382a851.rs_whisper_test.47e03ed10f129cdd-cgu.1.rcgu.o" "/workspaces/rs-whisper-test/target/release/deps/rs_whisper_test-b84891d36382a851.rs_whisper_test.47e03ed10f129cdd-cgu.2.rcgu.o" "/workspaces/rs-whisper-test/target/release/deps/rs_whisper_test-b84891d36382a851.rs_whisper_test.47e03ed10f129cdd-cgu.3.rcgu.o" "/workspaces/rs-whisper-test/target/release/deps/rs_whisper_test-b84891d36382a851.4wpbjtcnz2vuukzo.rcgu.o" "-Wl,--as-needed" "-L" "/workspaces/rs-whisper-test/target/release/deps" "-L" "/usr/local/cuda/lib64" "-L" "/opt/cuda/lib64" "-L" "/workspaces/rs-whisper-test/target/release/build/whisper-rs-sys-d6c814e72fa667d0/out/build" "-L" "/workspaces/rs-whisper-test/target/release/build/whisper-rs-sys-d6c814e72fa667d0/out" "-L" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/workspaces/rs-whisper-test/target/release/deps/libwhisper_rs-5c314781444888c4.rlib" "/workspaces/rs-whisper-test/target/release/deps/libwhisper_rs_sys-fa390a7b2f6f0f90.rlib" "/workspaces/rs-whisper-test/target/release/deps/libhound-42a21be6d88a314f.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-66d8041607d2929b.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-a57e2388c0aea9b1.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-dcd9be90ae2cb505.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libmemchr-516789932d161b4e.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-1ff34b0cf871cb60.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-0c110dd0650d6cb7.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-a6e97aae2681ad8f.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd_detect-b93dac2525ec4d1e.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-ce1d65fb391ae98b.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-8933a2fb54d88492.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-306712ebb1ee1a3f.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-349c574f342b0d30.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-65c422a3ad95273d.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-7e6330a6c0cb9441.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-39c59240bfdfab27.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-e9d126c51bb8b2bb.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-5af394d9b1f07bdc.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-693a8f23970c5917.rlib" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-13fc9d1ed9c7a2bc.rlib" "-Wl,-Bdynamic" "-lstdc++" "-lcublas" "-lcudart" "-lcublasLt" "-lculibos" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-L" "/root/.rustup/toolchains/1.76.0-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-o" "/workspaces/rs-whisper-test/target/release/deps/rs_whisper_test-b84891d36382a851" "-Wl,--gc-sections" "-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs"
  = note: /usr/bin/ld: /workspaces/rs-whisper-test/target/release/deps/libwhisper_rs_sys-fa390a7b2f6f0f90.rlib(ggml-cuda.cu.o): in function `ggml_cuda_pool_malloc(int, unsigned long, unsigned long*)':
          tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x23c30): undefined reference to `cuMemCreate'
          /usr/bin/ld: tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x23c66): undefined reference to `cuMemAddressReserve'
          /usr/bin/ld: tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x23c8e): undefined reference to `cuMemMap'
          /usr/bin/ld: tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x23ca0): undefined reference to `cuMemRelease'
          /usr/bin/ld: tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x23cdd): undefined reference to `cuMemSetAccess'
          /usr/bin/ld: tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x23e6b): undefined reference to `cuGetErrorString'
          /usr/bin/ld: tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x23e94): undefined reference to `cuGetErrorString'
          /usr/bin/ld: tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x23ebd): undefined reference to `cuGetErrorString'
          /usr/bin/ld: tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x23ee6): undefined reference to `cuGetErrorString'
          /usr/bin/ld: tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x23f0f): undefined reference to `cuGetErrorString'
          /usr/bin/ld: /workspaces/rs-whisper-test/target/release/deps/libwhisper_rs_sys-fa390a7b2f6f0f90.rlib(ggml-cuda.cu.o): in function `ggml_init_cublas.part.0':
          tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x26509): undefined reference to `cuDeviceGet'
          /usr/bin/ld: tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x26523): undefined reference to `cuDeviceGetAttribute'
          /usr/bin/ld: tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x2656f): undefined reference to `cuMemGetAllocationGranularity'
          /usr/bin/ld: tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x26837): undefined reference to `cuGetErrorString'
          /usr/bin/ld: tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x26860): undefined reference to `cuGetErrorString'
          /usr/bin/ld: tmpxft_00005a21_00000000-6_ggml-cuda.compute_70.cudafe1.cpp:(.text+0x26910): undefined reference to `cuGetErrorString'
          collect2: error: ld returned 1 exit status
          
  = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
  = note: use the `-l` flag to specify native libraries to link
  = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)

error: could not compile `rs-whisper-test` (bin "rs-whisper-test") due to 1 previous error

I'm not sure how to fix it, any help appreciate.

Thanks in advance.