Releases: ggerganov/whisper.cpp
Releases · ggerganov/whisper.cpp
v1.7.3
Overview
- Massive performance improvements for the Metal backend, especially for beams > 1 and for quantized models
- Reduce hallucinations during silence by @jkarthic in #2629
- Implement no_speech_thold by @jkarthic in #2625
| CPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| M2 Ultra | Metal | tiny | 1 | 1 | 7.90 | 1.26 | 0.35 | 0.01 | ed733e8 |
| M2 Ultra | Metal | tiny-q5_0 | 1 | 1 | 8.44 | 1.23 | 0.36 | 0.01 | ed733e8 |
| M2 Ultra | Metal | tiny-q5_1 | 1 | 1 | 8.26 | 1.27 | 0.37 | 0.01 | ed733e8 |
| M2 Ultra | Metal | tiny-q8_0 | 1 | 1 | 8.03 | 1.21 | 0.35 | 0.01 | ed733e8 |
| M2 Ultra | Metal | base | 1 | 1 | 13.77 | 1.80 | 0.42 | 0.02 | ed733e8 |
| M2 Ultra | Metal | base-q5_0 | 1 | 1 | 15.02 | 1.72 | 0.42 | 0.02 | ed733e8 |
| M2 Ultra | Metal | base-q5_1 | 1 | 1 | 14.93 | 1.74 | 0.42 | 0.02 | ed733e8 |
| M2 Ultra | Metal | base-q8_0 | 1 | 1 | 14.26 | 1.68 | 0.41 | 0.02 | ed733e8 |
| M2 Ultra | Metal | small | 1 | 1 | 39.76 | 3.54 | 0.85 | 0.05 | ed733e8 |
| M2 Ultra | Metal | small-q5_0 | 1 | 1 | 45.07 | 3.47 | 0.87 | 0.05 | ed733e8 |
| M2 Ultra | Metal | small-q5_1 | 1 | 1 | 44.82 | 3.49 | 0.87 | 0.05 | ed733e8 |
| M2 Ultra | Metal | small-q8_0 | 1 | 1 | 41.79 | 3.30 | 0.84 | 0.05 | ed733e8 |
| M2 Ultra | Metal | medium | 1 | 1 | 106.73 | 7.28 | 1.78 | 0.11 | ed733e8 |
| M2 Ultra | Metal | medium-q5_0 | 1 | 1 | 124.43 | 6.63 | 1.83 | 0.12 | ed733e8 |
| M2 Ultra | Metal | medium-q5_1 | 1 | 1 | 124.19 | 6.70 | 1.84 | 0.12 | ed733e8 |
| M2 Ultra | Metal | medium-q8_0 | 1 | 1 | 113.88 | 6.52 | 1.75 | 0.11 | ed733e8 |
| M2 Ultra | Metal | medium-dis | 1 | 1 | 94.97 | 0.97 | 0.22 | 0.01 | ed733e8 |
| M2 Ultra | Metal | large-v2 | 1 | 1 | 193.33 | 10.53 | 2.65 | 0.20 | ed733e8 |
| M2 Ultra | Metal | large-v2-q5_0 | 1 | 1 | 229.22 | 9.52 | 2.72 | 0.23 | ed733e8 |
| M2 Ultra | Metal | large-v2-q5_1 | 1 | 1 | 229.40 | 9.62 | 2.73 | 0.23 | ed733e8 |
| M2 Ultra | Metal | large-v2-q8_0 | 1 | 1 | 207.30 | 9.36 | 2.59 | 0.21 | ed733e8 |
| M2 Ultra | Metal | large-v2-dis | 1 | 1 | 171.43 | 1.09 | 0.25 | 0.02 | ed733e8 |
| M2 Ultra | Metal | large-v3-turbo | 1 | 1 | 173.45 | 1.73 | 0.41 | 0.03 | ed733e8 |
| M2 Ultra | Metal | large-v3-turbo-q5_0 | 1 | 1 | 205.52 | 1.52 | 0.42 | 0.04 | ed733e8 |
| M2 Ultra | Metal | large-v3-turbo-q8_0 | 1 | 1 | 185.90 | 1.48 | 0.40 | 0.03 | ed733e8 |
What's Changed
- sync : ggml by @ggerganov in #2573
- ruby : Follow source tree change by @KitaitiMakoto in #2580
- Add
q8_0models todownload-ggml-model.shby @mrienstra in #2589 - ruby : Add low-level methods to transcribe by @KitaitiMakoto in #2585
- sync : ggml by @ggerganov in #2608
- ruby : Sync whisper.cpp and model download feature by @KitaitiMakoto in #2617
- Fix typo in
download-ggml-model.shby @mrienstra in #2623 - Add Missing Include Directory for ggml-cpu in whisper.android CMakeLists by @Thamster in #2624
- fix: prevent division by zero in soft_max vulkan shader by @gn64 in #2633
- cmake : fix "amd64" processor string by @ggerganov in #2638
- Fix typo in Java Binding README by @crummyh in #2637
- Fix hallucinations during silence by @jkarthic in #2629
- Implement no_speech_thold by @jkarthic in #2625
- Improve consistency in stream exameple README commands by @crummyh in #2642
- ruby : Add no_speech_thold by @KitaitiMakoto in #2641
- sync : ggml by @ggerganov in #2639
- ci : msys enable SDL2 build by @ggerganov in #2635
New Contributors
- @Thamster made their first contribution in #2624
- @gn64 made their first contribution in #2633
- @crummyh made their first contribution in #2637
- @jkarthic made their first contribution in #2629
Full Changelog: v1.7.2...v1.7.3
Contributors
Thamster, KitaitiMakoto, and 5 other contributors
Assets 2
1
Join discussion
v1.7.3-pre
Overview
Massive performance improvements for the Metal backend, especially for beams > 1. Especially for quantized models.
Setting as "pre-release" since there have been major changes to the build system (now using CMake) and I wan't to gather some feedback about how well the project builds now on various platforms. Please leave comments in the discussion to help fix any remaining issues before the official release.
| CPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| M2 Ultra | Metal | tiny | 1 | 1 | 7.90 | 1.26 | 0.35 | 0.01 | ed733e8 |
| M2 Ultra | Metal | tiny-q5_0 | 1 | 1 | 8.44 | 1.23 | 0.36 | 0.01 | ed733e8 |
| M2 Ultra | Metal | tiny-q5_1 | 1 | 1 | 8.26 | 1.27 | 0.37 | 0.01 | ed733e8 |
| M2 Ultra | Metal | tiny-q8_0 | 1 | 1 | 8.03 | 1.21 | 0.35 | 0.01 | ed733e8 |
| M2 Ultra | Metal | base | 1 | 1 | 13.77 | 1.80 | 0.42 | 0.02 | ed733e8 |
| M2 Ultra | Metal | base-q5_0 | 1 | 1 | 15.02 | 1.72 | 0.42 | 0.02 | ed733e8 |
| M2 Ultra | Metal | base-q5_1 | 1 | 1 | 14.93 | 1.74 | 0.42 | 0.02 | ed733e8 |
| M2 Ultra | Metal | base-q8_0 | 1 | 1 | 14.26 | 1.68 | 0.41 | 0.02 | ed733e8 |
| M2 Ultra | Metal | small | 1 | 1 | 39.76 | 3.54 | 0.85 | 0.05 | ed733e8 |
| M2 Ultra | Metal | small-q5_0 | 1 | 1 | 45.07 | 3.47 | 0.87 | 0.05 | ed733e8 |
| M2 Ultra | Metal | small-q5_1 | 1 | 1 | 44.82 | 3.49 | 0.87 | 0.05 | ed733e8 |
| M2 Ultra | Metal | small-q8_0 | 1 | 1 | 41.79 | 3.30 | 0.84 | 0.05 | ed733e8 |
| M2 Ultra | Metal | medium | 1 | 1 | 106.73 | 7.28 | 1.78 | 0.11 | ed733e8 |
| M2 Ultra | Metal | medium-q5_0 | 1 | 1 | 124.43 | 6.63 | 1.83 | 0.12 | ed733e8 |
| M2 Ultra | Metal | medium-q5_1 | 1 | 1 | 124.19 | 6.70 | 1.84 | 0.12 | ed733e8 |
| M2 Ultra | Metal | medium-q8_0 | 1 | 1 | 113.88 | 6.52 | 1.75 | 0.11 | ed733e8 |
| M2 Ultra | Metal | medium-dis | 1 | 1 | 94.97 | 0.97 | 0.22 | 0.01 | ed733e8 |
| M2 Ultra | Metal | large-v2 | 1 | 1 | 193.33 | 10.53 | 2.65 | 0.20 | ed733e8 |
| M2 Ultra | Metal | large-v2-q5_0 | 1 | 1 | 229.22 | 9.52 | 2.72 | 0.23 | ed733e8 |
| M2 Ultra | Metal | large-v2-q5_1 | 1 | 1 | 229.40 | 9.62 | 2.73 | 0.23 | ed733e8 |
| M2 Ultra | Metal | large-v2-q8_0 | 1 | 1 | 207.30 | 9.36 | 2.59 | 0.21 | ed733e8 |
| M2 Ultra | Metal | large-v2-dis | 1 | 1 | 171.43 | 1.09 | 0.25 | 0.02 | ed733e8 |
| M2 Ultra | Metal | large-v3-turbo | 1 | 1 | 173.45 | 1.73 | 0.41 | 0.03 | ed733e8 |
| M2 Ultra | Metal | large-v3-turbo-q5_0 | 1 | 1 | 205.52 | 1.52 | 0.42 | 0.04 | ed733e8 |
| M2 Ultra | Metal | large-v3-turbo-q8_0 | 1 | 1 | 185.90 | 1.48 | 0.40 | 0.03 | ed733e8 |
What's Changed
- sync : ggml by @ggerganov in #2573
- ruby : Follow source tree change by @KitaitiMakoto in #2580
- Add
q8_0models todownload-ggml-model.shby @mrienstra in #2589 - ruby : Add low-level methods to transcribe by @KitaitiMakoto in #2585
- sync : ggml by @ggerganov in #2608
Full Changelog: v1.7.2...v1.7.3-pre
v1.7.2
Overview
- Various improvements in the Metal backend
- Fix extra memory usage for large samples
- Remove limit for
ggml_context(i.e. more beams and processors are supported)
| CPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| M2 Ultra | METAL | tiny | 1 | 1 | 9.51 | 1.39 | 0.41 | 0.01 | 83ac284 |
| M2 Ultra | METAL | tiny-q5_0 | 1 | 1 | 9.57 | 1.41 | 0.42 | 0.01 | 83ac284 |
| M2 Ultra | METAL | tiny-q5_1 | 1 | 1 | 8.74 | 1.39 | 0.42 | 0.01 | 83ac284 |
| M2 Ultra | METAL | tiny-q8_0 | 1 | 1 | 8.36 | 1.33 | 0.41 | 0.01 | 83ac284 |
| M2 Ultra | METAL | base | 1 | 1 | 14.27 | 1.90 | 0.63 | 0.02 | 83ac284 |
| M2 Ultra | METAL | base-q5_0 | 1 | 1 | 15.50 | 1.90 | 0.65 | 0.02 | 83ac284 |
| M2 Ultra | METAL | base-q5_1 | 1 | 1 | 15.67 | 1.88 | 0.65 | 0.02 | 83ac284 |
| M2 Ultra | METAL | base-q8_0 | 1 | 1 | 14.69 | 1.81 | 0.63 | 0.02 | 83ac284 |
| M2 Ultra | METAL | small | 1 | 1 | 40.85 | 3.77 | 1.43 | 0.05 | 83ac284 |
| M2 Ultra | METAL | small-q5_0 | 1 | 1 | 45.99 | 3.90 | 1.52 | 0.05 | 83ac284 |
| M2 Ultra | METAL | small-q5_1 | 1 | 1 | 46.19 | 3.83 | 1.50 | 0.06 | 83ac284 |
| M2 Ultra | METAL | small-q8_0 | 1 | 1 | 42.90 | 3.65 | 1.46 | 0.05 | 83ac284 |
| M2 Ultra | METAL | medium | 1 | 1 | 109.01 | 7.59 | 3.24 | 0.11 | 83ac284 |
| M2 Ultra | METAL | medium-q5_0 | 1 | 1 | 126.78 | 7.55 | 3.45 | 0.13 | 83ac284 |
| M2 Ultra | METAL | medium-q5_1 | 1 | 1 | 127.71 | 7.39 | 3.43 | 0.13 | 83ac284 |
| M2 Ultra | METAL | medium-q8_0 | 1 | 1 | 115.97 | 7.21 | 3.35 | 0.12 | 83ac284 |
| M2 Ultra | METAL | medium-dis | 1 | 1 | 97.74 | 1.06 | 0.36 | 0.01 | 83ac284 |
| M2 Ultra | METAL | large-v2 | 1 | 1 | 196.99 | 11.29 | 5.06 | 0.20 | 83ac284 |
| M2 Ultra | METAL | large-v2-q5_0 | 1 | 1 | 233.88 | 10.83 | 5.56 | 0.24 | 83ac284 |
| M2 Ultra | METAL | large-v2-q5_1 | 1 | 1 | 234.03 | 10.73 | 5.46 | 0.24 | 83ac284 |
| M2 Ultra | METAL | large-v2-q8_0 | 1 | 1 | 210.83 | 10.29 | 5.23 | 0.22 | 83ac284 |
| M2 Ultra | METAL | large-v2-dis | 1 | 1 | 175.37 | 1.18 | 0.42 | 0.02 | 83ac284 |
| M2 Ultra | METAL | large-v3-turbo | 1 | 1 | 177.35 | 1.85 | 0.73 | 0.03 | 83ac284 |
| M2 Ultra | METAL | large-v3-turbo-q5_0 | 1 | 1 | 209.31 | 1.69 | 0.80 | 0.04 | 83ac284 |
| M2 Ultra | METAL | large-v3-turbo-q8_0 | 1 | 1 | 189.55 | 1.64 | 0.75 | 0.03 | 83ac284 |
What's Changed
- Added OpenVino init on state by @sandrohanea in #2464
- Updating the Quick start by @stsfaroz in #2475
- max_length from max_target_positions by @CrispStrobe in #2477
- Add dtw preset for large-v3-turbo by @rotemdan in #2481
- make : fix GGML_VULKAN=1 build by @ggerganov in #2485
- Add Vulkan notice in README.md by @toboil-features in #2488
- Fix Ruby binding building by @KitaitiMakoto in #2484
- Update of README.md by @toboil-features in #2489
- whisper: fix index overflow by @Josscii in #2505
- ruby : Add Metal support by @KitaitiMakoto in #2516
- ruby: New segment callback by @KitaitiMakoto in #2506
- ruby : add more APIs by @KitaitiMakoto in #2518
- ruby: fix installation test by @KitaitiMakoto in #2519
- When DTW timestamps are enabled, defer new_segment_callback until after DTW compute step by @jettoblack in #2515
- ci : fix openblas build by @ggerganov in #2511
- whisper : reduce ggml_context usage by @ggerganov in #2525
- sync : ggml by @ggerganov in #2528
- passing samples_padded by ref to the threads. by @vinmisra in #2534
- fix ffmpeg v5 build by @stsydow in #2543
- fix: ggml-vulkan logs by @thewh1teagle in #2547
- Fix the instructions on the Ruby binding by @wilsonsilva in #2548
- whisper.swiftui : add model download list & bench methods by @jhen0409 in #2546
- ruby : Add more API by @KitaitiMakoto in #2551
- Fix building workflow for linux/arm64 container by @rai62 in #2555
- sync : ggml by @ggerganov in #2561
- whisper.swiftui : switch Mac dest to Mac (Designed for iPad) by @jhen0409 in #2562
- ci : use local ggml by @ggerganov in #2567
- sycl: fix example build by @stsydow in #2570
New Contributors
- @stsfaroz made their first contribution in #2475
- @CrispStrobe made their first contribution in #2477
- @toboil-features made their first contribution in #2488
- @KitaitiMakoto made their first contribution in #2484
- @Josscii made their first contribution in #2505
- @jettoblack made their first contribution in #2515
- @vinmisra made their first contribution in #2534
- @stsydow made their first contribution in #2543
- @wilsonsilva made their first contribution in #2548
- @rai62 made their first contribution in #2555
Full Changelog: v1.7.1...v1.7.2
Contributors
KitaitiMakoto, wilsonsilva, and 13 other contributors
Assets 2
v1.7.2-pre
Overview
This is a pre-release since I think there have been some reports about memory leaks which I haven't had the time to investigate and confirm. If these are resolved in the next days, will add them to the official 1.7.2 release next week.
- Various improvements in the Metal backend
- Fix extra memory usage for large samples
- Remove limit for
ggml_context(i.e. more beams and processors are supported)
| CPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| M2 Ultra | METAL | tiny | 1 | 1 | 9.51 | 1.39 | 0.41 | 0.01 | 83ac284 |
| M2 Ultra | METAL | tiny-q5_0 | 1 | 1 | 9.57 | 1.41 | 0.42 | 0.01 | 83ac284 |
| M2 Ultra | METAL | tiny-q5_1 | 1 | 1 | 8.74 | 1.39 | 0.42 | 0.01 | 83ac284 |
| M2 Ultra | METAL | tiny-q8_0 | 1 | 1 | 8.36 | 1.33 | 0.41 | 0.01 | 83ac284 |
| M2 Ultra | METAL | base | 1 | 1 | 14.27 | 1.90 | 0.63 | 0.02 | 83ac284 |
| M2 Ultra | METAL | base-q5_0 | 1 | 1 | 15.50 | 1.90 | 0.65 | 0.02 | 83ac284 |
| M2 Ultra | METAL | base-q5_1 | 1 | 1 | 15.67 | 1.88 | 0.65 | 0.02 | 83ac284 |
| M2 Ultra | METAL | base-q8_0 | 1 | 1 | 14.69 | 1.81 | 0.63 | 0.02 | 83ac284 |
| M2 Ultra | METAL | small | 1 | 1 | 40.85 | 3.77 | 1.43 | 0.05 | 83ac284 |
| M2 Ultra | METAL | small-q5_0 | 1 | 1 | 45.99 | 3.90 | 1.52 | 0.05 | 83ac284 |
| M2 Ultra | METAL | small-q5_1 | 1 | 1 | 46.19 | 3.83 | 1.50 | 0.06 | 83ac284 |
| M2 Ultra | METAL | small-q8_0 | 1 | 1 | 42.90 | 3.65 | 1.46 | 0.05 | 83ac284 |
| M2 Ultra | METAL | medium | 1 | 1 | 109.01 | 7.59 | 3.24 | 0.11 | 83ac284 |
| M2 Ultra | METAL | medium-q5_0 | 1 | 1 | 126.78 | 7.55 | 3.45 | 0.13 | 83ac284 |
| M2 Ultra | METAL | medium-q5_1 | 1 | 1 | 127.71 | 7.39 | 3.43 | 0.13 | 83ac284 |
| M2 Ultra | METAL | medium-q8_0 | 1 | 1 | 115.97 | 7.21 | 3.35 | 0.12 | 83ac284 |
| M2 Ultra | METAL | medium-dis | 1 | 1 | 97.74 | 1.06 | 0.36 | 0.01 | 83ac284 |
| M2 Ultra | METAL | large-v2 | 1 | 1 | 196.99 | 11.29 | 5.06 | 0.20 | 83ac284 |
| M2 Ultra | METAL | large-v2-q5_0 | 1 | 1 | 233.88 | 10.83 | 5.56 | 0.24 | 83ac284 |
| M2 Ultra | METAL | large-v2-q5_1 | 1 | 1 | 234.03 | 10.73 | 5.46 | 0.24 | 83ac284 |
| M2 Ultra | METAL | large-v2-q8_0 | 1 | 1 | 210.83 | 10.29 | 5.23 | 0.22 | 83ac284 |
| M2 Ultra | METAL | large-v2-dis | 1 | 1 | 175.37 | 1.18 | 0.42 | 0.02 | 83ac284 |
| M2 Ultra | METAL | large-v3-turbo | 1 | 1 | 177.35 | 1.85 | 0.73 | 0.03 | 83ac284 |
| M2 Ultra | METAL | large-v3-turbo-q5_0 | 1 | 1 | 209.31 | 1.69 | 0.80 | 0.04 | 83ac284 |
| M2 Ultra | METAL | large-v3-turbo-q8_0 | 1 | 1 | 189.55 | 1.64 | 0.75 | 0.03 | 83ac284 |
What's Changed
- Added OpenVino init on state by @sandrohanea in #2464
- Updating the Quick start by @stsfaroz in #2475
- max_length from max_target_positions by @CrispStrobe in #2477
- Add dtw preset for large-v3-turbo by @rotemdan in #2481
- make : fix GGML_VULKAN=1 build by @ggerganov in #2485
- Add Vulkan notice in README.md by @toboil-features in #2488
- Fix Ruby binding building by @KitaitiMakoto in #2484
- Update of README.md by @toboil-features in #2489
- whisper: fix index overflow by @Josscii in #2505
- ruby : Add Metal support by @KitaitiMakoto in #2516
- ruby: New segment callback by @KitaitiMakoto in #2506
- ruby : add more APIs by @KitaitiMakoto in #2518
- ruby: fix installation test by @KitaitiMakoto in #2519
- When DTW timestamps are enabled, defer new_segment_callback until after DTW compute step by @jettoblack in #2515
- ci : fix openblas build by @ggerganov in #2511
- whisper : reduce ggml_context usage by @ggerganov in #2525
- sync : ggml by @ggerganov in #2528
- passing samples_padded by ref to the threads. by @vinmisra in #2534
- fix ffmpeg v5 build by @stsydow in #2543
- fix: ggml-vulkan logs by @thewh1teagle in #2547
- Fix the instructions on the Ruby binding by @wilsonsilva in #2548
- whisper.swiftui : add model download list & bench methods by @jhen0409 in #2546
- ruby : Add more API by @KitaitiMakoto in #2551
- Fix building workflow for linux/arm64 container by @rai62 in #2555
- sync : ggml by @ggerganov in #2561
- whisper.swiftui : switch Mac dest to Mac (Designed for iPad) by @jhen0409 in #2562
New Contributors
- @stsfaroz made their first contribution in #2475
- @CrispStrobe made their first contribution in #2477
- @toboil-features made their first contribution in #2488
- @KitaitiMakoto made their first contribution in #2484
- @Josscii made their first contribution in #2505
- @jettoblack made their first contribution in #2515
- @vinmisra made their first contribution in #2534
- @stsydow made their first contribution in #2543
- @wilsonsilva made their first contribution in #2548
- @rai62 made their first contribution in #2555
Full Changelog: v1.7.1...v1.7.2-pre
Contributors
KitaitiMakoto, wilsonsilva, and 13 other contributors
Assets 2
v1.7.1
Overview
- Fix Vulkan crashes
- Performance stats for Vulkan on RTX 2060
| GPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| RTX 2060 | VULKAN | tiny | 1 | 0 | 30.38 | 1.37 | 1.04 | 0.05 | 9f346d0 |
| RTX 2060 | VULKAN | tiny-q5_0 | 1 | 0 | 20.98 | 1.38 | 0.99 | 0.05 | 9f346d0 |
| RTX 2060 | VULKAN | tiny-q5_1 | 1 | 0 | 20.74 | 1.30 | 0.96 | 0.05 | 9f346d0 |
| RTX 2060 | VULKAN | base | 1 | 0 | 44.69 | 1.59 | 1.78 | 0.09 | 9f346d0 |
| RTX 2060 | VULKAN | base-q5_0 | 1 | 0 | 39.72 | 2.11 | 1.72 | 0.08 | 9f346d0 |
| RTX 2060 | VULKAN | base-q5_1 | 1 | 0 | 39.45 | 2.01 | 1.63 | 0.08 | 9f346d0 |
| RTX 2060 | VULKAN | small | 1 | 0 | 160.02 | 3.53 | 4.64 | 0.23 | 9f346d0 |
| RTX 2060 | VULKAN | small-q5_0 | 1 | 0 | 141.52 | 4.54 | 4.44 | 0.20 | 9f346d0 |
| RTX 2060 | VULKAN | small-q5_1 | 1 | 0 | 141.03 | 4.63 | 4.18 | 0.20 | 9f346d0 |
| RTX 2060 | VULKAN | medium | 1 | 0 | 472.66 | 7.55 | 11.35 | 0.56 | 9f346d0 |
| RTX 2060 | VULKAN | medium-q5_0 | 1 | 0 | 395.55 | 9.81 | 10.64 | 0.49 | 9f346d0 |
| RTX 2060 | VULKAN | medium-q5_1 | 1 | 0 | 398.85 | 10.16 | 10.15 | 0.50 | 9f346d0 |
| RTX 2060 | VULKAN | medium-dis | 1 | 0 | 427.26 | 1.26 | 1.20 | 0.08 | 9f346d0 |
| RTX 2060 | VULKAN | large-v2 | 1 | 0 | 924.60 | 12.36 | 18.56 | 1.01 | 9f346d0 |
| RTX 2060 | VULKAN | large-v2-q5_0 | 1 | 0 | 774.21 | 17.25 | 17.17 | 0.85 | 9f346d0 |
| RTX 2060 | VULKAN | large-v2-q5_1 | 1 | 0 | 779.75 | 17.44 | 16.27 | 0.85 | 9f346d0 |
| RTX 2060 | VULKAN | large-v2-dis | 1 | 0 | 833.35 | 1.38 | 1.56 | 0.10 | 9f346d0 |
| RTX 2060 | VULKAN | large-v3-turbo | 1 | 0 | 839.90 | 2.11 | 2.70 | 0.16 | 9f346d0 |
| RTX 2060 | VULKAN | large-v3-turbo-q5_0 | 1 | 0 | 705.49 | 3.22 | 2.53 | 0.14 | 9f346d0 |
What's Changed
- Retry allocation with fallback flags by @SRHMorris in #2451
New Contributors
- @SRHMorris made their first contribution in #2451
Full Changelog: v1.7.0...v1.7.1
Binaries
https://github.com/ggerganov/whisper.cpp/actions/runs/11213279590
Contributors
SRHMorris
Assets 2
v1.7.0
Overview
- Fix crashes with high number of beams
- Reduce overal VRAM usage
- Optimize Encoder performance
Some performance numbers for this release:
M2 Ultra
Flash Attention ON:
| GPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| M2 Ultra | METAL | tiny | 1 | 1 | 8.37 | 1.44 | 0.48 | 0.01 | 6a94163 |
| M2 Ultra | METAL | tiny-q5_0 | 1 | 1 | 9.81 | 1.46 | 0.50 | 0.01 | 6a94163 |
| M2 Ultra | METAL | tiny-q5_1 | 1 | 1 | 8.80 | 1.47 | 0.50 | 0.01 | 6a94163 |
| M2 Ultra | METAL | base | 1 | 1 | 16.11 | 1.96 | 0.74 | 0.02 | 6a94163 |
| M2 Ultra | METAL | base-q5_0 | 1 | 1 | 16.38 | 1.99 | 0.78 | 0.02 | 6a94163 |
| M2 Ultra | METAL | base-q5_1 | 1 | 1 | 16.72 | 2.00 | 0.77 | 0.02 | 6a94163 |
| M2 Ultra | METAL | small | 1 | 1 | 41.26 | 3.88 | 1.66 | 0.05 | 6a94163 |
| M2 Ultra | METAL | small-q5_0 | 1 | 1 | 46.91 | 4.02 | 1.76 | 0.06 | 6a94163 |
| M2 Ultra | METAL | small-q5_1 | 1 | 1 | 47.05 | 4.00 | 1.73 | 0.06 | 6a94163 |
| M2 Ultra | METAL | medium | 1 | 1 | 111.29 | 7.79 | 3.63 | 0.11 | 6a94163 |
| M2 Ultra | METAL | medium-q5_0 | 1 | 1 | 129.78 | 7.71 | 3.85 | 0.13 | 6a94163 |
| M2 Ultra | METAL | medium-q5_1 | 1 | 1 | 129.29 | 7.71 | 3.87 | 0.13 | 6a94163 |
| M2 Ultra | METAL | medium-dis | 1 | 1 | 99.27 | 1.09 | 0.43 | 0.02 | 6a94163 |
| M2 Ultra | METAL | large-v2 | 1 | 1 | 198.81 | 11.54 | 5.59 | 0.20 | 6a94163 |
| M2 Ultra | METAL | large-v2-q5_0 | 1 | 1 | 236.18 | 11.12 | 6.11 | 0.24 | 6a94163 |
| M2 Ultra | METAL | large-v2-q5_1 | 1 | 1 | 235.88 | 11.14 | 6.01 | 0.24 | 6a94163 |
| M2 Ultra | METAL | large-v2-dis | 1 | 1 | 177.41 | 1.21 | 0.48 | 0.02 | 6a94163 |
| M2 Ultra | METAL | large-v3-turbo | 1 | 1 | 178.92 | 1.89 | 0.83 | 0.03 | 6a94163 |
| M2 Ultra | METAL | large-v3-turbo-q5_0 | 1 | 1 | 211.44 | 1.73 | 0.90 | 0.04 | 6a94163 |
Flash Attention OFF:
| GPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| M2 Ultra | METAL | tiny | 1 | 0 | 10.04 | 1.37 | 0.50 | 0.01 | 6a94163 |
| M2 Ultra | METAL | tiny-q5_0 | 1 | 0 | 10.02 | 1.36 | 0.53 | 0.01 | 6a94163 |
| M2 Ultra | METAL | tiny-q5_1 | 1 | 0 | 11.08 | 1.37 | 0.53 | 0.01 | 6a94163 |
| M2 Ultra | METAL | base | 1 | 0 | 17.84 | 1.93 | 0.77 | 0.02 | 6a94163 |
| M2 Ultra | METAL | base-q5_0 | 1 | 0 | 18.57 | 1.92 | 0.81 | 0.02 | 6a94163 |
| M2 Ultra | METAL | base-q5_1 | 1 | 0 | 18.66 | 1.93 | 0.82 | 0.02 | 6a94163 |
| M2 Ultra | METAL | small | 1 | 0 | 48.26 | 3.95 | 1.73 | 0.05 | 6a94163 |
| M2 Ultra | METAL | small-q5_0 | 1 | 0 | 53.68 | 3.99 | 1.85 | 0.06 | 6a94163 |
| M2 Ultra | METAL | small-q5_1 | 1 | 0 | 53.86 | 4.00 | 1.82 | 0.06 | 6a94163 |
| M2 Ultra | METAL | medium | 1 | 0 | 130.09 | 8.01 | 3.82 | 0.13 | 6a94163 |
| M2 Ultra | METAL | medium-q5_0 | 1 | 0 | 148.18 | 7.92 | 4.11 | 0.14 | 6a94163 |
| M2 Ultra | METAL | medium-q5_1 | 1 | 0 | 147.95 | 7.94 | 4.11 | 0.14 | 6a94163 |
| M2 Ultra | METAL | medium-dis | 1 | 0 | 116.97 | 1.11 | 0.42 | 0.02 | 6a94163 |
| M2 Ultra | METAL | large-v2 | 1 | 0 | 232.43 | 12.34 | 5.87 | 0.22 | 6a94163 |
| M2 Ultra | METAL | large-v2-q5_0 | 1 | 0 | 269.72 | 11.68 | 6.44 | 0.26 | 6a94163 |
| M2 Ultra | METAL | large-v2-q5_1 | 1 | 0 | 269.71 | 11.82 | 6.36 | 0.26 | 6a94163 |
| M2 Ultra | METAL | large-v2-dis | 1 | 0 | 209.25 | 1.25 | 0.48 | 0.02 | 6a94163 |
| M2 Ultra | METAL | large-v3-turbo | 1 | 0 | 211.09 | 1.98 | 0.84 | 0.03 | 6a94163 |
| M2 Ultra | METAL | large-v3-turbo-q5_0 | 1 | 0 | 244.23 | 1.81 | 0.92 | 0.04 | 6a94163 |
Ryzen 9 5950X + RTX 2060
Flash Attention ON:
| GPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| RTX 2060 | AVX2 CUDA | tiny | 1 | 1 | 7.35 | 0.78 | 0.24 | 0.01 | 6a94163 |
| RTX 2060 | AVX2 CUDA | tiny-q5_0 | 1 | 1 | 6.45 | 0.67 | 0.14 | 0.01 | 6a94163 |
| RTX 2060 | AVX2 CUDA | tiny-q5_1 | 1 | 1 | 6.39 | 0.66 | 0.14 | 0.01 | 6a94163 |
| RTX 2060 | AVX2 CUDA | base | 1 | 1 | 10.20 | 0.88 | 0.30 | 0.01 | 6a94163 |
| RTX 2060 | AVX2 CUDA | base-q5_0 | 1 | 1 | 11.38 | 0.92 | 0.21 | 0.02 | 6a94163 |
| RTX 2060 | AVX2 CUDA | base-q5_1 | 1 | 1 | 11.76 | 0.91 | 0.20 | 0.02 | 6a94163 |
| RTX 2060 | AVX2 CUDA | small | 1 | 1 | 33.06 | 2.00 | 0.56 | 0.03 | 6a94163 |
| RTX 2060 | AVX2 CUDA | small-q5_0 | 1 | 1 | 35.84 | 1.84 | 0.43 | 0.04 | 6a94163 |
| RTX 2060 | AVX2 CUDA | small-q5_1 | 1 | 1 | 36.89 | 1.82 | 0.42 | 0.04 | 6a94163 |
| RTX 2060 | AVX2 CUDA | medium | 1 | 1 | 90.65 | 4.54 | 1.13 | 0.08 | 6a94163 |
| RTX 2060 | AVX2 CUDA | medium-q5_0 | 1 | 1 | 104.01 | 3.80 | 0.91 | 0.10 | 6a94163 |
| RTX 2060 | AVX2 CUDA | medium-q5_1 | 1 | 1 | 107.98 | 3.72 | 0.87 | 0.10 | 6a94163 |
| RTX 2060 | AVX2 CUDA | medium-dis | 1 | 1 | 79.08 | 0.68 | 0.17 | 0.01 | 6a94163 |
| RTX 2060 | AVX2 CUDA | large-v2 | 1 | 1 | 162.00 | 7.52 | 1.92 | 0.14 | 6a94163 |
| RTX 2060 | AVX2 CUDA | large-v2-q5_0 | 1 | 1 | 184.59 | 5.64 | 1.50 | 0.16 | 6a94163 |
| RTX 2060 | AVX2 CUDA | large-v2-q5_1 | 1 | 1 | 193.85 | 5.55 | 1.44 | 0.17 | 6a94163 |
| RTX 2060 | AVX2 CUDA | large-v2-dis | 1 | 1 | 140.75 | 0.84 | 0.37 | 0.02 | 6a94163 |
| RTX 2060 | AVX2 CUDA | large-v3-turbo | 1 | 1 | 143.38 | 1.29 | 0.36 | 0.02 | 6a94163 |
| RTX 2060 | AVX2 CUDA | large-v3-turbo-q5_0 | 1 | 1 | 163.30 | 0.93 | 0.28 | 0.03 | 6a94163 |
Flash Attention OFF:
| GPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| RTX 2060 | AVX2 CUDA | tiny | 1 | 0 | 12.49 | 0.87 | 0.23 | 0.01 | 6a94163 |
| RTX 2060 | AVX2 CUDA | tiny-q5_0 | 1 | 0 | 10.65 | 0.78 | 0.19 | 0.02 | 6a94163 |
| RTX 2060 | AVX2 CUDA | tiny-q5_1 | 1 | 0 | 10.82 | 0.77 | 0.19 | 0.02 | 6a94163 |
| RTX 2060 | AVX2 CUDA | base | 1 | 0 | 18.97 | 1.04 | 0.34 | 0.02 | 6a94163 |
| RTX 2060 | AVX2 CUDA | base-q5_0 | 1 | 0 | 20.22 | 1.09 | 0.27 | 0.02 | 6a94163 |
| RTX 2060 | AVX2 CUDA | base-q5_1 | 1 | 0 | 20.48 | 1.07 | 0.27 | 0.02 | 6a94163 |
| RTX 2060 | AVX2 CUDA | small | 1 | 0 | 59.52 | 2.37 | 0.70 | 0.05 | 6a94163 |
| RTX 2060 | AVX2 CUDA | small-q5_0 | 1 | 0 | 62.98 | 2.23 | 0.60 | 0.06 | 6a94163 |
| RTX 2060 | AVX2 CUDA | small-q5_1 | 1 | 0 | 63.64 | 2.21 | 0.59 | 0.06 | 6a94163 |
| RTX 2060 | AVX2 CUDA | medium | 1 | 0 | 161.53 | 5.36 | 1.53 | 0.13 | 6a94163 |
| RTX 2060 | AVX2 CUDA | medium-q5_0 | 1 | 0 | 174.96 | 4.64 | 1.32 | 0.15 | 6a94163 |
| RTX 2060 | AVX2 CUDA | medium-q5_1 | 1 | 0 | 178.42 | 4.57 | 1.29 | 0.15 | 6a94163 |
| RTX 2060 | AVX2 CUDA | medium-dis | 1 | 0 | 149.65 | 0.75 | 0.20 | 0.02 | 6a94163 |
| RTX 2060 | AVX2 CUDA | large-v2 | 1 | 0 | 280.55 | 8.74 | 2.51 | 0.23 | 6a94163 |
| RTX 2060 | AVX2 CUDA | large-v2-q5_0 | 1 | 0 | 306.87 | 6.92 | 2.08 | 0.25 | 6a94163 |
| RTX 2060 | AVX2 CUDA | large-v2-q5_1 | 1 | 0 | 314.25 | 6.82 | 2.02 | 0.26 | 6a94163 |
| RTX 2060 | AVX2 CUDA | large-v2-dis | 1 | 0 | 259.39 | 0.91 | 0.37 | 0.02 | 6a94163 |
| RTX 2060 | AVX2 CUDA | large-v3-turbo | 1 | 0 | 261.83 | 1.44 | 0.41 | 0.04 | 6a94163 |
| RTX 2060 | AVX2 CUDA | large-v3-turbo-q5_0 | 1 | 0 | 282.99 | 1.09 | 0.33 | 0.04 | 6a94163 |
Vulkan:
| GPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| RTX 2060 | VULKAN | tiny | 1 | 0 | 30.38 | 1.37 | 1.04 | 0.05 | 9f346d0 |
| RTX 2060 | VULKAN | tiny-q5_0 | 1 | 0 | 20.98 | 1.38 | 0.99 | 0.05 | 9f346d0 |
| RTX 2060 | VULKAN | tiny-q5_1 | 1 | 0 | 20.74 | 1.30 | 0.96 | 0.05 | 9f346d0 |
| RTX 2060 | VULKAN | base | 1 | 0 | 44.69 | 1.59 | 1.78 | 0.09 | 9f346d0 |
| RTX 2060 | VULKAN | base-q5_0 | 1 | 0 | 39.72 | 2.11 | 1.72 | 0.08 | 9f346d0 |
| RTX 2060 | VULKAN | base-q5_1 | 1 | 0 | 39.45 | 2.01 | 1.63 | 0.08 | 9f346d0 |
| RTX 2060 | VULKAN | small | 1 | 0 | 160.02 | 3.53 | 4.64 | 0.23 | 9f346d0 |
| RTX 2060 | VULKAN | small-q5_0 | 1 | 0 | 141.52 | 4.54 | 4.44 | 0.20 | 9f346d0 |
| RTX 2060 | VULKA... |
v1.6.2
Overview
Bugfix when using multiple whisper_state in parallel: #2182
What's Changed
- Update ruby bindings by @taf2 in #2154
- Update server.cpp by @dvaldivia in #2181
- Revert "whisper : remove extra backend instance (huh?)" by @ggerganov in #2182
New Contributors
- @dvaldivia made their first contribution in #2181
Full Changelog: v1.6.1...v1.6.2
Contributors
taf2, ggerganov, and dvaldivia
Assets 2
v1.6.1
Minor release adding initial ffmpeg support in the examples #2133 (thx @WilliamTambellini)
What's Changed
- ci: Update build.yml to suppress warnings about node.js versions by @tamo in #2166
- node : add flash_attn param by @pprobst in #2170
- Add support for decoding input with ffmpeg (Linux) by @WilliamTambellini in #2133
New Contributors
- @WilliamTambellini made their first contribution in #2133
Full Changelog: v1.6.0...v1.6.1
Contributors
WilliamTambellini, tamo, and pprobst
Assets 2
v1.6.0
Overview
- Can optionally enable Flash Attention for faster processing on CUDA and Metal devices (#2152)
- Faster ppc64 performance (40aeeee) (not tested)
- Fix
mainslowdown bug (#2070)
Shoutout to @JohannesGaessler for contributing efficient FA CUDA kernels
Some performance numbers for this release:
M1 Pro
| CPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| M1 Pro | METAL | tiny | 1 | 0 | 39.21 | 1.74 | 0.61 | 0.04 | 22c96b4 |
| M1 Pro | METAL | base | 1 | 0 | 70.76 | 2.60 | 0.93 | 0.06 | 22c96b4 |
| M1 Pro | METAL | small | 1 | 0 | 217.28 | 6.42 | 2.14 | 0.17 | 22c96b4 |
| M1 Pro | METAL | medium | 1 | 0 | 596.74 | 14.43 | 4.75 | 0.45 | 22c96b4 |
| CPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| M1 Pro | METAL | tiny | 1 | 1 | 30.77 | 1.59 | 0.54 | 0.03 | 22c96b4 |
| M1 Pro | METAL | base | 1 | 1 | 60.42 | 2.29 | 0.81 | 0.05 | 22c96b4 |
| M1 Pro | METAL | small | 1 | 1 | 183.82 | 5.12 | 1.81 | 0.14 | 22c96b4 |
| M1 Pro | METAL | medium | 1 | 1 | 517.92 | 11.60 | 4.01 | 0.38 | 22c96b4 |
M2 Ultra
| CPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| M2 ULTRA | METAL | tiny | 1 | 0 | 12.32 | 1.35 | 0.49 | 0.01 | 22c96b4 |
| M2 ULTRA | METAL | tiny-q5_0 | 1 | 0 | 11.65 | 1.30 | 0.51 | 0.01 | 22c96b4 |
| M2 ULTRA | METAL | tiny-q5_1 | 1 | 0 | 12.08 | 1.30 | 0.51 | 0.01 | 22c96b4 |
| M2 ULTRA | METAL | base | 1 | 0 | 17.58 | 1.90 | 0.76 | 0.02 | 22c96b4 |
| M2 ULTRA | METAL | base-q5_0 | 1 | 0 | 18.89 | 1.86 | 0.79 | 0.02 | 22c96b4 |
| M2 ULTRA | METAL | base-q5_1 | 1 | 0 | 20.69 | 1.88 | 0.79 | 0.02 | 22c96b4 |
| M2 ULTRA | METAL | small | 1 | 0 | 49.32 | 3.85 | 1.71 | 0.05 | 22c96b4 |
| M2 ULTRA | METAL | small-q5_0 | 1 | 0 | 54.91 | 3.81 | 1.82 | 0.06 | 22c96b4 |
| M2 ULTRA | METAL | small-q5_1 | 1 | 0 | 54.92 | 3.81 | 1.79 | 0.06 | 22c96b4 |
| M2 ULTRA | METAL | medium | 1 | 0 | 134.34 | 8.04 | 3.82 | 0.13 | 22c96b4 |
| M2 ULTRA | METAL | medium-q5_0 | 1 | 0 | 151.68 | 7.59 | 4.07 | 0.14 | 22c96b4 |
| M2 ULTRA | METAL | medium-q5_1 | 1 | 0 | 151.58 | 7.67 | 4.07 | 0.14 | 22c96b4 |
| M2 ULTRA | METAL | medium-dis | 1 | 0 | 120.82 | 1.07 | 0.41 | 0.02 | 22c96b4 |
| M2 ULTRA | METAL | large-v2 | 1 | 0 | 235.63 | 12.27 | 5.85 | 0.22 | 22c96b4 |
| M2 ULTRA | METAL | large-v2-q5_0 | 1 | 0 | 273.38 | 11.17 | 6.40 | 0.26 | 22c96b4 |
| M2 ULTRA | METAL | large-v2-q5_1 | 1 | 0 | 272.44 | 11.32 | 6.29 | 0.26 | 22c96b4 |
| M2 ULTRA | METAL | large-v2-dis | 1 | 0 | 212.51 | 1.20 | 0.47 | 0.02 | 22c96b4 |
| CPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| M2 ULTRA | METAL | tiny | 1 | 1 | 9.07 | 1.33 | 0.45 | 0.01 | 22c96b4 |
| M2 ULTRA | METAL | tiny-q5_0 | 1 | 1 | 9.74 | 1.33 | 0.47 | 0.01 | 22c96b4 |
| M2 ULTRA | METAL | tiny-q5_1 | 1 | 1 | 8.93 | 1.31 | 0.46 | 0.01 | 22c96b4 |
| M2 ULTRA | METAL | base | 1 | 1 | 15.75 | 1.87 | 0.71 | 0.02 | 22c96b4 |
| M2 ULTRA | METAL | base-q5_0 | 1 | 1 | 17.04 | 1.83 | 0.74 | 0.02 | 22c96b4 |
| M2 ULTRA | METAL | base-q5_1 | 1 | 1 | 17.17 | 1.83 | 0.74 | 0.02 | 22c96b4 |
| M2 ULTRA | METAL | small | 1 | 1 | 42.33 | 3.64 | 1.60 | 0.05 | 22c96b4 |
| M2 ULTRA | METAL | small-q5_0 | 1 | 1 | 47.61 | 3.63 | 1.70 | 0.05 | 22c96b4 |
| M2 ULTRA | METAL | small-q5_1 | 1 | 1 | 47.70 | 3.66 | 1.68 | 0.05 | 22c96b4 |
| M2 ULTRA | METAL | medium | 1 | 1 | 114.42 | 7.53 | 3.55 | 0.11 | 22c96b4 |
| M2 ULTRA | METAL | medium-q5_0 | 1 | 1 | 132.63 | 7.02 | 3.77 | 0.13 | 22c96b4 |
| M2 ULTRA | METAL | medium-q5_1 | 1 | 1 | 132.28 | 7.10 | 3.76 | 0.13 | 22c96b4 |
| M2 ULTRA | METAL | medium-dis | 1 | 1 | 102.34 | 1.01 | 0.42 | 0.01 | 22c96b4 |
| M2 ULTRA | METAL | large-v2 | 1 | 1 | 203.01 | 11.03 | 5.45 | 0.20 | 22c96b4 |
| M2 ULTRA | METAL | large-v2-q5_0 | 1 | 1 | 240.05 | 10.18 | 5.98 | 0.23 | 22c96b4 |
| M2 ULTRA | METAL | large-v2-q5_1 | 1 | 1 | 239.22 | 10.23 | 5.87 | 0.23 | 22c96b4 |
| M2 ULTRA | METAL | large-v2-dis | 1 | 1 | 181.14 | 1.14 | 0.48 | 0.02 | 22c96b4 |
Ryzen 9 5950X + RTX 2060
| CPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| Ryzen 9 5950X | AVX2 | tiny | 8 | 0 | 195.29 | 1.57 | 0.51 | 0.26 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | tiny-q5_0 | 8 | 0 | 213.33 | 1.10 | 0.50 | 0.30 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | tiny-q5_1 | 8 | 0 | 219.38 | 1.18 | 0.53 | 0.32 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | base | 8 | 0 | 424.85 | 3.71 | 1.03 | 0.46 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | base-q5_0 | 8 | 0 | 473.61 | 1.81 | 0.82 | 0.52 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | base-q5_1 | 8 | 0 | 484.14 | 1.92 | 0.85 | 0.56 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | small | 8 | 0 | 1458.32 | 12.66 | 3.09 | 1.26 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | small-q5_0 | 8 | 0 | 1673.22 | 6.42 | 2.18 | 1.45 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | small-q5_1 | 8 | 0 | 1724.78 | 6.72 | 2.32 | 1.52 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | medium | 8 | 0 | 4333.87 | 36.80 | 8.56 | 3.37 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | medium-q5_0 | 8 | 0 | 5194.09 | 19.21 | 5.71 | 3.97 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | medium-q5_1 | 8 | 0 | 5450.39 | 20.01 | 5.99 | 4.17 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | medium-dis | 8 | 0 | 3995.19 | 5.08 | 1.21 | 0.55 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | large-v2 | 8 | 0 | 8056.16 | 69.74 | 16.11 | 6.13 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | large-v2-q5_0 | 8 | 0 | 9799.58 | 35.16 | 10.49 | 7.28 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | large-v2-q5_1 | 8 | 0 | ms | 36.74 | 11.02 | 7.65 | 22c96b4 |
| Ryzen 9 5950X | AVX2 | large-v2-dis | 8 | 0 | 7490.03 | 7.40 | 1.70 | 0.72 | 22c96b4 |
| GPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| RTX 2060 | AVX2 CUDA | tiny | 8 | 0 | 12.54 | 0.93 | 0.29 | 0.02 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | tiny-q5_0 | 8 | 0 | 12.73 | 0.98 | 0.24 | 0.02 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | tiny-q5_1 | 8 | 0 | 12.72 | 0.99 | 0.24 | 0.02 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | base | 8 | 0 | 24.14 | 1.28 | 0.41 | 0.03 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | base-q5_0 | 8 | 0 | 24.58 | 1.38 | 0.35 | 0.03 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | base-q5_1 | 8 | 0 | 24.58 | 1.37 | 0.35 | 0.03 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | small | 8 | 0 | 74.70 | 2.91 | 0.84 | 0.07 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | small-q5_0 | 8 | 0 | 76.12 | 2.84 | 0.77 | 0.08 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | small-q5_1 | 8 | 0 | 76.14 | 2.84 | 0.76 | 0.08 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | medium | 8 | 0 | 200.69 | 6.46 | 1.83 | 0.17 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | medium-q5_0 | 8 | 0 | 204.80 | 5.90 | 1.65 | 0.19 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | medium-q5_1 | 8 | 0 | 205.61 | 5.85 | 1.61 | 0.19 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | medium-dis | 8 | 0 | 186.17 | 0.86 | 0.24 | 0.02 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | large-v2 | 8 | 0 | 347.22 | 10.36 | 2.82 | 0.29 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | large-v2-q5_0 | 8 | 0 | 357.06 | 8.81 | 2.58 | 0.34 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | large-v2-q5_1 | 8 | 0 | 356.97 | 8.62 | 2.49 | 0.33 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | large-v2-dis | 8 | 0 | 318.05 | 1.03 | 0.34 | 0.04 | 22c96b4 |
| GPU | Config | Model | Th | FA | Enc. | Dec. | Bch5 | PP | Commit |
|---|---|---|---|---|---|---|---|---|---|
| RTX 2060 | AVX2 CUDA | tiny | 8 | 1 | 7.21 | 0.76 | 0.29 | 0.02 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | tiny-q5_0 | 8 | 1 | 7.42 | 0.82 | 0.18 | 0.02 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | tiny-q5_1 | 8 | 1 | 7.38 | 0.82 | 0.18 | 0.02 | 22c96b4 |
| RTX 2060 | AVX2 CUDA | ... |
v1.5.5
Overview
Many small incremental updates + Token level timestamps with DTW by @denersc in #1485
Feedback is welcome!
Full Changelog: v1.5.4...v1.5.5
What's Changed
- server : fix server temperature + add temperature_inc by @ggerganov in #1729
- main : add cli option to disable system prints by @ggerganov in #1740
- server: add request path by @eschmidbauer in #1741
- Optional Piper TTS support for talk-llama example. by @RhinoDevel in #1749
- fix/1748 by @nank1ro in #1750
- Don't compute timestamps when not printing them. by @ghindle in #1755
- Add more parameters to server api by @ghindle in #1754
- Add SetInitialPrompt method to go bindings by @blib in #1753
- ggml : fix 32-bit ARM compat for IQ2_XS by @ggerganov in #1758
- refactor: get all scripts to be POSIX Compliant by @sonphantrung in #1725
- whisper : load the model into multiple buffers of max size 1GB by @ggerganov in #1763
- rebase against your -np changes (thx) and add better python file to be used on the command line or as library by @contractorwolf in #1744
- examples/talk-llama: Add optional commandline parameter to set the bot name. by @RhinoDevel in #1764
- server : fix building and simplify lib deps on Windows by @przemoc in #1772
- talk-llama: optional wake-up command and audio confirmation by @Rakksor in #1765
- examples/server: implement "verbose_json" format with token details by @rmmh in #1781
- whisper.android: Return output from benchmarks by @luciferous in #1785
- libwhisper.so should be position independent by @trixirt in #1792
- Docs: try to make model options / model install methods clearer by @mrienstra in #1806
- common : fix input buffer check by @ggerganov in #1812
- Update Makefile by @jwijffels in #1813
- Add fields to
verbose_jsonresponse and show examples on the home page by @JacobLinCool in #1802 - common: fix wav buffer detection by @JacobLinCool in #1819
- Add macOS deployment target option to Makefile by @didzis in #1839
- Expose CUDA device setting in public API by @didzis in #1840
- whisper.android: How to build with CLBlast by @luciferous in #1809
- server: Allow CORS request with authorization headers by @valenting in #1850
- Embed Metal library source into compiled binary by @didzis in #1842
- added audio_ctx argument to main and server examples by @dscripka in #1857
- whisper : fix external encoder by @ggerganov in #1860
- swift : package no longer use ggml dependency by @ggerganov in #1861
- fix openvino setup docs by @jumpers775 in #1874
- clean up common code in examples by @felrock in #1871
- main : check if input files exist before proceeding by @Theldus in #1872
- Linking issue fix via Makefile when CUBLAS enabled in the WSL #1876 by @lbluep in #1878
- main : fix file existence check in main.cpp by @Theldus in #1889
- openvino : fix convert-whisper-to-openvino.py for v2023.0.0 (#1870) by @st-gr in #1890
- ggml : 32-bit arm compat by @ggerganov in #1891
- Add SYCL logic in whisper by @abhilash1910 in #1863
- talk and talk-llama: Pass text_to_speak as a file by @tamo in #1865
- Stream.wasm: Fix invalid memory access when no segments are returned by @Andrews54757 in #1902
- Update README to Recommend MacOS Sonoma for Core ML to avoid hallucination by @gavin1818 in #1917
- Add library versioning by @kenneth-ge in #1352
- Fix SF(segment fault) issue in Android JNI by @zhouwg in #1929
- Fix typo in source file whisper.cpp by @zhouwg in #1925
- bench:fix typo by @zhouwg in #1933
- Auto lowercase language parameter by @F1L1Pv2 in #1928
- ggml : try fix 32-bit arm compat by @ggerganov in #1938
- whisper : make beam candidate sort more stable by @josharian in #1943
- bindings/go : add linker flags to make metal work by @josharian in #1944
- whisper : improve beam search candidate diversity by @josharian in #1947
- whisper : document whisper_batch.n_seq_id by @josharian in #1942
- Rename --audio-context to --audio-ctx, as per help text by @joliss in #1953
- [DRAFT] Token level timestamps with DTW (#375) by @denersc in #1485
- Fedora dependencies needed (SDL2) by @Man2Dev in #1970
- libcuda.so.1 in PATH in Docker Container by @tiagofassoni in #1966
- ruby : fix build by @ggerganov in #1980
- Improve support for distil-large-v3 by @sanchit-gandhi in #1982
- whisper : improve handling of prompts by @ggerganov in #1981
- sync : ggml by @ggerganov in #2001
- Implemented command-style grammar in the main example. by @ulatekh in #1998
- Use pkg-config for OpenBLAS by @przemoc in #1778
- ci : add building in MSYS2 environments (Windows) by @przemoc in #1994
- Support CUDA versions < 11.1 by @primenko-v in #2020
- Create solution folders in the CMake build by @ulatekh in #2004
- Allow a regular expression to describe tokens to suppress by @ulatekh in #1997
- "main" example now allows a response-file as the sole parameter by @ulatekh in #2019
- Support for CPU BLAS build via Intel MKL by @slashlib in #2024
- Set stdin to binary mode on Windows. Fixes #2023 by @rotemdan in #2025
- Fix file-handle leak in read_wav() by @ulatekh in #2026
- Fix DTW memory access by @bradmurray-dt in #2012
- whisper: update grammar-parser.cpp by @eltociear in #2058
- fix missing reference to "model" variable in actual shell command run in whisper.nvim by @sixcircuit in #2049
- build : detect AVX512 in Makefile, add AVX512 option in CMake by @didzis in #2043
- feature/no timestamps node by @pprobst in #2048
- Update embedded Metal library generation process to include dependency by @didzis in #2045
- server.cpp: add dtw by @eschmidbauer in #2044
New Contributors
- @eschmidbauer made their first contribution in #1741
- @RhinoDevel made their first contribution in #1749
- @nank1ro made their first contribution in #1750
- @ghindle made their first contribution in #1755
- @blib made their first contribution in #1753
- @sonphantrung made their first contribution in #1725
- @contractorwolf made their first contribution in #1744
- @Rakksor made their first contribution in #1765
- @rmmh made their f...
