Performance on Intel + macOS became 2-4 times slower since commit b050283

[iandundas]

Hi, tracked down this commit because (with our app) we've received a lot of reports of 2-4x slower performance from users on Intel since we updated them to use the v1.5.0 release (of whisper.cpp)

The timings below are from my 4,2 GHz i7 iMac 32GB

Steps to reproduce:

• Checkout ec7a6f0 (the last good commit for this symptom with Intel)
• Open the SwiftUI demo project
• Follow the README instructions to copy in jfk.wav and a model: I used ggml-base.en.bin because it's slightly more pronounced vs tiny. (so this line needs updating to use that model name)
• Run the project, click "Transcribe" and check the console:

whisper_print_timings:     load time =   144.61 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =    15.64 ms
whisper_print_timings:   sample time =    16.96 ms /    29 runs (    0.58 ms per run)
whisper_print_timings:   encode time =  2235.45 ms /     1 runs ( 2235.45 ms per run)
whisper_print_timings:   decode time =   423.00 ms /    29 runs (   14.59 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time =  2693.90 ms

• Now checkout the next commit made after that: b050283 whisper : add full CUDA and Metal offloading #1472, (or indeed just checkout master):

whisper_print_timings:     load time =   146.26 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =    20.99 ms
whisper_print_timings:   sample time =    16.83 ms /    29 runs (    0.58 ms per run)
whisper_print_timings:   encode time =  9821.01 ms /     1 runs ( 9821.01 ms per run)
whisper_print_timings:   decode time =   422.12 ms /    29 runs (   14.56 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time = 10281.38 ms

So the time taken increases from 2.6 seconds to 10.2 seconds for the same file + model. This is the same for longer audio files too - 15 minutes can become 45 mins, etc.

[jordibruin]

We've been able to reproduce this on a few Intel Macs only, but since that commit adds a lot it's hard for us to determine where to start. Would love any pointers so we can get to the root of it!

[ggerganov]

Can you confirm that #1583 fixes the issue?

[iandundas]

Great! For commit f870da3 I get this:

whisper_print_timings:     load time =   241.37 ms
whisper_print_timings:     fallbacks =   0 p /   0 h
whisper_print_timings:      mel time =    17.53 ms
whisper_print_timings:   sample time =    16.74 ms /     1 runs (   16.74 ms per run)
whisper_print_timings:   encode time =  2392.90 ms /     1 runs ( 2392.90 ms per run)
whisper_print_timings:   decode time =   456.58 ms /    29 runs (   15.74 ms per run)
whisper_print_timings:   batchd time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:   prompt time =     0.00 ms /     1 runs (    0.00 ms per run)
whisper_print_timings:    total time =  2884.67 ms

Awesome, thanks for such a quick fix