Not working on MacOS (ARM)

[undefdev]

Hi, I've been trying to get this to work a few times, but it always fails with an illegal hardware instruction error.

E.g. for ./main -m models/ggml-small.bin -f samples/jfk.wav I get the following output:

whisper_model_load: loading model from 'models/ggml-small.bin'
whisper_model_load: n_vocab       = 51865
whisper_model_load: n_audio_ctx   = 1500
whisper_model_load: n_audio_state = 768
whisper_model_load: n_audio_head  = 12
whisper_model_load: n_audio_layer = 12
whisper_model_load: n_text_ctx    = 448
whisper_model_load: n_text_state  = 768
whisper_model_load: n_text_head   = 12
whisper_model_load: n_text_layer  = 12
whisper_model_load: n_mels        = 80
whisper_model_load: f16           = 1
whisper_model_load: type          = 3
whisper_model_load: mem_required  = 1048.00 MB
whisper_model_load: adding 1608 extra tokens
whisper_model_load: ggml ctx size = 533.05 MB
fish: Job 1, './main -m models/ggml-small.b...' terminated by signal SIGILL (Illegal instruction)

I've tried other models as well, but the result is always the same.

[ggerganov]

That is interesting. What device are you using?

What is the output of:

make clean
make

[undefdev]

I'm using a MacBook Pro with M1 Max.

output of make clean:

rm -f *.o main libwhisper.a

output of make:

cc  -O3 -std=c11   -Wall -Wextra -Wno-unused-parameter -Wno-unused-function -pthread -mavx -mavx2 -mfma -mf16c   -c ggml.c
c++ -O3 -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wno-unused-function -pthread -c whisper.cpp
c++ -O3 -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wno-unused-function -pthread main.cpp whisper.o ggml.o -o main 
./main -h

usage: ./main [options] file0.wav file1.wav ...

options:
  -h,       --help           show this help message and exit
  -s SEED,  --seed SEED      RNG seed (default: -1)
  -t N,     --threads N      number of threads to use during computation (default: 4)
  -o N,     --offset N       offset in milliseconds (default: 0)
  -v,       --verbose        verbose output
            --translate      translate from source language to english
  -otxt,    --output-txt     output result in a text file
  -ovtt,    --output-vtt     output result in a vtt file
  -osrt,    --output-srt     output result in a srt file
  -ps,      --print_special  print special tokens
  -nt,      --no_timestamps  do not print timestamps
  -l LANG,  --language LANG  spoken language (default: en)
  -m FNAME, --model FNAME    model path (default: models/ggml-base.en.bin)
  -f FNAME, --file FNAME     input WAV file path

[ggerganov]

Ok, I see the problem - the Makefile incorrectly thinks that this is x86 architecture and adds AVX flags: -mavx -mavx2 -mfma -mf16c

To fix this, I need to know the output of the following commands:

uname -s
uname -p
uname -m

[undefdev]

Interesting... uname -mps gives me:

Darwin x86_64 i386

[ggerganov]

That's unexpected.

In the meantime, if you just want to give whisper.cpp a try, you should be able to compile it properly like this:

cc  -O3 -std=c11   -Wall -Wextra -Wno-unused-parameter -Wno-unused-function -pthread -DGGML_USE_ACCELERATE   -c ggml.c
c++ -O3 -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wno-unused-function -pthread -c whisper.cpp
c++ -O3 -std=c++11 -Wall -Wextra -Wno-unused-parameter -Wno-unused-function -pthread main.cpp whisper.o ggml.o -o main  -framework Accelerate

This is what I use on my MacBook M1 Pro and it works.

If you figure out why uname reports this as x86_64 please let me know, so I can somehow update the Makefile to support M1 Max.

[undefdev]

uname -v gives me Darwin Kernel Version 21.6.0: Mon Aug 22 20:19:52 PDT 2022; root:xnu-8020.140.49~2/RELEASE_ARM64_T6000 however... Apparently I'm not the first having this problem

Homebrew/discussions#2226
https://apple.stackexchange.com/questions/420452/running-uname-m-gives-x86-64-on-m1-mac-mini

I have also checked, and terminal is not running in rosetta.

[abelbabel]

On FreeBSD we have sysctl ... is there something equivalent on Mac OS (it's somehow one familiy ...)?

$ sysctl -n hw.machine_arch
amd64

on an ARM, it's like

aarch64

or maybe better:

$ sysctl -n kern.supported_archs
amd64 i386

or on ARM (Raspberry Pi 3 B+)

aarch64 armv7

[undefdev]

There is sysctl, but it works somewhat differently. So your oids aren't supported on Mac it seems.

Particularly useful queries are:

$ sysctl -n hw.optional.arm64
1
$ sysctl -n machdep.cpu.brand_string
Apple M1 Max

Some other interesting fields I've found:

machdep.cpu.features: FPU VME DE PSE TSC MSR PAE MCE CX8 APIC SEP MTRR PGE MCA CMOV PAT PSE36 CLFSH DS ACPI MMX FXSR SSE SSE2 SS HTT TM PBE SSE3 PCLMULQDQ DTSE64 MON DSCPL VMX EST TM2 SSSE3 CX16 TPR PDCM SSE4.1 SSE4.2 AES SEGLIM64

or

$ sysctl -a | grep arm
hw.optional.arm.FEAT_FlagM: 1
hw.optional.arm.FEAT_FlagM2: 1
hw.optional.arm.FEAT_FHM: 1
hw.optional.arm.FEAT_DotProd: 1
hw.optional.arm.FEAT_SHA3: 1
hw.optional.arm.FEAT_RDM: 1
hw.optional.arm.FEAT_LSE: 1
hw.optional.arm.FEAT_SHA256: 1
hw.optional.arm.FEAT_SHA512: 1
hw.optional.arm.FEAT_SHA1: 1
hw.optional.arm.FEAT_AES: 1
hw.optional.arm.FEAT_PMULL: 1
hw.optional.arm.FEAT_SPECRES: 0
hw.optional.arm.FEAT_SB: 1
hw.optional.arm.FEAT_FRINTTS: 1
hw.optional.arm.FEAT_LRCPC: 1
hw.optional.arm.FEAT_LRCPC2: 1
hw.optional.arm.FEAT_FCMA: 1
hw.optional.arm.FEAT_JSCVT: 1
hw.optional.arm.FEAT_PAuth: 1
hw.optional.arm.FEAT_PAuth2: 0
hw.optional.arm.FEAT_FPAC: 0
hw.optional.arm.FEAT_DPB: 1
hw.optional.arm.FEAT_DPB2: 1
hw.optional.arm.FEAT_BF16: 0
hw.optional.arm.FEAT_I8MM: 0
hw.optional.arm.FEAT_ECV: 1
hw.optional.arm.FEAT_LSE2: 1
hw.optional.arm.FEAT_CSV2: 1
hw.optional.arm.FEAT_CSV3: 1
hw.optional.arm.FEAT_FP16: 1
hw.optional.arm.FEAT_SSBS: 1
hw.optional.arm.FEAT_BTI: 0
hw.optional.armv8_1_atomics: 1
hw.optional.armv8_2_fhm: 1
hw.optional.armv8_2_sha512: 1
hw.optional.armv8_2_sha3: 1
hw.optional.armv8_3_compnum: 1
hw.optional.armv8_crc32: 1
hw.optional.armv8_gpi: 1
hw.optional.arm64: 1

[ggerganov]

@undefdev
I pushed a branch that uses sysctl -n hw.optional.arm64.
Can you give it a try and see if it works:

git checkout make-mac-os-arm

[undefdev]

@Vimy No, apparently it was an old homebrew installation that caused the issue. I hadn't reinstalled it since it gained m1 support.

@ggerganov I tried it, and it worked, albeit it was about 4x slower than when I just hardcoded UNAME_P := arm and
UNAME_M := arm64. It seems like you commented out the ifeq ($(SYSCTL_M),1) case, but if i put it back in it won't work.

In the meantime I also fixed my wrong arch output by reinstalling homebrew. Now the performance is also a lot better!
Before, 5 seconds of audio took about 20 seconds to transcribe. Now it's only a bit longer than one second!
So fixing the terminal is definitely worth it.

I think it's probably best to check sysctl and give a warning linking to this issue, if there's a contradicting output.
Otherwise I think it's easier to just overwrite the UNAME variables.
The code in the pull request I made should do that.

[ggerganov]

@undefdev
Thanks for the investigation! I like the proposed change and have merged it into upstream.