Macbook Air M1 (2020) recommendation "--mlock" switch.

[olinorwell]

A very simple PSA: When using this particular machine you should use the --mlock switch.

Without it even with a 7B model it's painfully slow for me, perhaps 3 minutes to answer what are the days of the week.

As soon as I set that switch, it's almost real time, that is the answer appears at the speed you would say it if you were talking quite slowly.

I'm just putting this out here to hopefully help someone else out there. I've spent about 15 hours messing with this stuff this week and this is the first moment when I can finally run a model on my laptop offline in a usable manner.

So my suggestion is that the this switch should potentially even be the default, when running on machines with a similar configuration. Without it people will think the system is completely unusable, when the reality is that with the switch it feels extremely pleasant to use.

[x4080]

Hi, I'm using M2 pro mac mini 16GB, on my machine different is not significant, I found your post because of when I do summary/ q&a using pretty big prompt, the speed goes down. then I tried your mlock suggestion but makes no different

Edit : It seems in chat mode, --mlock is make a difference, for 1 shot in my case not really

[PiotrZak]

It helps a lot, my results after adding this flag:

llama_print_timings:        load time = 10035.29 ms
llama_print_timings:      sample time =   349.98 ms /   463 runs   (    0.76 ms per token,  1322.92 tokens per second)
llama_print_timings: prompt eval time =  1180.43 ms /    14 tokens (   84.32 ms per token,    11.86 tokens per second)
llama_print_timings:        eval time = 49680.74 ms /   462 runs   (  107.53 ms per token,     9.30 tokens per second)
llama_print_timings:       total time = 51264.31 ms

[ggerganov]

What are the numbers without this flag and what is your system spec?

[ianscrivener]

@ozzeruk,
Did you build with METAL and run with GPU/Metal?
Improves performance significantly!

# build with MacOS Metal
LLAMA_METAL=1 make -j

# run with Metal
./main -ngl 1  ...

[ianscrivener]

For reference, I have a MacBook Pro M2 16Gb RAM. With a 5.5Gb llama2 model my times are

• first run: 31.47 seconds
• second run: 13.18 seconds

These are timings with time ./main
... they differ considerably from llama_print_timings: total time = 5216.87 ms