main.py not running on M1 Mac due to llama_context_default_params symbol not found

[keldenl]

Things were working fine until i closed my terminal window and opened a new one and starting seeing issues (don't remember the error). I went ahead and did a quick update (via "development") steps in readme and started getting this issue when running python3 -m llama_cpp.server

Traceback (most recent call last):
  File "/Users/kelden/opt/anaconda3/lib/python3.9/runpy.py", line 188, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/Users/kelden/opt/anaconda3/lib/python3.9/runpy.py", line 111, in _get_module_details
    __import__(pkg_name)
  File "/Users/kelden/Documents/tiny-leaps/llama-cpp-python/llama_cpp/__init__.py", line 1, in <module>
    from .llama_cpp import *
  File "/Users/kelden/Documents/tiny-leaps/llama-cpp-python/llama_cpp/llama_cpp.py", line 99, in <module>
    _lib.llama_context_default_params.argtypes = []
  File "/Users/kelden/opt/anaconda3/lib/python3.9/ctypes/__init__.py", line 395, in __getattr__
    func = self.__getitem__(name)
  File "/Users/kelden/opt/anaconda3/lib/python3.9/ctypes/__init__.py", line 400, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x308a36490, llama_context_default_params): symbol not found

I've gone in and done make in llama.cpp again, run the develop script again and again to no avail. deleted the .so file and rebuilt it multiple times, made sure the MODEL variable is set properly too :/ what am i doing wrong

[keldenl]

Noting that I did force terminate the process and terminal window the last time this was working, immediately opened a new one and started having these issues

[keldenl]

I blew everything up and re-cloned and installed the repo from scratch and it resolved!

But... the response speed is even slower than before?? like 10x slower than I reported here a couple days ago (which was already slow) #49 (comment)

llama_print_timings:        load time =  9864.39 ms
llama_print_timings:      sample time =     5.98 ms /     8 runs   (    0.75 ms per run)
llama_print_timings: prompt eval time = 29445.04 ms /    24 tokens ( 1226.88 ms per token)
llama_print_timings:        eval time =  9915.22 ms /     8 runs   ( 1239.40 ms per run)
llama_print_timings:       total time = 39367.82 ms
INFO:     ::1:51331 - "POST /v1/completions HTTP/1.1" 200 OK

llama_print_timings:        load time =  9864.39 ms
llama_print_timings:      sample time =     4.48 ms /     6 runs   (    0.75 ms per run)
llama_print_timings: prompt eval time = 86300.92 ms /    70 tokens ( 1232.87 ms per token)
llama_print_timings:        eval time =  6203.12 ms /     5 runs   ( 1240.62 ms per run)
llama_print_timings:       total time = 92510.58 ms
INFO:     ::1:51430 - "POST /v1/chat/completions HTTP/1.1" 200 OK

120ms -> 1200ms per token :(

[jmtatsch]

On my m1 MacBook I had very slow generation speeds with llama.cpp until I set the --mlock flag to force it to keep everything in memory. Did you try that?

[abetlen]

@keldenl I'll close this issue and try to address the performance issues in #49