Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve instructions for CPUs without AVX2 #3786

Merged
merged 2 commits into from
Sep 11, 2023
Merged

Improve instructions for CPUs without AVX2 #3786

merged 2 commits into from
Sep 11, 2023

Conversation

netrunnereve
Copy link
Contributor

See #3731 for more details.

Checklist:

@jllllll
Copy link
Contributor

jllllll commented Sep 8, 2023

You can also get versions of llama_cpp_python_cuda and llama_cpp_python_ggml_cuda without AVX2 by changing the cu117 in the wheel name to cu117avx or cu117basic.

Sorry for the confusing naming schemes. There are just so many build configurations for llama-cpp-python and Python doesn't give many options for differentiating wheels of different configurations. In fact, Python devs are planning to eventually remove the version tagging system that I'm using. No clue why. Hopefully they reconsider or implement a better system in the future as that will make it significantly more difficult to manage and organize the 6367 wheels I currently have in my repo.


An alternative, more convenient approach to installing compatible versions of these packages is to use this command:

python -m pip install llama-cpp-python llama-cpp-python-cuda llama-cpp-python-ggml llama-cpp-python-ggml-cuda --force-reinstall --no-deps --index-url=https://jllllll.github.io/llama-cpp-python-cuBLAS-wheels/AVX/cpu --extra-index-url=https://jllllll.github.io/llama-cpp-python-cuBLAS-wheels/textgen/AVX/cu117

This will install AVX versions of all the llama-cpp-python packages used in the webui. You can change AVX to basic in the index URLs to install SSE versions instead.

@oobabooga oobabooga merged commit 92f3cd6 into oobabooga:main Sep 11, 2023
@carlblacknz
Copy link

The AVX versions of these don't work on my Ivy Bridge Xeon, but using basic does. Are they still using FP16 maybe?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants