Push Docker images to Dockerhub using Github actions for running a llama-cpp-python REST server

[gjmulder]

A lot of people would like to run their own server, but don't have the necessary DevOps skills to configure and build a llama-cpp-python + python + llama.cpp environment.

I'm working on developing some Dockerfiles that are run via a Github action to publish to Docker Hub similar to llama.cpp's workflows/docker.yml for both OpenBLAS (i.e. no NVidia GPU) and CuBLAS (NVidia GPU via Docker) support.

Which CC licensed models are now available that are compatible with llama.cpp's new quantized format? Ideally we want to start with small models to keep the Docker image sizes manageable.

[gjmulder]

Looks like upstream support of Replit + MPT is maturing with less restrictive licensing:

• replit-code-v1-3b is: CC BY-SA-4.0
• MPT-7B-StoryWriter-65k+ is: Apache 2.0
• MPT-7B-Instruct is: CC-By-SA-3.0
• MPT-7B-Chat is: CC-By-NC-SA-4.0

[jmtatsch]

While in principle a great idea this will be a support nightmare.

The problem is that the binaries of llama.cpp are so optimized to using every possible acceleration feature that they will not run on the many different hardware architectures (especially the older ones which non power users are likely to have).
It will segfault then with a non-descriptive error message and people will be crying in the issues 😉
When building without those acceleration features it will become unusable slow.

[gjmulder]

It will segfault then with a non-descriptive error message and people will be crying in the issues

@jmtatsch better forewarned than snowed in issues, thx. How about if the images require a docker build against whatever Docker exposes in terms of local hardware, but come pre-loaded with models?

[jmtatsch]

Yes, that would solve the issue and the model can be downloaded within the build from huggingface.

[gjmulder]

#270 does most of the heavy lifting. Seems like people are already pushing images to Docker Hub and running into "illegal instructions" due to AVX, etc.

[gjmulder]

FORCE_CMAKE=1 \
CMAKE_ARGS="\
-DLLAMA_AVX=ON -DFMA=ON -DF16C=ON -D SSE3=ON \
-DLLAMA_AVX2=OFF -DLLAMA_AVX512=OFF -DAVX512_VBMI=OFF -AVX512_VNNI=OFF \
-DLLAMA_OPENBLAS=ON"

might be crippled enough to work for most people on x86_64

[gjmulder]

And we now have open llama w/Apache License Version 2.0