Feature Request: Ship llama.cpp binaries in AppImage format

[rgerganov]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

Shipping prebuilt binaries for Linux which "just work" is non-trivial task and using the AppImage format seems to be the best shot for this.

I don't know how feasible is to package a llama.cpp binary with all of the available backends but it should be doable for a single backend. For example llama-server-vulkan-x86_64.AppImage would package all of the dependencies needed for running llama-server with Vulkan backend. The user downloads this file and it "just works".

I think this can be combined with other solutions for Windows and MacOS into a single landing page where the user selects an OS, an application (e.g. llama-cli, llama-server, etc) and a backend and gets a downloadable file which is ready to run.

Thoughts?

Motivation

There should be an easy way to download and run llama.cpp binaries on various Linux distributions.

Possible Implementation

Ship llama.cpp binaries along with dependencies in AppImage format

[auriocus]

Llama.cpp itself doesn't have dependencies. You can run the binary anywhere. However the accelerated backend depend on the acceleration, it depends on eg cuda. I don't know what the policies for cuda redistribution is. For vulkan, my understanding is that you should not redistribute the library, because it is on the graphics driver and that depends on your card.

[rgerganov]

Llama.cpp itself doesn't have dependencies.

llama.cpp binaries depend on libllama, libggml, libggml-base, libcurl and bunch of other transitive dependencies. If you download llama-xxx-bin-ubuntu-x64.zip release from Github, unpack and run llama-cli, it may work because the dynamic loader will search for dependencies in the same dir. But if you move llama-cli to some other directory it will fail to run because of missing dependencies. Btw, right now it fails even without moving the files on Ubuntu 20 because of some glibc version mismatch:

➜  bin ./llama-cli
./llama-cli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./llama-cli)
./llama-cli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by ./llama-cli)
./llama-cli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by ./llama-cli)
./llama-cli: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by ./llama-cli)
./llama-cli: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by ./llama-cli)
./llama-cli: /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.13' not found (required by ./llama-cli)
./llama-cli: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by libllama.so)
./llama-cli: /lib/x86_64-linux-gnu/libstdc++.so.6: version `CXXABI_1.3.13' not found (required by libllama.so)
./llama-cli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by libllama.so)
./llama-cli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by libllama.so)
./llama-cli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by libggml.so)
./llama-cli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by libggml.so)
./llama-cli: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by libggml-base.so)
./llama-cli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by libggml-cpu.so)
./llama-cli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by libggml-cpu.so)
./llama-cli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by libggml-cpu.so)
./llama-cli: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29' not found (required by libggml-rpc.so)
./llama-cli: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by libggml-rpc.so)

For vulkan, my understanding is that you should not redistribute the library, because it is on the graphics driver and that depends on your card.

EmuDeck is packaging libvulkan.so in their AppImage and it works both on my Ubuntu workstation and my Steam Deck

[ggerganov]

Btw we discussed the distributions of binaries for Linux in this thread: #11392. The current binaries are targeting Ubuntu 22.04 and later and will not work for earlier versions because of libc version incompatibility.

In that thread, @slaren proposed an idea for making a .sh script that can be curld and piped to build llama.cpp from source on linux environments.

[rgerganov]

The current binaries are targeting Ubuntu 22.04

fair enough, my Ubuntu 20 is using glibc 2.31 which is quite old

In that thread, @slaren proposed an idea for making a .sh script that can be curld and piped to build llama.cpp from source on linux environments.

this could work for CPU but not for other backends

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.