Bug: RPC server doesn't load GPU if I use Vulkan

[metal3d]

What happened?

I compiled llamacpp with Vulkan backend. The "rpc-server" binary is linked to libvulkan but it never uses my GPUs. While "llama-cli" is OK.

Name and Version

version: 3384 (4e24cff)
built with cc (GCC) 14.1.1 20240701 (Red Hat 14.1.1-7) for x86_64-redhat-linux

What operating system are you seeing the problem on?

Linux

Relevant log output

./rpc-server
create_backend: using CPU backend
Starting RPC server on 0.0.0.0:50052, backend memory: 23967 MB


ldd ./rpc-server
        linux-vdso.so.1 (0x00007f18759f2000)
        libllama.so => /home/metal3d/Projects/ML/llama.cpp/build-rpc/src/libllama.so (0x00007f1875879000)
        libggml.so => /home/metal3d/Projects/ML/llama.cpp/build-rpc/ggml/src/libggml.so (0x00007f1875400000)
        libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f1875000000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f187531c000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f187582b000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f1874e0f000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f18759f4000)
        libvulkan.so.1 => /lib64/libvulkan.so.1 (0x00007f18757af000)
        libgomp.so.1 => /lib64/libgomp.so.1 (0x00007f18752c6000)

[rgerganov]

The Vulkan backend is using the tensor->extra property which is not supported by the RPC backend. There is the same issues with the SYCL backend (PR #7682)

[xvim]

is any plan to support vulkan when using RPC backend?

[rgerganov]

I will try to find out how to avoid using tensor->extra in Vulkan. Maybe adding a global map ggml_tensor -> ggml_tensor_extra_gpu

[slaren]

The extras in the Vulkan backend are not really necessary, all the data that they contain is already present (directly or indirectly) in other fields of the tensor. At this point I think they are only there for legacy reasons, but could be removed with a refactor.

[metal3d]

Big thanks 👍 Le jeu. 3 oct. 2024, 12:01, Radoslav Gerganov ***@***.***> a écrit :

…

Closed #8536 <#8536> as completed via 841713e <841713e> . — Reply to this email directly, view it on GitHub <#8536 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAYN4BA5Z3SEW2IYQU5IDDZZUIWVAVCNFSM6AAAAABLAGD476VHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJUGUYDCOBWGEZDKMA> . You are receiving this because you authored the thread.Message ID: ***@***.***>