llama-server not using GPU

[RakshitAralimatti]

After I install llama-cpp-python-server with cuda support and run
python3 -m llama_cpp.server --model starcoderbase-3b/starcoderbase-3b.Q4_K_M.gguf --n_gpu_layers 10
The GPU is not getting used its running on the CPU

[pepijndevos]

I'm seeing the same problem with Vulkan.

I see all layers of the model actually getting loaded on the GPU, and nvtop shows significant memory use.
But then htop shows it's using 100% CPU and only a small blip of GPU.

llama.cpp itself works fine on the same hardware.

[pepijndevos]

Wait maybe it's actually using the GPU but just insanely bottlenecked on Python CPU performance?