GPU NOT used during "normal generation" when ONE LAYER offloaded (But GPU used in prompt evaluation)

[Nate687]

Hello,

I am using a GGUF version of dolphin-2.2-mistral-7B-GGUF (dolphin-2.2-mistral-7b.Q8_0.gguf) and have offloaded ONLY ONE LAYER of the model on the GPU using the "n-gpu-layers" options (since I am using a GPU having 2GB VRAM I can not offload anymore without getting a CUDA out of memory error during inference)

All works well during the prompt evaluation and I can see that the GPU being used (around 80% usage of the GPU is observed)

However the GPU usage immediately drops to 0 and the CPU usage reaches 100% (i.e no GPU usage) during the normal generation of the response (i.e once prompt is evaluated and the response of the model starts being printed out on the screen)

As such my "prompt eval time" is around 35 ms per token .... but the "eval time" is around 500 ms per token

I may be wrong, but I thought that even though only one layer was offloaded to the GPU, the GPU would be used during the normal generation of the response (to help bring down the "eval time")

Any guidance or tips on what I may be doing wrong would be highly appreciated.

Thanks,
Nate

[staviq]

That is pretty much "normal", prompt processing is GPU accelerated if you compiled llamacpp with GPU acceleration, and then when the actual inference happens, offloading just one layer doesn't add much performance benefit, but adds memory transfer overhead to and from the GPU.

Processing is still mostly done layer by layer, so the GPU works on the first layer, forwards the processing back to the CPU, and then the GPU sits idle while the rest of layers is being processed on the CPU.

So basically, you would want to offload enough layers to at least let the GPU make up for the memory transfer overhead with its compute speed, otherwise there are no gains from using the GPU.

With a 2G VRAM GPU, you might actually have better performance without offloading any layers and just having the prompt processing accelerated, but you need to test that as it depends a lot on a particular hardware.

[Dampfinchen]

Yes, I'm experiencing this too. Before, I was getting 250 ms/t generation time, now I'm getting 1300 ms/t with 25 layers offloaded running a 13B q4k_s model on my 6 GB RTX 2060. Prompt processing is unaffected.

[Dampfinchen]

I've fixed my problem by compiling it with AVX2 (it isn't compiled with it by default anymore)

But yeah, with 1 layer the GPU won't do much. That is normal.

[cduk]

That is pretty much "normal", prompt processing is GPU accelerated if you compiled llamacpp with GPU acceleration, and then when the actual inference happens, offloading just one layer doesn't add much performance benefit, but adds memory transfer overhead to and from the GPU.

I want to do just this. Have the prompt processing GPU accelerated then the rest by CPU. Is there anything I have to do to explicitly to enable the GPU acclerated prompt processing? I have layers to GPU as zero.

[cduk]

I've fixed my problem by compiling it with AVX2 (it isn't compiled with it by default anymore)

How do you compile it with AVX2?

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.