How to run Llama3-8b instruct model on multiple GPUs? #7086
Replies: 3 comments 2 replies
-
|
Why would you run it on multiple GPUs? |
Beta Was this translation helpful? Give feedback.
-
|
to test benchmark. other models seems to be supported on multiple GPUs. |
Beta Was this translation helpful? Give feedback.
-
|
It only actually runs inference on one GPU at a time. You get the benefit of the additional VRAM but not the benefit of the additional processing power. Llama 3 8B should work on multiple GPUs. Your question doesn't really provide a lot of context. To maximize the possibility of someone being able to understand your issue and to help you, please review this article on How To Ask Questions The Smart Way. |
Beta Was this translation helpful? Give feedback.
-
|
One reason for trying to run on multiple GPUs, as in my case is, it does not fit on one. I have 2 Nividia 2080 tI, each has around 11GB of memory. By default it choses GPU:0. I was wondering if I can parallelize between the two GPUs to run it. |
Beta Was this translation helpful? Give feedback.
-
|
The Q4_K_M quant is about 4.92 GB. It should fit in 11 GB of VRAM with space left for inference. It sounds like you might be interested in this discussion: #8725 |
Beta Was this translation helpful? Give feedback.
-
Deos llama.cpp support llama3-8b to run on multiple GPUs?
Beta Was this translation helpful? Give feedback.
All reactions