Performance of llama.cpp with Vulkan

[netrunnereve]

This is similar to the Apple Silicon benchmark thread, but for Vulkan! Many improvements have been made to the Vulkan backend and I think it's good to consolidate and discuss our results here.

We'll be testing the Llama 2 7B model like the other thread to keep things consistent, and use Q4_0 as it's simple to compute and small enough to fit on a 4GB GPU. You can download it here.

Instructions

Either run the commands below or download one of our Vulkan releases.

wget https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q4_0.gguf
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
mkdir build
cd build
cmake .. -DGGML_VULKAN=on -DCMAKE_BUILD_TYPE=Release
make
llama-bench -m ../../llama-2-7b.Q4_0.gguf -ngl 100 (add any extra options here)

Share your llama-bench results along with the git hash and Vulkan info string in the comments. Feel free to try other models and compare backends, but only valid runs will be placed on the scoreboard.

If multiple entries are posted for the same device the one with the highest tg128 score will be used. Performance may vary depending on driver, operating system, board manufacturer, etc. even if the chip is the same.

Vulkan Scoreboard for Llama 2 7B, Q4_0 (no FA)

GPU	pp512 t/s	tg128 t/s	Commit	Comments
AMD Radeon RX 7900 XTX	3236.63 ± 71.63	148.21 ± 0.94	902368a	Best of multiple submissions
Nvidia RTX 3090	3301.47 ± 33.76	123.72 ± 0.14	0d52a69
AMD Radeon RX 7800 XT	1260.54 ± 10.51	107.53 ± 0.07	ee02ad0
AMD Radeon RX 6900 XT	1257.98 ± 1.55	101.42 ± 0.02	44e18ef	Best of multiple submissions
Nvidia RTX 4070	3179.37 ± 46.16	92.29 ± 0.28	9a48399
AMD Radeon RX 6800 XT	863.03 ± 0.70	91.59 ± 0.40	0d52a69
AMD Radeon Instinct MI60	369.26 ± 2.48	78.16 ± 1.40	504af20
AMD Radeon Pro VII	329.86 ± 0.80	75.22 ± 0.05	2739a71	Best of multiple submissions
AMD Radeon Instinct MI50	348.66 ± 0.55	72.06 ± 0.07	d80be89	Best of multiple submissions
AMD Radeon RX 5700 XT	439.42 ± 0.28	70.13 ± 0.05	c05e8c9
Nvidia RTX 2070 SUPER	1199.13 ± 7.70	64.64 ± 0.20	b7552cf
Nvidia RTX 3080	1706.07 ± 139.33	62.16 ± 1.98	4da69d1	Result appears lower than expected, maybe non-release build?
AMD Radeon Instinct MI25	439.42 ± 0.34	54.69 ± 0.03	2739a71
Nvidia RTX 3060	1298.03 ± 23.40	54.28 ± 1.05	6171c9d
AMD Radeon RX 6600 XT	574.65 ± 0.86	53.92 ± 0.11	091592d	Best of multiple submissions
Intel Arc A770	314.24 ± 1.04	45.22 ± 0.25	ba8a1f9	Windows result, Linux is slower in pp512
Intel Arc B580	175.56 ± 2.65	44.12 ± 0.09	9a48399
Nvidia RTX 4050 Mobile	1154.28 + 15.76	41.89 + 0.10	d79d8f3
AMD RX 470	161.47 ± 0.43	33.45 ± 0.04	4da69d1
AMD FirePro W8100	137.10 ± 0.44	28.51 ± 0.12	4da69d1
Intel Arc A750	88.86 ± 0.14	27.57 ± 0.03	8d59d91
Apple M3 MacBook Pro	263.70 ± 0.02	26.39 ± 0.14	b9ab0a4	MoltenVK with special patch
AMD FirePro S10000	94.78 ± 0.02	25.32 ± 0.02	914a82d	Two GPU chips on one card
AMD Ryzen 8840HS	245.79 ± 2.97	20.10 ± 0.07	19d3c82
AMD Ryzen Z1 Extreme	199.36 ± 7.02	18.77 ± 0.02	53ff6b9
Apple M2 MacBook Air	38.67 ± 0.03	11.07 ± 0.04	017cc5f	Asahi Linux
MediaTek Dimensity 9400	38.36 ± 15.15	8.92 ± 0.06	b9ab0a4	GPU supports coopmat but pp512 is faster with it turned off
Intel i7-1185G7	42.02 ± 0.07	7.28 ± 0.24	ff3fcab
AMD Ryzen 3400G	46.47 ± 5.15	5.99 ± 0.71	0893e01
Intel Core i7-1065G7	25.58 ± 0.00	4.25 ± 0.18	N/A
Intel i5-8350U	25.28 ± 0.00	3.23 ± 0.00	f26c874

Vulkan Scoreboard for Llama 2 7B, Q4_0 (with FA)

GPU	pp512 t/s	tg128 t/s	Commit	Comments
Nvidia RTX 3090	4761.91 ± 5.84	107.32 ± 0.34	4dbc8b9	coopmat2
Nvidia RTX 4070	4293.57 ± 27.70	91.49 ± 0.89	9a48399	coopmat2

[netrunnereve]

AMD FirePro W8100

ggml_vulkan: 0 = AMD Radeon FirePro W8100 (RADV HAWAII) (radv) | uma: 0 | fp16: 0 | warp size: 64 | matrix cores: none
build: 4da69d1a (4351)

model	size	params	backend	ngl	threads	sm	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	8	none	pp512	137.10 ± 0.44
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	8	none	tg128	28.51 ± 0.12

[netrunnereve]

AMD RX 470

ggml_vulkan: 1 = AMD Radeon RX 470 Graphics (RADV POLARIS10) (radv) | uma: 0 | fp16: 0 | warp size: 64 | matrix cores: none
build: 4da69d1a (4351)

model	size	params	backend	ngl	threads	main_gpu	sm	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	8	1	none	pp512	161.47 ± 0.43
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	8	1	none	tg128	33.45 ± 0.04

[max-krasnyansky]

ubuntu 24.04, vulkan and cuda installed from official APT packages.

ggml_vulkan: 0 = NVIDIA GeForce RTX 3080 (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	99	pp512	1706.07 ± 139.33
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	99	tg128	62.16 ± 1.98

build: 4da69d1 (4351)

vs CUDA on the same build/setup

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3080, compute capability 8.6, VMM: yes

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	pp512	4499.47 ± 60.66
llama 7B Q4_0	3.56 GiB	6.74 B	CUDA	99	tg128	131.01 ± 0.43

build: 4da69d1 (4351)

[hkbu-kennycheng]

Macbook Air M2 on Asahi Linux

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Apple M2 (G14G B0) (Honeykrisp) | uma: 1 | fp16: 1 | warp size: 32 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	38.67 ± 0.03
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	11.07 ± 0.04

[build build: 017cc5f](build: 017cc5f)

[ericcurtin]

For the record I think this is slow on the HoneyKrisp side rather than llama.cpp

[hkbu-kennycheng]

Gentoo Linux on ROG Ally (2023) Ryzen Z1 Extreme

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1103_R1) (radv) | uma: 1 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	199.36 ± 7.02
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	18.77 ± 0.02

[build build: 53ff6b9](build: 53ff6b9)

[hkbu-kennycheng]

ggml_vulkan: Found 4 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
ggml_vulkan: 2 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
ggml_vulkan: 3 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	1545.39 ± 6.58
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	88.12 ± 1.06

[build build: 53ff6b9](build: 53ff6b9)

[0cc4m]

Cool setup! Could you also post the result of 1, 2 and 3 7900 XTX GPUs? You can use only the first GPU with export GGML_VK_VISIBLE_DEVICES=0, the first two with export GGML_VK_VISIBLE_DEVICES=0,1 and so on.

[hkbu-kennycheng]

env GGML_VK_VISIBLE_DEVICES=0 ./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	2022.59 ± 10.08
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	136.24 ± 0.30

env GGML_VK_VISIBLE_DEVICES=1 ./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	2039.24 ± 18.08
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	140.68 ± 2.09

env GGML_VK_VISIBLE_DEVICES=2 ./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	2062.17 ± 5.36
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	143.99 ± 0.23

env GGML_VK_VISIBLE_DEVICES=3 ./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	1997.04 ± 5.78
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	136.98 ± 1.73

env GGML_VK_VISIBLE_DEVICES=0,1 ./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100
ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	1668.19 ± 12.78
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	100.62 ± 0.66

env GGML_VK_VISIBLE_DEVICES=0,1,2 ./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100
ggml_vulkan: Found 3 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
ggml_vulkan: 2 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	1566.38 ± 8.01
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	97.96 ± 1.13

env GGML_VK_VISIBLE_DEVICES=0,1,2,3 ./build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100
ggml_vulkan: Found 4 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
ggml_vulkan: 1 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
ggml_vulkan: 2 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat
ggml_vulkan: 3 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	1484.04 ± 6.01
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	91.48 ± 0.63

[netrunnereve]

For this multi GPU case getting Vulkan to support #6017 pipeline parallelism might help improve the prompt processing speed.

[hkbu-kennycheng]

@netrunnereve I updated the commit id in all my result.

[0cc4m]

build: 0d52a69 (4439)

NVIDIA GeForce RTX 3090 (NVIDIA)

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	3301.47 ± 33.76
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	123.72 ± 0.14

AMD Radeon RX 6800 XT (RADV NAVI21) (radv)

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	863.03 ± 0.70
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	91.59 ± 0.40

AMD Radeon (TM) Pro VII (RADV VEGA20) (radv)

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	312.02 ± 0.97
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	70.17 ± 0.25

Intel(R) Arc(tm) A770 Graphics (DG2) (Intel open-source Mesa driver)

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	95.52 ± 0.12
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	44.49 ± 0.03

[0cc4m]

@netrunnereve Some of the tg results here are a little low, I think they might be debug builds. The cmake step (at least on Linux) might require cmake .. -DGGML_VULKAN=on -DCMAKE_BUILD_TYPE=Release

[netrunnereve]

I've added -DCMAKE_BUILD_TYPE=Release to the post, but honestly I've always built without this flag for both Vulkan and CPU backends and never noticed a difference in performance. Having Release set might strip the debug symbols but it shouldn't affect the compiler optimizations.

My release numbers for the RX 470 are basically identical to the ones I posted earlier without the flag.

model	size	params	backend	ngl	threads	main_gpu	sm	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	8	1	none	pp512	160.08 ± 0.38
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	8	1	none	tg128	33.41 ± 0.15

[0cc4m]

Maybe not in your case, but some other results are suspiciously low in tg (for example the RTX 3080)

[qnixsynapse]

Build: 8d59d91 (4450)
ggml_vulkan: 0 = Intel(R) Arc(tm) A750 Graphics (DG2) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | warp size: 32 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	99	pp512	88.86 ± 0.14
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	99	tg128	27.57 ± 0.03

Lack of proper Xe coopmat support in the ANV driver is a setback honestly.
Compared to SYCL:

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	SYCL	99	pp512	1616.11 ± 5.28
llama 7B Q4_0	3.56 GiB	6.74 B	SYCL	99	tg128	36.64 ± 0.05

edit: retested both with the default batch size.

[0cc4m]

They do have vtune but it needs a third party kernel module to run which I don't like tbh.

Also, I don't know whether it supports Vulkan apps or not. But it does seem to support opencl.

I put my A770 into a Windows PC and gave Intel GPA and vtune a shot: GPA just crashes most of the time, I couldn't get it to trace anything useful. vtune works, but does not support Vulkan. It just shows some high-level metrics in that case, not really useful sadly.

[qnixsynapse]

Your Vulkan tg result is lower than expected, can you retry with the cmake build type set like in the updated instructions? It might be due to a debug build.

I did build it with cmake with build type Release.

[0cc4m]

In that case it's something else, cause it should be performing similarly to my A770. I suspect the mesa version, there was something in newer mesa versions that slowed down tg on Intel.

[qnixsynapse]

A750 has 448 CUs, A770 has 512 CUs I think. Personally, I am not worried about tg. I am worried about pp here. The gemm batch quickly saturates my GPU.

[qnixsynapse]

@0cc4m https://gitlab.freedesktop.org/mesa/mesa/-/issues/12585

[0cc4m]

Here's something exotic: An AMD FirePro S10000 dual GPU from 2012 with 2x 3GB GDDR5.

build: 914a82d (4452)

ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD FirePro W8000 (RADV TAHITI) (radv) | uma: 0 | fp16: 0 | warp size: 64 | matrix cores: none
ggml_vulkan: 1 = AMD FirePro W8000 (RADV TAHITI) (radv) | uma: 0 | fp16: 0 | warp size: 64 | matrix cores: none

model	size	params	backend	ngl	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	99	8	pp512	94.78 ± 0.02
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	99	8	tg128	25.32 ± 0.02

[netrunnereve]

Very interesting, and looks like it's pretty close to the W8100 in tg despite being a dual GPU card. Your backend scales pretty well with layer splitting which is why I find it worthwhile to run my RX470 and W8100 together (I end up getting results that are close to the average of both cards).

ggml_vulkan: Found 2 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon FirePro W8100 (RADV HAWAII) (radv) | uma: 0 | fp16: 0 | warp size: 64 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon RX 470 Graphics (RADV POLARIS10) (radv) | uma: 0 | fp16: 0 | warp size: 64 | matrix cores: none

model	size	params	backend	ngl	threads	main_gpu	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	8	1	pp512	147.84 ± 0.38
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	8	1	tg128	30.77 ± 0.00

[vkhodygo]

Latest arch with Vulkan Instance Version: 1.4.303 on a i7-1185G7 laptop. The config is not completely stock, I had to deal with thermals ages ago to boost the performance, so it doesn't throttle.

For the sake of consistency I run every bit in a script and also build every target from scratch (for some reason cmake doesn't want to clean everything):

kill -STOP -1

timeout 240s $COMMAND

kill -CONT -1

Vulkan only:

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Iris(R) Xe Graphics (TGL GT2) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | warp size: 32 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	42.02 ± 0.07
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	7.28 ± 0.24

build: ff3fcab (4459)

Vulkan and OpenBLAS w/ default 4 threads:

model	size	params	backend	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan,BLAS	4	pp512	42.05 ± 0.04
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan,BLAS	4	tg128	7.35 ± 0.26

This bit seems to underutilise both GPU and CPU in real conditions based on top activities.

Vulkan and OpenBLAS w/ default 8 threads:

model	size	params	backend	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan,BLAS	8	pp512	41.89 ± 0.06
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan,BLAS	8	tg128	7.22 ± 0.20

[0cc4m]

Unless you reduce the number of GPU layers, threads and openblas/non-openblas is not gonna make any difference. Try it with ngl 0, then only prompt processing is accelerated using Vulkan, the rest runs on CPU. This is often a good setting for integrated GPUs.

[vkhodygo]

That's something I didn't think about, with -ngl 0 it goes like this:

model	size	params	backend	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan,BLAS	4	pp512	30.51 ± 0.25
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan,BLAS	4	tg128	9.87 ± 0.05

build: ba8a1f9 (4460)

model	size	params	backend	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan,BLAS	8	pp512	32.11 ± 0.45
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan,BLAS	8	tg128	9.49 ± 0.18

[vkhodygo]

It seems latest patches has improved the results a bit:

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Iris(R) Xe Graphics (TGL GT2) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | warp size: 32 | matrix cores: none

model	size	params	backend	ngl	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	1	pp512	50.86 ± 0.03
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	1	tg128	8.30 ± 0.05
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	2	pp512	50.90 ± 0.01
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	2	tg128	8.11 ± 0.25
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	4	pp512	50.91 ± 0.02
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	4	tg128	7.99 ± 0.25
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	8	pp512	50.89 ± 0.04
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	8	tg128	7.92 ± 0.24

[0cc4m]

Intel ARC A770 on Windows:

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	99	pp512	314.24 ± 1.04
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	99	tg128	45.22 ± 0.25

build: ba8a1f9 (4460)

[8XXD8]

Single GPU Vulkan

Radeon Instinct MI25

ggml_vulkan: 0 = AMD Radeon Instinct MI25 (RADV VEGA10) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	439.42 ± 0.34
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	54.69 ± 0.03

build: 2739a71 (4461)

Radeon PRO VII

ggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	329.86 ± 0.80
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	75.22 ± 0.05

build: 2739a71 (4461)

Multi GPU Vulkan

ggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
ggml_vulkan: 2 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	324.55 ± 0.55
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	38.39 ± 0.09

build: 2739a71 (4461)

ggml_vulkan: 0 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
ggml_vulkan: 1 = AMD Radeon Instinct MI25 (RADV VEGA10) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
ggml_vulkan: 2 = AMD Radeon Instinct MI25 (RADV VEGA10) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
ggml_vulkan: 3 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
ggml_vulkan: 4 = AMD Radeon Pro VII (RADV VEGA20) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 70B Q5_K - Medium	46.51 GiB	70.55 B	Vulkan	100	pp512	32.29 ± 0.04
llama 70B Q5_K - Medium	46.51 GiB	70.55 B	Vulkan	100	tg128	4.75 ± 0.00

build: 2739a71 (4461)

Single GPU Rocm

Device 0: AMD Radeon Instinct MI25, compute capability 9.0, VMM: no

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	100	pp512	409.83 ± 0.23
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	100	tg128	63.94 ± 0.06

build: 2739a71 (4461)

Device 0: AMD Radeon Pro VII, compute capability 9.0, VMM: no

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	100	pp512	1064.99 ± 1.18
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	100	tg128	87.45 ± 0.04

build: 2739a71 (4461)

Multi GPU Rocm

Device 0: AMD Radeon Pro VII, compute capability 9.0, VMM: no
Device 1: AMD Radeon Pro VII, compute capability 9.0, VMM: no
Device 2: AMD Radeon Pro VII, compute capability 9.0, VMM: no

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	100	pp512	1061.87 ± 0.26
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	100	tg128	81.49 ± 0.41

build: 2739a71 (4461)

Layer split
Device 0: AMD Radeon Pro VII, compute capability 9.0, VMM: no
Device 1: AMD Radeon Pro VII, compute capability 9.0, VMM: no
Device 2: AMD Radeon Pro VII, compute capability 9.0, VMM: no
Device 3: AMD Radeon Instinct MI25, compute capability 9.0, VMM: no
Device 4: AMD Radeon Instinct MI25, compute capability 9.0, VMM: no

model	size	params	backend	ngl	test	t/s
llama 70B Q5_K - Medium	46.51 GiB	70.55 B	ROCm	100	pp512	16.36 ± 0.02
llama 70B Q5_K - Medium	46.51 GiB	70.55 B	ROCm	100	tg128	6.43 ± 0.01

build: 2739a71 (4461)

Row split
Device 0: AMD Radeon Pro VII, compute capability 9.0, VMM: no
Device 1: AMD Radeon Pro VII, compute capability 9.0, VMM: no
Device 2: AMD Radeon Pro VII, compute capability 9.0, VMM: no
Device 3: AMD Radeon Instinct MI25, compute capability 9.0, VMM: no
Device 4: AMD Radeon Instinct MI25, compute capability 9.0, VMM: no

model	size	params	backend	ngl	sm	test	t/s
llama 70B Q5_K - Medium	46.51 GiB	70.55 B	ROCm	100	row	pp512	30.86 ± 0.03
llama 70B Q5_K - Medium	46.51 GiB	70.55 B	ROCm	100	row	tg128	12.52 ± 0.21

build: 2739a71 (4461)

Single GPU speed is decent, but multi GPU trails Rocm by a wide margin, especially with large models due to the lack of row split.

[cb88]

What is the power profile for this MI25? Mine is 110W but its running slower than yours on git from today.

[8XXD8]

Mine defaults to 220w.
You can increase the power with rocm-smi --setpoweroverdrive 220

[daniandtheweb]

AMD Radeon RX 5700 XT on Arch using mesa-git and setting a higher GPU power limit compared to the stock card.
build: c05e8c9 (4462)

Vulkan:

ggml_vulkan: 0 = AMD Radeon RX 5700 XT (RADV NAVI10) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	439.42 ± 0.28
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	70.13 ± 0.05

HIP:

  Device 0: AMD Radeon RX 5700 XT, compute capability 10.1, VMM: no

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	100	pp512	354.17 ± 0.18
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	100	tg128	67.55 ± 0.04

I also think it could be interesting adding the flash attention results to the scoreboard (even if the support for it still isn't as mature as CUDA's).

Vulkan FA:

ggml_vulkan: 0 = AMD Radeon RX 5700 XT (RADV NAVI10) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	1	pp512	214.48 ± 2.31
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	1	tg128	23.21 ± 0.08

HIP FA:

  Device 0: AMD Radeon RX 5700 XT, compute capability 10.1, VMM: no

model	size	params	backend	ngl	fa	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	100	1	pp512	314.17 ± 0.29
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	100	1	tg128	62.02 ± 0.05

[0cc4m]

There is no Vulkan flash attention support (except with coopmat2 on very new nvidia drivers). What you're measuring here is a CPU fallback.

[daniandtheweb]

I see, I was sure about the CPU fallback but didn't know there was no flash attention support at all.

[FNsi]

I tried but there's nothing after 1 hrs , ok, might be 40 mins...

Anyway I run the llama_cli for a sample eval...

build: 4419 (46e3556e)

./llama-cli -m ~/storage/llama-2-7b.Q4_0.gguf -p "can u" -ngl 100                         ggml_vulkan: Found 1 Vulkan devices:                  ggml_vulkan: 0 = Mali-G57 (Mali-G57) | uma: 1 | fp16: 1 | warp size: 16 | matrix cores: none                build: 4419 (46e3556e) with clang version 19.1.6 for aarch64-unknown-linux-android24

llama_perf_sampler_print:    sampling time =       3.31 ms /    24 runs   (    0.14 ms per token,  7242.00 tokens per second)                                     llama_perf_context_print:        load time =   28544.85 ms                                                  llama_perf_context_print: prompt eval time =    3788.63 ms /     3 tokens ( 1262.88 ms per token,     0.79 tokens per second)                                     llama_perf_context_print:        eval time =   23248.44 ms /    20 runs   ( 1162.42 ms per token,     0.86 tokens per second)                                     llama_perf_context_print:       total time =   27591.65 ms /    23 tokens

Meanwhile OpenBLAS

llama_perf_sampler_print:    sampling time =       5.00 ms /    43 runs   (    0.12 ms per token,  8608.61 tokens per second)                                     llama_perf_context_print:        load time =   10871.74 ms                                                  llama_perf_context_print: prompt eval time =    1228.38 ms /     3 tokens (  409.46 ms per token,     2.44 tokens per second)                                     llama_perf_context_print:        eval time =   17010.39 ms /    39 runs   (  436.16 ms per token,     2.29 tokens per second)                                     llama_perf_context_print:       total time =   18639.62 ms /    42 tokens

[netrunnereve]

Even at below 1t/s llama-bench shouldn't run for an hour. The support just isn't there atm for Vulkan on Android.

[FNsi]

Truth is ...

(0.79 tokens per second),

3788.63 ms / 3 tokens

So it's not even...it just slower...

[zaps166]

Linux 6.12.9, Radeon RX 6900 XT, build 44e18ef (4503), pp_power_profile_mode set to VR, 2489 MHz GPU clock

---

AMDGPU-PRO (6.3.0)

ggml_vulkan: 0 = AMD Radeon RX 6900 XT (AMD proprietary driver) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	1257.98 ± 1.55
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	101.42 ± 0.02

---

Mesa RADV 23.3.3

ggml_vulkan: 0 = AMD Radeon RX 6900 XT (RADV NAVI21) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	1018.94 ± 2.02
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	95.70 ± 0.01

---

AMD HIP (6.2.4) for comparison

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 6900 XT, compute capability 10.3, VMM: no

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	100	pp512	1963.96 ± 2.14
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	100	tg128	85.48 ± 0.17

[3Simplex]

Windows 11, AMD RX6900XT, AMD Ryzen 7 5800X 8-Core Processor 3.80 GHz

.\llama-bench.exe -m "...\llama-2-7b.Q4_0.gguf" -ngl 99

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 6900 XT (AMD proprietary driver) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
ggml_vulkan: Compiling shaders.............................................Done!
| llama 7B Q4_0                  |   3.56 GiB |     6.74 B | Vulkan     |  99 |         pp512 |       1118.54 ± 1.33 |
| llama 7B Q4_0                  |   3.56 GiB |     6.74 B | Vulkan     |  99 |         tg128 |         98.09 ± 0.13 |

build: 6171c9d2 (4524)

[3Simplex]

.\llama-bench.exe -m "...\llama-2-7b.Q4_0.gguf" -ngl 99
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 6900 XT (AMD proprietary driver) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
| llama 7B Q4_0                  |   3.56 GiB |     6.74 B | Vulkan     |  99 |         pp512 |       1121.36 ± 0.91 |
| llama 7B Q4_0                  |   3.56 GiB |     6.74 B | Vulkan     |  99 |         tg128 |         98.89 ± 0.03 |

build: d774ab3a (4644)

[krga]

• Nvidia RTX 3060 (12GB)
• Manjaro Linux (Arch Linux), Linux Kernel 6.6.65
• Build: 6171c9d

./bin/llama-bench -m ../../llama-2-7b.Q4_0.gguf -ngl 100

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 3060 (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	1298.03 ± 23.40
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	54.28 ± 1.05

[Diablo-D3]

• 7900XTX (Powercolor Red Devil)
• Ryzen 9800x3D
• Windows 11 24H2 26100.2894
• Build: 902368a

./llama-b4646-bin-win-vulkan-x64/llama-bench.exe -m ./TheBloke/Llama-2-7B-GGUF/llama-2-7b.Q4_0.gguf -ngl 100 -r 100

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (AMD proprietary driver) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan,RPC	100	pp512	3236.63 ± 71.63
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan,RPC	100	tg128	148.21 ± 0.94

./llama-b4646-bin-win-hip-x64-gfx1100/llama-bench.exe -m ./TheBloke/Llama-2-7B-GGUF/llama-2-7b.Q4_0.gguf -ngl 100 -r 100

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm,RPC	100	pp512	3266.07 ± 29.65
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm,RPC	100	tg128	157.91 ± 0.13

[krga]

• Nvidia RTX 2070 SUPER (8 GB VRAM)
• Manjaro Linux (Arch Linux), Linux Kernel 6.6.65
• Build: b7552cf

./bin/llama-bench -m ../../llama-2-7b.Q4_0.gguf -ngl 100

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 2070 SUPER (NVIDIA) | uma: 0 | fp16: 1 | warp size: 32 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	1199.13 ± 7.70
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	64.64 ± 0.20

[srevn]

FreeBSD 14.2
ggml_vulkan: 0 = AMD Radeon RX 6600 XT (RADV NAVI23) (radv) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	428.92 ± 0.17
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	47.94 ± 0.70

build: 2e3a7d3c (4650)

macOS Ventura 13.7.2
ggml_vulkan: 0 = AMD Radeon RX 6900 XT (MoltenVK) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: none

model	size	params	backend	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan,BLAS	10	pp512	713.77 ± 0.62
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan,BLAS	10	tg128	83.54 ± 0.08

build: d2fe216 (4667)

[AngelaCooljx]

• Android 15
• Oppo Find x8 mtk9400(mt6991) with Mali-G925-Immortalis MC12

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Mali-G925-Immortalis MC12 (Mali-G925-Immortalis MC12) | uma: 1 | fp16: 1 | warp size: 16 | shared memory: 32768 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	8.25 ± 0.25
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	8.72 ± 0.10

build: aaa5505 (4673)

llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100	llama-run -m DeepSeek-R1-Distill-Qwen-7B-Q4_K_M.gguf -ngl 29

llama.cpp is built using cmake -B build -DGGML_VULKAN=ON; cmake --build build --config Release in termux. Package Dependencies: shaderc vulkan-tools vulkan-load-android cmake git..
It seems that the Vulkan backend is working correctly on the Mali-G925?

[AngelaCooljx]

model	size	params	backend	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CPU	8	pp512	37.88 ± 2.57
llama 7B Q4_0	3.56 GiB	6.74 B	CPU	8	tg128	6.47 ± 0.06

build: aaa5505 (4673)

Build without -DGGML_VULKAN=ON. CPU backend

[netrunnereve]

I see there might be something wrong cause the right pic show 131072 ctx warning.

That just means that the set context size is smaller than what the model was trained with, and it shouldn't affect performance. If you set the context size to 131072 on that phone you'd probably run out of memory 😏.

Build without GGML_VK_DISABLE_COOPMAT=1. CPU backend

Oh I meant that you could try the Vulkan backend with GGML_VK_DISABLE_COOPMAT=1 set, which disables coopmat for Vulkan and uses regular GPU shaders for the matrix multiplications. Your phone might be slowly emulating the coopmat instructions instead of actually running it on a matrix multiplier unit.

[FNsi]

I see there might be something wrong cause the right pic show 131072 ctx warning.

That just means that the set context size is smaller than what the model was trained with, and it shouldn't affect performance. If you set the context size to 131072 on that phone you'd probably run out of memory 😏.

i mean, llama 2 default ctx should be 4096, thus...

[AngelaCooljx]

I see there might be something wrong cause the right pic show 131072 ctx warning.

That just means that the set context size is smaller than what the model was trained with, and it shouldn't affect performance. If you set the context size to 131072 on that phone you'd probably run out of memory 😏.

Build without GGML_VK_DISABLE_COOPMAT=1. CPU backend

Oh I meant that you could try the Vulkan backend with GGML_VK_DISABLE_COOPMAT=1 set, which disables coopmat for Vulkan and uses regular GPU shaders for the matrix multiplications. Your phone might be slowly emulating the coopmat instructions instead of actually running it on a matrix multiplier unit.

Sorry for the confusion about this parameter. The score of running llama-bench with the environment variable GGML_VK_DISABLE_COOPMAT=1 is as follows.

➜  ~ GGML_VK_DISABLE_COOPMAT=1 llama.cpp/build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100
Could not open module param file '/sys/module/mali_kbase/parameters/large_page_conf'
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Mali-G925-Immortalis MC12 (Mali-G925-Immortalis MC12) | uma: 1 | fp16: 1 | warp size: 16 | shared memory: 32768 | matrix cores: none
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
[  680.128686] (mali-event-hand) BASE: basep_event_dump_error <unknown>
==>[WARN] Received a GROUP_ERROR_TIMEOUT error on group(0)
[  691.549718] (mali-event-hand) BASE: basep_event_dump_error <unknown>
==>[WARN] Received a GROUP_ERROR_TIMEOUT error on group(0)
[  702.844590] (mali-event-hand) BASE: basep_event_dump_error <unknown>
==>[WARN] Received a GROUP_ERROR_TIMEOUT error on group(0)
[  748.290818] (mali-event-hand) BASE: basep_event_dump_error <unknown>
==>[WARN] Received a GROUP_ERROR_TIMEOUT error on group(0)
[  759.584474] (mali-event-hand) BASE: basep_event_dump_error <unknown>
==>[WARN] Received a GROUP_ERROR_TIMEOUT error on group(0)
[  770.864396] (mali-event-hand) BASE: basep_event_dump_error <unknown>
==>[WARN] Received a GROUP_ERROR_TIMEOUT error on group(0)
| llama 7B Q4_0                  |   3.56 GiB |     6.74 B | Vulkan     | 100 |         pp512 |        38.36 ± 15.15 |
| llama 7B Q4_0                  |   3.56 GiB |     6.74 B | Vulkan     | 100 |         tg128 |          8.92 ± 0.06 |

build: b9ab0a4d (4687)

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	38.36 ± 15.15
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	8.92 ± 0.06

build: b9ab0a4 (4687)

The phone becomes extremely laggy at runtime, and it throws the warning:(mali-event-hand) BASE: basep_event_dump_error <unknown> ==> [WARN] Received a GROUP_ERROR_TIMEOUT error on group(0) after several minutes. 🤔

[netrunnereve]

Yeah those numbers make much more sense now.

The phone becomes extremely laggy at runtime, and it throws the warning:(mali-event-hand) BASE: basep_event_dump_error ==> [WARN] Received a GROUP_ERROR_TIMEOUT error on group(0) after several minutes. 🤔

I'm not sure about the errors but the laggy phone means that the shaders are working hard and using the GPU to its full capacity. I sometimes see this on my desktop computers as well, and I think it's really up to the driver to schedule tasks appropriately.

i mean, llama 2 default ctx should be 4096, thus...

Good point, maybe the model has incorrect metadata or something? Regardless this doesn't affect performance.

[jms13]

Ryzen 8840HS with Radeon 780M

ggml_vulkan: 0 = AMD Radeon Graphics (RADV GFX1103_R1) (radv) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	245.79 ± 2.97
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	20.10 ± 0.07

build: 19d3c82 (4677)

[Sidiox]

I have a pair of Amd Instinct MI50s that I'd like to add test results for, but I can't get Vulkan to recognize them to have llama.cpp run. I'm using the amdgpu installer script like this: amdgpu-install --vulkan=amdvlk --usecase=rocm,lrt,opencl,openclsdk,hip,hiplibsdk,dkms,mllib
I'm not sure what I'm missing. vulkaninfo is not even recognizing the driver. (ROCM does work fine)

[cb88]

@netrunnereve yeah both ROCm and Vulkan should be bandwidth bound on TG... PP is much much faster on rocm though.
@Said-Akbar VLLM has parallelism that lammacpp does not. So you are actually getting increased performance from having 2 cards. Speculative decoding assumes you have enough extra compute to do it... also try a Q4_0 quantized main and draft model instead of Q4_K and Q8_0. MI60 is fastest for the Q4_0 quantization by alot.

[0cc4m]

As you noticed Vulkan prompt processing is slower than ROCM though so pick your poison I guess. I've started working on matrix multiplication improvements but I don't think I can beat a proper manufacturer BLAS.

@netrunnereve I'm also looking into that, but it's quite complicated. Let me know if you need help or want to collaborate.

[netrunnereve]

I'm also looking into that, but it's quite complicated.

Right now I'm trying to make the dequantization more efficient, as it currently only does two values per loop with a bunch of setup in between. That's relatively easy to fix and should help both coopmat and the regular shaders.

At the end of the day though most of the time is taken up by the actual sgemm, and I don't want to spend a lot of time rewriting that when all new GPUs have built in fast matrix multipliers. Your shader also already does several layers of tiling and caching which should be on a high level similar to what the vendors are doing. For ROCBLAS and CUBLAS they probably write it in assembly and tune it for the architecture but the general idea should be the same.

I've thought of things like separately storing the scales and deltas (so I can do a single multiplication per tiled dot product) but that's a lot of work for an unknown improvement.

I'm definitely interested in hearing what you have in mind though.

[0cc4m]

That's good, I'm looking into improvements of the matmul part. Specifically, I'm trying to get int8 (quantized) matmul to work. I got a draft for q8_0 x q8_0 (including the quantization shader for src1) ready, but it's not yet working and needs to be debugged. Once I found the problem, I'll optimize it and check how it performs compared to the fp16/fp32 matmul. I don't expect much, since for good performance on Vega20+ it needs DP4A, @jeffbolznv is working on the GLSL integer vec dot extension. I also have to port this to coopmat afterwards. It'll take a while.

I'm happy to leave the dequantization part with you. Maybe we can also clean up the code to move the dequant stuff into functions, similar to the other shaders.

[netrunnereve]

Specifically, I'm trying to get int8 (quantized) matmul to work. I got a draft for q8_0 x q8_0 (including the quantization shader for src1) ready, but it's not yet working and needs to be debugged. Once I found the problem, I'll optimize it and check how it performs compared to the fp16/fp32 matmul. I don't expect much, since for good performance on Vega20+ it needs DP4A, @jeffbolznv is working on the GLSL integer vec dot extension.

Cool! Even if we don't have DP4A this shouldn't slow down matmul too much as the math takes longer than the dequantization (and we can explore integer FMAs and all that as well). For the actual mat vec side if the GPU is memory bound the quantized B might even help a bit.

Is the long term plan for this to completely replace the existing float based implementation, or are we going to stick with both? Things will probably get quite messy with two competing sets of code.

[wbruna]

Ryzen 3400G
AMD Radeon Vega 11 Graphics (RADV RAVEN) (radv) | uma: 1 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: none

model	size	params	backend	VRAM	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	16G	99	pp512	46.47 ± 5.15
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	16G	99	tg128	5.99 ± 0.71
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	0	99	pp512	35.65 ± 3.49
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	0	99	tg128	4.52 ± 0.21
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	2G	99	pp512	33.98 ± 4.51
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	2G	99	tg128	4.26 ± 0.41
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	256M	99	pp512	21.84 ± 0.33
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	256M	99	tg128	3.77 ± 0.15
llama 7B Q4_0	3.56 GiB	6.74 B	CPU?		0	pp512	6.69 ± 0.20
llama 7B Q4_0	3.56 GiB	6.74 B	CPU?		0	tg128	3.01 ± 0.12

build: 0893e01 (4682)

48G DDR4-3000, Debian 12, amdgpu driver from kernel 6.12.9, Mesa 24.2.8.

The '0' rows ran with 256M dedicated VRAM, but forcing host memory allocation (GGML_VK_PREFER_HOST_MEMORY=1). 'CPU?' rows were simply with '-ngl 0', but looks like llama-bench disables GPU prompt processing in that case?

[wbruna]

With some thermal fine-tuning:

model	size	params	backend	VRAM	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	16G	100	pp512	73.32 ± 4.53
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	16G	100	tg128	9.73 ± 0.70

[slp]

MacBook M3 Pro with 36 GB of RAM, running macOS 15.3 with MoltenVK patched with spec constants fixed (KhronosGroup/MoltenVK#2441).

$ ./bin/llama-bench -m ~/models/llama-2-7b.Q4_0.gguf -ngl 100
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Apple M3 Pro (MoltenVK) | uma: 1 | fp16: 1 | warp size: 32 | shared memory: 32768 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	259.11 ± 1.46
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	26.23 ± 0.04

build: b9ab0a4 (4687)

model	size	params	backend	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan,BLAS	6	pp512	263.70 ± 0.02
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan,BLAS	6	tg128	26.39 ± 0.14

build: b9ab0a4 (4687)

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Metal	100	pp512	371.72 ± 0.19
llama 7B Q4_0	3.56 GiB	6.74 B	Metal	100	tg128	31.54 ± 0.02

build: b9ab0a4 (4687)

model	size	params	backend	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Metal,BLAS	6	pp512	368.92 ± 4.92
llama 7B Q4_0	3.56 GiB	6.74 B	Metal,BLAS	6	tg128	31.52 ± 0.01

build: b9ab0a4 (4687)

[0cc4m]

BLAS doesn't make a difference for Vulkan or Metal, since the GPU backends take over its role anyways.

But the difference between Metal and Vulkan is surprisingly small, considering Vulkan cannot access the matrix accelerator hardware.

[ddpasa]

Intel Iris Plus Graphics G7 iGPU on Intel Core i7-1065G7 benchmarks are below. 16GB system ram, but vulkaninfo reports around 8GB of available vram.

CPU vs Vulkan below. Vulkan is much faster on pp512 (almost x2), but slightly slower on tg128. Not sure if I can do anything to speed up token generation.

./llama-bench -m models/llama-2-7b.Q4_0.gguf

model	size	params	backend	threads	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	CPU	4	pp512	14.11 ± 0.98
llama 7B Q4_0	3.56 GiB	6.74 B	CPU	4	tg128	5.43 ± 0.25

GGML_VK_PREFER_HOST_MEMORY=1 llama-bench -m models/llama-2-7b.Q4_0.gguf -ngl 0,10,20,30,33
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Iris(R) Plus Graphics (ICL GT2) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | warp size: 32 | shared memory: 65536 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	0	pp512	13.59 ± 0.17
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	0	tg128	5.90 ± 0.21
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	10	pp512	13.66 ± 1.02
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	10	tg128	4.75 ± 0.32
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	20	pp512	16.19 ± 0.20
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	20	tg128	4.65 ± 0.16
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	30	pp512	22.35 ± 0.06
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	30	tg128	4.39 ± 0.10
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	33	pp512	25.58 ± 0.00
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	33	tg128	4.25 ± 0.18

[ddpasa]

A followup revealed some interesting behaviour: I'm getting much better performance from Vulkan with GGML_VK_PREFER_HOST_MEMORY=1. Maybe @0cc4m knows why this might be so.

llama-bench --batch-size 128,256,512,768,1024 -m models/llama3_8b-deepseek-r1.gguf -ngl 33

model	size	params	backend	ngl	n_batch	test	t/s
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	Vulkan	33	128	pp512	17.64 ± 0.52
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	Vulkan	33	128	tg128	3.35 ± 0.06
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	Vulkan	33	256	pp512	16.30 ± 0.01
llama 8B Q4_K - Medium	4.58 GiB	8.03 B	Vulkan	33	256	tg128	3.18 ± 0.06

It can't run --batch-size 512 or higher because Vulkan can't allocate more memory (vk::DeviceLostError).

GGML_VK_PREFER_HOST_MEMORY=1 llama-bench --batch-size 128,256,512,768,1024 -m models/llama-2-7b.Q4_0.gguf -ngl 33

model	size	params	backend	ngl	n_batch	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	33	128	pp512	21.65 ± 1.93
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	33	128	tg128	4.38 ± 0.11
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	33	256	pp512	17.93 ± 1.07
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	33	256	tg128	4.12 ± 0.10
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	33	512	pp512	25.58 ± 0.00
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	33	512	tg128	4.32 ± 0.02
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	33	768	pp512	25.58 ± 0.00
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	33	768	tg128	4.31 ± 0.04
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	33	1024	pp512	25.58 ± 0.00
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	33	1024	tg128	4.33 ± 0.05

[netrunnereve]

Not sure if I can do anything to speed up token generation.

For your case the best choice is probably to run with -ngl 0 so that you use the CPU for inference and the GPU for prompt processing. The CPU backend (and CUDA and ROCM) use quantized activations, but Vulkan uses full fp16 or fp32 activations which slow things down when you're memory bound.

[ddpasa]

Not sure if I can do anything to speed up token generation.

For your case the best choice is probably to run with -ngl 0 so that you use the CPU for inference and the GPU for prompt processing. The CPU backend (and CUDA and ROCM) use quantized activations, but Vulkan uses full fp16 or fp32 activations which slow things down when you're memory bound.

Thanks! Wouldn't -ngl 0 put everything, including prompt processing, to the CPU? How do I use vulkan for prompt processing only?

[netrunnereve]

Actually if you build with Vulkan and use -ngl 0 the prompt processing is still done on the GPU. If you increase the ngl value then more layers end up on the GPU for inference purposes only. On a physical GPU that affects prompt processing speed as well but since your memory is shared between the GPU and CPU you won't lose much prompt processing speed when running with 0 layers.

[rmatif]

I did some testing on my Samsung Galaxt A34 (Mali-G68 MC4) here are the results :

build: 73e2ed3 (4735)

Vulkan :

~ $ llama.cpp/build/bin/llama-bench -m Llama-3.2-1B-Instruct-Q4_0.gguf -ngl 100
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Mali-G68 MC4 (Mali-G68 MC4) | uma: 1 | fp16: 1 | warp size: 16 | shared memory: 32768 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 1B Q4_0	729.75 MiB	1.24 B	Vulkan	100	pp512	4.74 ± 0.06
llama 1B Q4_0	729.75 MiB	1.24 B	Vulkan	100	tg128	6.75± 0.02

Same results with GGML_VK_DISABLE_COOPMAT=1

~ $ llama.cpp/build/bin/llama-bench -m Llama-3.2-1B-Instruct-Q4_0.gguf -ngl 0
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Mali-G68 MC4 (Mali-G68 MC4) | uma: 1 | fp16: 1 | warp size: 16 | shared memory: 32768 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 1B Q4_0	729.75 MiB	1.24 B	Vulkan	0	pp512	33.14 ± 0.75
llama 1B Q4_0	729.75 MiB	1.24 B	Vulkan	0	tg128	4.48 ± 0.03

CPU :

llama.cpp/build/bin/llama-bench -m Llama-3.2-1B-Instruct-Q4_0.gguf

model	size	params	backend	threads	test	t/s
llama 1B Q4_0	729.75 MiB	1.24 B	CPU	8	pp512	73.90 ± 5.65
llama 1B Q4_0	729.75 MiB	1.24 B	CPU	8	tg128	14.23 ± 0.33

Besides the fact that the Vulkan implementation for mobile GPUs is not there yet which I was aware of. But I'm really surprised by the big difference between a CPU build and a GPU with ngl 0. Theoretically, it should be quite the same, no?

[jeffbolznv]

Can you try disabling the large and medium mul mat shaders (set mul_mat_l/m to false) and see if it helps with prompt processing? The sizes are tuned for desktop devices and may be too large for mobile devices.

[rmatif]

    device->mul_mat_l[i] = false;
    device->mul_mat_m[i] = false;
    device->mul_mat_s[i] = true;
    device->mul_mat_id_l[i] = false;
    device->mul_mat_id_m[i] = false;
    device->mul_mat_id_s[i] = true;
    break;

model	size	params	backend	ngl	test	t/s
llama 1B Q4_0	729.75 MiB	1.24 B	Vulkan	100	pp512	3.59 ± 0.00
llama 1B Q4_0	729.75 MiB	1.24 B	Vulkan	100	tg128	7.14 ± 0.02

    device->mul_mat_l[i] = false;
    device->mul_mat_m[i] = true;
    device->mul_mat_s[i] = true;
    device->mul_mat_id_l[i] = false;
    device->mul_mat_id_m[i] = true;
    device->mul_mat_id_s[i] = true;
    break;

model	size	params	backend	ngl	test	t/s
llama 1B Q4_0	729.75 MiB	1.24 B	Vulkan	100	pp512	8.13 ± 0.02
llama 1B Q4_0	729.75 MiB	1.24 B	Vulkan	100	tg128	7.14 ± 0.02

    device->mul_mat_l[i] = false;
    device->mul_mat_m[i] = true;
    device->mul_mat_s[i] = true;
    device->mul_mat_id_l[i] = true;
    device->mul_mat_id_m[i] = true;
    device->mul_mat_id_s[i] = true;
    break;

model	size	params	backend	ngl	test	t/s
llama 1B Q4_0	729.75 MiB	1.24 B	Vulkan	100	pp512	8.03 ± 0.01
llama 1B Q4_0	729.75 MiB	1.24 B	Vulkan	100	tg128	7.14 ± 0.02

    device->mul_mat_l[i] = false;
    device->mul_mat_m[i] = false;
    device->mul_mat_s[i] = true;
    device->mul_mat_id_l[i] = true;
    device->mul_mat_id_m[i] = true;
    device->mul_mat_id_s[i] = true;
    break;

model	size	params	backend	ngl	test	t/s
llama 1B Q4_0	729.75 MiB	1.24 B	Vulkan	100	pp512	3.58 ± 0.00
llama 1B Q4_0	729.75 MiB	1.24 B	Vulkan	100	tg128	7.14 ± 0.01

build: 63ac128 (4738)

Disabling both mul_mat_l and mul_mat_m makes it worse. However, keeping mul_mat_m enabled and only disabling mul_mat_l does help a bit, though the performance is still far behind the CPU-only. I don't know much about Vulkan, but could we imagine something like CLBlast’s tuner, where users run small benchmarks, report their results, and then use those results to write custom shaders and tune performance accordingly?

[AngelaCooljx]

AMD Radeon RX 7800 XT on linux

llama.cpp/build/bin/llama-bench -m llama-2-7b.Q4_0.gguf -ngl 100 
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7800 XT (RADV GFX1101) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	883.20 ± 0.80
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	103.98 ± 0.23

build: ee02ad0 (4749)

[0cc4m]

Nice result. Your GPU also supports coopmat matrix core acceleration, but it's not active in your test. I guess that you're using a mesa driver that's too old or the Vulkan API and GLSL you used to compile the program predates the VK_KHR_cooperative_matrix extension.

If you figure that out, your prompt processing should more or less double in speed.

[AngelaCooljx]

Right! These're ubuntu jammy packages. Now I install the lastest mesa drivers from ppa:kisak/kisak-mesa. The score is higher.

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7800 XT (RADV NAVI32) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	pp512	1260.54 ± 10.51
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	100	tg128	107.53 ± 0.07

build: ee02ad0 (4749)

[0cc4m]

I guess I was a little optimistic with double, but it's still a good bit faster. Neat!

[ocean1ee1]

Integrated GPU of Ultra 7 Processor 165U, installed the intel gpu driver

./llama.cpp/build-20250217-Vulkan/build/bin$ ./llama-bench -m ./models/llama-2-7b.Q4_0.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Graphics (MTL) (Intel open-source Mesa driver) | uma: 1 | fp16: 1 | warp size: 32 | shared memory: 65536 | matrix cores: none

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	99	pp512	25.56 ± 0.00
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	99	tg128	0.77 ± 0.00

build: b9ab0a4 (4687)

./llama.cpp/build-20250211-sycl/build/bin$ ./llama-bench -m ./models/llama-2-7b.Q4_0.gguf
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory
get_memory_info: [warning] ext_intel_free_memory is not supported (export/set ZES_ENABLE_SYSMAN=1 to support), use total memory as free memory

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	SYCL	99	pp512	82.19 ± 0.10
llama 7B Q4_0	3.56 GiB	6.74 B	SYCL	99	tg128	5.60 ± 0.11

build: b9ab0a4 (4687)

---

AMD external GPU(7600M XT) connected via a Thunderbolt 4.0 USB-C

./llama.cpp/build-20250217-Vulkan/build/bin$ ./llama-bench -m ./models/llama-2-7b.Q4_0.gguf
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon™ RX 7600M XT (RADV NAVI33) (radv) | uma: 0 | fp16: 1 | warp size: 64 | shared memory: 65536 | matrix cores: KHR_coopmat

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	99	pp512	459.39 ± 2.34
llama 7B Q4_0	3.56 GiB	6.74 B	Vulkan	99	tg128	45.28 ± 0.10

./llama.cpp/build-20250213-hip/build/bin$ ./llama-bench -m ./models/llama-2-7b.Q4_0.gguf
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon™ RX 7600M XT, gfx1102 (0x1102), VMM: no, Wave Size: 32

model	size	params	backend	ngl	test	t/s
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	99	pp512	1068.74 ± 1.46
llama 7B Q4_0	3.56 GiB	6.74 B	ROCm	99	tg128	33.10 ± 0.01

build: b9ab0a4 (4687)

---

For me, the Vulkan version is not my first choice, no matter use external gpu or not.