Vulkan: VK_KHR_cooperative_matrix support to speed up prompt processing #10597

0cc4m · 2024-11-30T09:52:13Z

This PR implements basic tensor core/matrix core/xmx engine support using the vendor-neutral VK_KHR_cooperative_matrix Vulkan extension for the matrix multiplication shader. I need to spend some further time optimizing the code, but I would like to get some feedback already.

This is different from #10206 in that it should work on some non-Nvidia GPUs too.

It works on:

Nvidia RTX 2000 and newer

It should work on:

AMD RX 7000 series and newer
Intel ARC

HOWEVER:
It seems AMD's and Intel's drivers like to pretend they support this extension, despite either not having implemented proper hardware support (like my Intel A770 on Linux mesa), or not even having the necessary hardware at all (AMD's proprietary driver for older generations.. why?). So this probably also needs a whitelist feature to avoid regressions on hardware where support isn't actually there.

I have read the contributing guidelines
Self-reported review complexity:
- Low
- Medium
- High

0cc4m · 2024-11-30T10:00:02Z

Some preliminary benchmarks, but there's probably some more optimization that can be done.

RTX 3090

model	size	params	backend	ngl	test	t/s master	t/s PR	t/s CUDA
llama 7B Q4_0	3.83 GiB	7.24 B	Vulkan	99	pp512	825.42 ± 1.14	1933.18 ± 9.55	5059.05 ± 11.44
llama 8B Q4_K - Small	4.36 GiB	8.03 B	Vulkan	99	pp512	680.42 ± 0.73	1381.26 ± 7.01	4692.44 ± 4.78

Intel just for fun:

Intel A770 (Linux Mesa)

model	size	params	backend	ngl	test	t/s master	t/s PR
llama 8B Q4_K - Small	4.36 GiB	8.03 B	Vulkan	99	pp512	150.46 ± 0.05	25.48 ± 0.12

qnixsynapse · 2024-11-30T14:31:59Z

It seems AMD's and Intel's drivers like to pretend they support this extension, despite either not having implemented proper hardware support (like my Intel A770 on Linux mesa),

This should be reported to Freedesktop Gitlab IMO.

Also, I do test the Vulkan backend on my Intel Arc, I found out that Vulkan backend only uses the shader engines instead of compute engines. The later being faster. Is this intentional?

0cc4m · 2024-11-30T16:58:40Z

It seems AMD's and Intel's drivers like to pretend they support this extension, despite either not having implemented proper hardware support (like my Intel A770 on Linux mesa),

This should be reported to Freedesktop Gitlab IMO.

They know.

Also, I do test the Vulkan backend on my Intel Arc, I found out that Vulkan backend only uses the shader engines instead of compute engines. The later being faster. Is this intentional?

Assuming you mean shader cores and XMX engines, that's exactly what this PR is about. By default Vulkan can run compute shaders only on regular shader cores, it needs extensions like VK_KHR_cooperative_matrix to use the specialized hardware of tensor cores, AMD matrix cores or Intel XMX engines.

jeffbolznv

I didn't review all the addressing math in detail.

ggml/src/ggml-vulkan/ggml-vulkan.cpp

ggml/src/ggml-vulkan/vulkan-shaders/mul_mm.comp

jeffbolznv · 2024-11-30T17:52:43Z

Very cool to see this! I tested the changes on RTX 4070, and test-backend-ops is passing, but I don't see much perf gain as-is (like +/- 20% testing a Q4_K model).

I see a couple perf opportunities. First, the branches around the loads for quants like Q4_K can cause the shader to stall inside the if/else waiting for the data. I had to "flatten" the control flow (e.g. https://github.com/ggerganov/llama.cpp/pull/10206/files#diff-508f8330faff804d4067ccb566c1d3eb27a7021a03df4f914c00c977d114514fR181) for the coopmat2 dequant functions, I suspect something similar is needed here.

Second, add [[unroll]] around the store loops. The dynamic indexing of the sums array is problematic and goes away with unrolling.

I tried these locally/hackily and then I get a much better speedup (like 2.5x).

0cc4m · 2024-11-30T19:09:37Z

I see a couple perf opportunities. First, the branches around the loads for quants like Q4_K can cause the shader to stall inside the if/else waiting for the data. I had to "flatten" the control flow (e.g. https://github.com/ggerganov/llama.cpp/pull/10206/files#diff-508f8330faff804d4067ccb566c1d3eb27a7021a03df4f914c00c977d114514fR181) for the coopmat2 dequant functions, I suspect something similar is needed here.

Interesting. So that the GPU has to wait for the load from global memory, but this is worsened by being in a branch?

Second, add [[unroll]] around the store loops. The dynamic indexing of the sums array is problematic and goes away with unrolling.

I tried these locally/hackily and then I get a much better speedup (like 2.5x).

I implemented those changes. I hope you don't mind that I used your loading code. Maybe in the future we can just use your dequant_funcs file for mul_mm too.

I don't see a drastic improvement, but it's 7-8% faster for me on 3090 on Q4_K_S, so that's good. Let me know if you see anything else.

I'm also curious if you learned these GLSL optimization techniques from experience, or if you can recommend some resources I can use to learn. This is my first Vulkan project and the first time going deep into GPU programming, and it's been difficult to find information I can apply.

jeffbolznv · 2024-11-30T20:47:13Z

Interesting. So that the GPU has to wait for the load from global memory, but this is worsened by being in a branch?

Yeah, it's challenging for a compiler to track memory accesses across basic blocks, so doing loads inside a branch can cause the value to be waited on inside the branch. And that prevents doing other useful work while the load is pending.

I'm also curious if you learned these GLSL optimization techniques from experience, or if you can recommend some resources I can use to learn. This is my first Vulkan project and the first time going deep into GPU programming, and it's been difficult to find information I can apply.

I think most performance advice is likely to come from vendor specific documentation/tooling, even though a lot of the information and optimization techniques can apply across different hardware. I'd suggest looking at nsight documentation and cuda forums as they'll likely have a lot more low-level information about NVIDIA hardware than you would find in Vulkan-centric documentation. For example, https://developer.nvidia.com/blog/identifying-shader-limiters-with-the-shader-profiler-in-nvidia-nsight-graphics/ talks about using nsight and also touches on this issue of dynamically indexed arrays.

jeffbolznv · 2024-11-30T20:58:59Z

I got a device lost error while trying to collect perf results, running Meta-Llama-3-8B-Instruct-Q4_K_S.gguf with pp512. But these two other models run:

master
| llama 7B Q4_0                  |   3.56 GiB |     6.74 B | Vulkan     | 1000 |         pp512 |        831.12  4.03 |
| phi3 3B Q4_K - Medium          |   2.23 GiB |     3.82 B | Vulkan     | 1000 |         pp512 |       1135.54  1.70 |

PR
| llama 7B Q4_0                  |   3.56 GiB |     6.74 B | Vulkan     | 1000 |         pp512 |       1904.89  8.26 |
| phi3 3B Q4_K - Medium          |   2.23 GiB |     3.82 B | Vulkan     | 1000 |         pp512 |      2213.51  25.68 |

characharm · 2024-12-01T08:41:40Z

Windows OS

this pr
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | warp size: 32 | matrix cores: 1

model	size	params	backend	ngl	test	t/s
ggml_vulkan: Compiling shaders..............................Done!
qwen2 14B Q5_K - Medium	9.78 GiB	14.77 B	Vulkan	99	pp512	20.84 ± 0.03
qwen2 14B Q5_K - Medium	9.78 GiB	14.77 B	Vulkan	99	tg128	16.01 ± 0.06

build: 5660976 (4231)

master
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(TM) A770 Graphics (Intel Corporation) | uma: 0 | fp16: 1 | warp size: 32

model	size	params	backend	ngl	test	t/s
ggml_vulkan: Compiling shaders..............................Done!
qwen2 ?B Q5_K - Medium	9.78 GiB	14.77 B	Vulkan,RPC	99	pp512	67.85 ± 0.52
qwen2 ?B Q5_K - Medium	9.78 GiB	14.77 B	Vulkan,RPC	99	tg128	15.52 ± 0.06

build: 25669aa (4179)

easyfab · 2024-12-01T20:53:06Z

Sadly, I have the same regression with A770.

By the way wich is the best mesa version on linux ( docker )?
Because I tried default ( ubuntu 24.04 ) and ppa:kisak/kisak-mesa or oibaf/graphics-drivers.

I was suprised that the default version is faster than newers versions 25 t/s vs 20t/s ? Is this expected ?

Default Mesa 24.0.9-0ubuntu0.2 (LLVM 17.0.6)

MESA-INTEL: warning: cannot initialize blitter engine
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(tm) A770 Graphics (DG2) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | warp size: 32
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
ggml_vulkan: Compiling shaders..............................Done!
| llama 8B Q4_K - Medium         |   4.58 GiB |     8.03 B | Vulkan     |  99 |         pp512 |        154.20 ± 0.02 |
| llama 8B Q4_K - Medium         |   4.58 GiB |     8.03 B | Vulkan     |  99 |         tg128 |         24.99 ± 0.02 |

Mesa 24.3.0 - kisak-mesa PPA (LLVM 17.0.6)

ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = Intel(R) Arc(tm) A770 Graphics (DG2) (Intel open-source Mesa driver) | uma: 0 | fp16: 1 | warp size: 32
| model                          |       size |     params | backend    | ngl |          test |                  t/s |
| ------------------------------ | ---------: | ---------: | ---------- | --: | ------------: | -------------------: |
ggml_vulkan: Compiling shaders..............................Done!
| llama 8B Q4_K - Medium         |   4.58 GiB |     8.03 B | Vulkan     |  99 |         pp512 |        151.10 ± 0.05 |
| llama 8B Q4_K - Medium         |   4.58 GiB |     8.03 B | Vulkan     |  99 |         tg128 |         19.96 ± 0.01 |

0cc4m · 2024-12-01T21:01:42Z

@jeffbolznv I reworked your shared memory shader selection code because I started getting segfaults on Intel that might be related to the growing number of shaders. Now only those that actually get used get compiled, and since Intel only uses the smallest tile size, this reduces the number greatly. This did fix the segfault.

The logic is very different now, maybe you can look over it and check it still makes sense.

jeffbolznv · 2024-12-02T01:45:13Z

ggml/src/ggml-vulkan/ggml-vulkan.cpp

        ggml_vk_create_queue(device, device->compute_queue, compute_queue_family_index, 0, { vk::PipelineStageFlagBits::eComputeShader | vk::PipelineStageFlagBits::eTransfer }, false);

        // Shaders
+        // Disable matmul tile sizes early if not supported


I guess this is replacing the old vendor "guess" functions, but what does "if not supported" mean? Is this still needed or does the shared memory calculation replace it?

"Not performant" would be more correct here. The large tiles only run well on Nvidia. The medium tiles only run well on AMD and Nvidia. Intel can only run the small tiles at any decent speed. I think this is related to registers spilling to global memory or overloading caches.

jeffbolznv · 2024-12-02T01:46:27Z

ggml/src/ggml-vulkan/ggml-vulkan.cpp

        return aligned ? mmp->a_s : mmp->s;
    }
-    if (m <= 64 || n <= 64) {
+    if ((ctx->device->mul_mat_m && (m <= 64 || n <= 64 || ctx->device->coopmat_support)) || !ctx->device->mul_mat_l) {


I don't think coopmat_support should force medium, at least not on NVIDIA.

When it first started working I ran the three tile sizes next to each other and the large tile was very slow. I'll test it again.

It's not as slow as I remember, but performance drops significantly on RTX 3090 if I allow the large tile.

I narrowed it down a little further: Performance drops significantly with the large tiles when working with quants. With float16 or float32 the large tiles work well with coopmat.

Looks like adding the remaining [[unroll]]s fixes the perf for large tiles.

You... are right. So basically just unroll every loop you can get your hands on?

I guess it's dynamic access to shared memory this time. Why did that affect the large tile so significantly?

I think it's still the sums array, but maybe it was getting unrolled automatically for the medium size but not for the large. The compilers aren't as aggressive about this as they probably should be, so anytime you have a dynamically indexed array where unrolling would make the indices constant, it's usually worth doing (and that's why #pragma unroll shows up literally hundreds of times in the cuda kernels).

jeffbolznv · 2024-12-02T01:51:07Z

For the Intel perf issues, my best guesses are to add more [[unroll]] on the coopmatmuladd loops, and maybe use subgroup size control to force 32 invocations per subgroup.

qnixsynapse · 2024-12-02T05:01:28Z

It seems AMD's and Intel's drivers like to pretend they support this extension, despite either not having implemented proper hardware support (like my Intel A770 on Linux mesa),

This should be reported to Freedesktop Gitlab IMO.

They know.

Also, I do test the Vulkan backend on my Intel Arc, I found out that Vulkan backend only uses the shader engines instead of compute engines. The later being faster. Is this intentional?

Assuming you mean shader cores and XMX engines, that's exactly what this PR is about. By default Vulkan can run compute shaders only on regular shader cores, it needs extensions like VK_KHR_cooperative_matrix to use the specialized hardware of tensor cores, AMD matrix cores or Intel XMX engines.

I see.

Well, backend failing on lot of tests:

 MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=256,bs=[2,3],nr=[1,1],per=[0,2,1,3]): [MUL_MAT] NaN at index 25 (Vulkan0=-nan CPU=-0.489127) FAIL
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=256,bs=[2,3],nr=[1,1],per=[0,1,3,2]): [MUL_MAT] NaN at index 98 (Vulkan0=nan CPU=-2.006299) FAIL
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=8,k=256,bs=[2,3],nr=[1,1],per=[0,3,2,1]): [MUL_MAT] NaN at index 43 (Vulkan0=nan CPU=-3.059767) FAIL
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=16,k=256,bs=[2,3],nr=[1,1],per=[0,2,1,3]): [MUL_MAT] NaN at index 17 (Vulkan0=-nan CPU=15.675148) FAIL
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=16,k=256,bs=[2,3],nr=[1,1],per=[0,1,3,2]): [MUL_MAT] NaN at index 42 (Vulkan0=nan CPU=-5.011763) FAIL
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=16,k=256,bs=[2,3],nr=[1,1],per=[0,3,2,1]): [MUL_MAT] NaN at index 27 (Vulkan0=-nan CPU=1.719916) FAIL

MUL_MAT(type_a=f16,type_b=f32,m=16,n=16,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 19 (Vulkan0=nan CPU=-2.400129) FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=16,k=256,bs=[10,1],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 10 (Vulkan0=-nan CPU=-0.972326) FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=16,k=256,bs=[10,1],nr=[2,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 353 (Vulkan0=-nan CPU=-3.618169) FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=16,k=256,bs=[10,10],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 58 (Vulkan0=-nan CPU=2.903507) FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=16,k=256,bs=[10,10],nr=[2,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 73 (Vulkan0=-nan CPU=-0.198813) FAIL

  MUL_MAT(type_a=f16,type_b=f32,m=16,n=8,k=256,bs=[2,3],nr=[1,1],per=[0,2,1,3]): [MUL_MAT] NaN at index 9 (Vulkan0=nan CPU=10.375449) FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=8,k=256,bs=[2,3],nr=[1,1],per=[0,1,3,2]): [MUL_MAT] NaN at index 107 (Vulkan0=nan CPU=-6.503006) FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=8,k=256,bs=[2,3],nr=[1,1],per=[0,3,2,1]): [MUL_MAT] NaN at index 203 (Vulkan0=-nan CPU=-2.009472) FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=16,k=256,bs=[2,3],nr=[1,1],per=[0,2,1,3]): [MUL_MAT] NaN at index 0 (Vulkan0=nan CPU=3.655150) FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=16,k=256,bs=[2,3],nr=[1,1],per=[0,1,3,2]): [MUL_MAT] NaN at index 81 (Vulkan0=-nan CPU=-0.457537) FAIL
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=16,k=256,bs=[2,3],nr=[1,1],per=[0,3,2,1]): [MUL_MAT] NaN at index 128 (Vulkan0=-nan CPU=7.094562) FAIL

 MUL_MAT(type_a=f16,type_b=f16,m=16,n=16,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NMSE = 1.000019055 > 0.000500000 FAIL
  MUL_MAT(type_a=f16,type_b=f16,m=16,n=16,k=256,bs=[10,1],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 161 (Vulkan0=-nan CPU=-0.011188) FAIL
  MUL_MAT(type_a=f16,type_b=f16,m=16,n=16,k=256,bs=[10,1],nr=[2,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 51 (Vulkan0=nan CPU=-4.520026) FAIL
  MUL_MAT(type_a=f16,type_b=f16,m=16,n=16,k=256,bs=[10,10],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 8 (Vulkan0=-nan CPU=9.592922) FAIL
  MUL_MAT(type_a=f16,type_b=f16,m=16,n=16,k=256,bs=[10,10],nr=[2,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 106 (Vulkan0=nan CPU=-2.529743) FAIL

 MUL_MAT(type_a=f16,type_b=f16,m=16,n=8,k=256,bs=[2,3],nr=[1,1],per=[0,2,1,3]): [MUL_MAT] NaN at index 224 (Vulkan0=nan CPU=-4.796510) FAIL
  MUL_MAT(type_a=f16,type_b=f16,m=16,n=8,k=256,bs=[2,3],nr=[1,1],per=[0,1,3,2]): [MUL_MAT] NaN at index 57 (Vulkan0=nan CPU=-0.507002) FAIL
  MUL_MAT(type_a=f16,type_b=f16,m=16,n=8,k=256,bs=[2,3],nr=[1,1],per=[0,3,2,1]): [MUL_MAT] NaN at index 90 (Vulkan0=-nan CPU=-3.574138) FAIL
  MUL_MAT(type_a=f16,type_b=f16,m=16,n=16,k=256,bs=[2,3],nr=[1,1],per=[0,2,1,3]): [MUL_MAT] NaN at index 17 (Vulkan0=nan CPU=3.233534) FAIL
  MUL_MAT(type_a=f16,type_b=f16,m=16,n=16,k=256,bs=[2,3],nr=[1,1],per=[0,1,3,2]): [MUL_MAT] NaN at index 24 (Vulkan0=-nan CPU=2.170926) FAIL
  MUL_MAT(type_a=f16,type_b=f16,m=16,n=16,k=256,bs=[2,3],nr=[1,1],per=[0,3,2,1]): [MUL_MAT] NaN at index 305 (Vulkan0=-nan CPU=3.681449) FAIL

 MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=16,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 10 (Vulkan0=nan CPU=12.519882) FAIL
  MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=16,k=256,bs=[10,1],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 2315 (Vulkan0=-nan CPU=11.811843) FAIL
  MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=16,k=256,bs=[10,1],nr=[2,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 4867 (Vulkan0=-nan CPU=-1.555186) FAIL
  MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=16,k=256,bs=[10,10],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 16179 (Vulkan0=-nan CPU=-1.033244) FAIL
  MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=16,k=256,bs=[10,10],nr=[2,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 3040 (Vulkan0=-nan CPU=2.866659) FAIL

 MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=16,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 67 (Vulkan0=nan CPU=7.281193) FAIL
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=16,k=256,bs=[10,1],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 825 (Vulkan0=nan CPU=-6.018594) FAIL
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=16,k=256,bs=[10,1],nr=[2,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 1920 (Vulkan0=-nan CPU=6.858445) FAIL
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=16,k=256,bs=[10,10],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 9058 (Vulkan0=nan CPU=14.321257) FAIL
  MUL_MAT(type_a=q4_0,type_b=f32,m=16,n=16,k=256,bs=[10,10],nr=[2,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 792 (Vulkan0=-nan CPU=-3.903759) FAIL

  MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=16,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 0 (Vulkan0=-nan CPU=6.785239) FAIL
  MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=16,k=256,bs=[10,1],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 2433 (Vulkan0=nan CPU=-2.455450) FAIL
  MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=16,k=256,bs=[10,1],nr=[2,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 3928 (Vulkan0=nan CPU=-0.206156) FAIL
  MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=16,k=256,bs=[10,10],nr=[1,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 2928 (Vulkan0=-nan CPU=-1.648083) FAIL
  MUL_MAT(type_a=q4_1,type_b=f32,m=16,n=16,k=256,bs=[10,10],nr=[2,1],per=[0,1,2,3]): [MUL_MAT] NaN at index 2418 (Vulkan0=nan CPU=-2.766699) FAIL

 MUL_MAT(type_a=q4_K,type_b=f32,m=16,n=16,k=256,bs=[1,1],nr=[1,1],per=[0,1,2,3]): terminate called after throwing an instance of 'vk::DeviceLostError'
  what():  vk::Device::waitForFences: ErrorDeviceLost
Aborted (core dumped)

Still uses the 3D shader engines for computation with this PR.
GPU: Intel Arc A750

0cc4m · 2024-12-02T06:30:14Z

For the Intel perf issues, my best guesses are to add more [[unroll]] on the coopmatmuladd loops, and maybe use subgroup size control to force 32 invocations per subgroup.

At least on Linux, Mesa's ANV Intel driver currently just emulates the coopmat extension on the shader cores in an inefficient way, until proper support is added. I don't know what the WIndows driver is doing, but I assume something similar.

Before this PR is ready to merge, I'll have to implement a driver/device whitelist to make sure the coopmat codepath only happens (by default) on devices that actually support it properly.

Well, backend failing on lot of tests:
[...]
Still uses the 3D shader engines for computation with this PR. GPU: Intel Arc A750

Thank you for reporting the failing tests on Windows.

It will always use the shader engines as well, the XMX engines are specialized and can only do some specific operations. But it seems the Intel Windows driver also hasn't implemented coopmat support properly. I assume if you can see whether an application uses shader or compute engines, and this PR still doesn't use the compute engines, then Intel on Windows is also just emulating support.

qnixsynapse · 2024-12-02T06:50:11Z

It will always use the shader engines as well, the XMX engines are specialized and can only do some specific operations. But it seems the Intel Windows driver also hasn't implemented coopmat support properly. I assume if you can see whether an application uses shader or compute engines, and this PR still doesn't use the compute engines, then Intel on Windows is also just emulating support.

I'm sorry for not mentioning this earlier. It's mesa ANV driver on Arch Linux.

0cc4m · 2024-12-02T07:10:48Z

I'm sorry for not mentioning this earlier. It's mesa ANV driver on Arch Linux.

Oh, interesting. So you're monitoring through intel_gpu_top. I assume it just shows this backend under Render/3D because it's Vulkan, not OpenCL/LevelZero. There's nothing I can do about that.

However, on my A770 under Linux Mesa 24.0.9-0ubuntu0.2, the tests do pass. Performance is low, but the results are correct. Which version are you using?

qnixsynapse · 2024-12-02T07:18:31Z

I am using Mesa version: 24.2.7, Latest mainline Linux kernel. Everything is up to date.

I assume it just shows this backend under Render/3D because it's Vulkan, not OpenCL/LevelZero. There's nothing I can do about that.

Yeah. I think I need to have a talk upstream. Reason why I am testing this PR.

characharm · 2024-12-02T07:57:09Z

Vulkan has been loading Compute Units for quite some time now, but unlike SYCL, Windows Task Manager also reports some load on 3D

jeffbolznv · 2024-12-03T21:44:44Z

I got a device lost error while trying to collect perf results, running Meta-Llama-3-8B-Instruct-Q4_K_S.gguf with pp512.

I no longer see this with the latest code. test-backend-ops passes, I touch tested a few models and perf is quite good and they seem to be working correctly. So things seem to be in pretty good shape.

What else do you think remains to make this no longer "Draft"?

0cc4m · 2024-12-03T21:53:09Z

I got a device lost error while trying to collect perf results, running Meta-Llama-3-8B-Instruct-Q4_K_S.gguf with pp512.

I no longer see this with the latest code. test-backend-ops passes, I touch tested a few models and perf is quite good and they seem to be working correctly. So things seem to be in pretty good shape.

What else do you think remains to make this no longer "Draft"?

In the current state I'm pretty sure this PR will tank the performance of most AMD GPUs on Windows, since that driver erroneously reports support for coopmats on every GPU. I'm gonna have to figure out a way to tell which devices have proper support (RX 7000 series) and which are just pretending.

0cc4m · 2024-12-04T09:59:32Z

It's even worse than I expected. I tested it with an RX 6800 XT on Windows and it advertises support, but outright crashes when compiling a coopmat shader. On RX 7900 XTX it runs, but gives incorrect results, so I'll disable coopmats for AMD on Windows for now. Linux Mesa should work, I hope someone can test it.

0cc4m · 2024-12-04T15:15:21Z

It was now tested on Linux Mesa with an AMD RX 7900 XTX and works fine. So the result is that the feature will be available for Nvidia (Turing+) on Windows and Linux and for AMD RX 7000 series on Linux. 7000 series on Windows and Intel ARC on Linux and Windows will need proper driver-side implementations before they can utilize coopmats.

jeffbolznv · 2024-12-05T04:52:21Z

I ran out of time today, I'll review this tomorrow.

…operative_matrix support

0cc4m · 2024-12-05T21:01:09Z

I think this is basically ready. I still want to check if converting the B matrices to fp16 would improve performance of the coopmat shader, but that's not urgent.

@jeffbolznv Please check if I messed up any coopmat2 code while merging. I checked that it still works, at least.

ggml/src/ggml-vulkan/ggml-vulkan.cpp

ggml/src/ggml-vulkan/vulkan-shaders/mul_mm.comp

jeffbolznv · 2024-12-05T22:17:02Z

I did some testing, and everything looks good with #10597 (comment) fixed. I'll +1 it now, please go ahead and merge after fixing that.

0cc4m · 2024-12-06T12:25:33Z

I forgot to apply the coopmat2 shader selection logic to the mul_mat_id selection function. I'll do that soon, afterwards I'll merge.

mmerecki · 2024-12-06T15:04:30Z

I run the backend test (test-backend-ops.exe -b Vulkan0) on ARC A750, on Windows, using driver ver. 32.0.101.6314.
All tests passed (test-backend-ops.log).
Is test-backend-ops test the one that was failing on Intel Windows?

0cc4m · 2024-12-06T15:33:21Z

I run the backend test (test-backend-ops.exe -b Vulkan0) on ARC A750, on Windows, using driver ver. 32.0.101.6314. All tests passed (test-backend-ops.log). Is test-backend-ops test the one that was failing on Intel Windows?

The feature was disabled on Intel for now, so the tests pass and performance is the same as on master.

mmerecki · 2024-12-06T15:55:28Z

The feature was disabled on Intel for now, so the tests pass and performance is the same as on master.

Yes, I noticed that and enabled it back again for the test. I might have not used the correct binary though.
I will check again to see if these issues are still present with newer Windows drivers. Thank you for this PR.

airlied · 2024-12-06T23:02:51Z

So I think in theory the A770 should have the hw instructions hooked up in the mesa driver, they emulate it on hw where it doesn't and Intel claimed it might be more optimal than having manually doing it, but I doubt this is always true.

https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/intel/compiler/brw_compiler.c?ref_type=heads#L104 is the dpas instruction lowering for hw that doesn't have it.

0cc4m · 2024-12-07T06:32:53Z

So I think in theory the A770 should have the hw instructions hooked up in the mesa driver, they emulate it on hw where it doesn't and Intel claimed it might be more optimal than having manually doing it, but I doubt this is always true.

https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/intel/compiler/brw_compiler.c?ref_type=heads#L104 is the dpas instruction lowering for hw that doesn't have it.

Here's the issue tracking proper VK_KHR_cooperative_matrix support: https://gitlab.freedesktop.org/mesa/mesa/-/issues/9250

oscarbg · 2024-12-07T06:46:21Z

Hi,
sorry for asking here.. but once this is merged I plan to run benches on NV with Vulkan dev driver supporting coop matrix 2 and want to select running coop matrix 1, coop matrix 2 and without coop matrix paths, any enviroment variable or command line argument to forcely select each of these 3 "backends"?
thanks..

0cc4m · 2024-12-07T07:38:14Z

Hi, sorry for asking here.. but once this is merged I plan to run benches on NV with Vulkan dev driver supporting coop matrix 2 and want to select running coop matrix 1, coop matrix 2 and without coop matrix paths, any enviroment variable or command line argument to forcely select each of these 3 "backends"? thanks..

Yes, we have provided environment variables to disable coopmat and coopmat2 support if needed. They are GGML_VK_DISABLE_COOPMAT and GGML_VK_DISABLE_COOPMAT2.

I have used them to generate these graphs:

airlied · 2024-12-07T09:12:25Z

So I think in theory the A770 should have the hw instructions hooked up in the mesa driver, they emulate it on hw where it doesn't and Intel claimed it might be more optimal than having manually doing it, but I doubt this is always true.
https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/intel/compiler/brw_compiler.c?ref_type=heads#L104 is the dpas instruction lowering for hw that doesn't have it.

Here's the issue tracking proper VK_KHR_cooperative_matrix support: https://gitlab.freedesktop.org/mesa/mesa/-/issues/9250

Pretty sure nearly all that work was completed and upstream nearly a year ago, I played around with coop matrix on llama.cpp and a770 a while back, and ran into some of this and talked to a few Intel folks on irc, but I ended up getting distracted.

oscarbg · 2024-12-07T12:05:09Z

GGML_VK_DISABLE_COOPMAT

many thanks!!

jeffbolznv · 2024-12-07T19:25:06Z

@oscarbg we'll release an updated developer driver soon (next week, hopefully) with some additional optimizations for coopmat2, so it might make sense to wait for that.

…ng (ggml-org#10597) * Vulkan: Implement VK_KHR_cooperative_matrix support in the matrix matrix multiplication shader * Improve performance with better q4_k and q5_k dequant and store unrolling * Add Vulkan MUL_MAT and MUL_MAT_ID accumulator precision selection * Rework mulmat shader selection and compilation logic, avoid compiling shaders that won't get used by device * Vulkan: Implement accumulator switch for specific mul mat mat shaders * Vulkan: Unroll more loops for more mul mat mat performance * Vulkan: Add VK_AMD_shader_core_properties2 support to read Compute Unit count for split_k logic * Disable coopmat support on AMD proprietary driver * Remove redundant checks * Add environment variable GGML_VK_DISABLE_COOPMAT to disable VK_KHR_cooperative_matrix support * Fix rebase typo * Fix coopmat2 MUL_MAT_ID pipeline selection

* Merge vulkan code from mainline up to commit of 6/28/2025 * Vulkan Optimizations and Fixes (ggml-org#8959) * Optimize Vulkan REPEAT performance * Use Vulkan GLSL fused multiply-add instruction where possible * Add GGML_VULKAN_PERF option to output performance data per operator * Rework and fix Vulkan descriptor set and descriptor pool handling * Fix float32 concat f16 shader validation error * Add Vulkan GROUP_NORM eps parameter * Fix validation error with transfer queue memory barrier flags * Remove trailing whitespaces vulkan : do not use tensor->extra (ggml-org#9407) * vulkan : do not use tensor->extra This patch allows using the Vulkan backend with the RPC backend as tensor->extra is no longer used. Ref: ggml-org#8536 * Adapt GGML_VULKAN_CHECK_RESULTS to extra removal (F1LM1#2) --------- Co-authored-by: 0cc4m <picard12@live.de> # Conflicts: # ggml/src/ggml-vulkan.cpp vulkan : fix build (#0) ggml-ci Improve Vulkan shader build system (ggml-org#9239) * Improve Vulkan shader builds system - Add dependency to vulkan-shaders-gen to rebuild shaders when changing the shader compilation utility. - Add option to generate debug info for Vulkan shaders to provide shader source to Vulkan shader profiling tools * remove not required self dependency ggml : fix build break for the vulkan-debug (ggml-org#9265) - windows build : Ok. - linux build : Ok. Signed-off-by: Changyeon Kim <cyzero.kim@samsung.com> vulkan: correctly report support for OP_CONT (ggml/946) test-backend-ops fails because ggml_cont aborts when invoked passing an unsupported type. This commit makes ggml_cont tests pass Signed-off-by: Salvatore Mesoraca <s.mesoraca16@gmail.com> vulkan: add dryrun support to sin and cos ops (ggml/947) sin and cos failed test-backend-ops because they tried to dereference a context pointer that is null on dry runs. This commit prevents that segfault. Signed-off-by: Salvatore Mesoraca <s.mesoraca16@gmail.com> # Conflicts: # ggml/src/ggml-vulkan.cpp Overlap cmdbuffer creation and cmdbuffer execution in Vulkan backend by submitting smaller cmdbuffers early. (ggml-org#9118) * Overlap cmdbuffer creation and cmdbuffer execution in Vulkan backend by submitting smaller cmdbuffers early. * fix compile issues * Fix issues where the last submit wasn't executed or handled properly. * remove trailing whitespace * Repair GGML_VULKAN_CHECK_RESULTS * Increase submit counter only if actual work has been submitted and increase submit count to 100. * Fix some nodes are not checked with GGML_VULKAN_CHECK_RESULTS enabled. # Conflicts: # ggml/src/ggml-vulkan.cpp Enable use to the rebar feature to upload buffers to the device. (ggml-org#9251) vulkan : argsort barriers must be under uniform control flow (ggml/951) a return before a barrier (that happens only in some threads in a workgroup) leads to UB. While the old code actually works on some devices, it fails on some others (i.e. "smaller" GPUs). BTW, I think it would be better to set specialization constants when the graph is built, in that way the local workgroup could be sized appropriately. But it would take a lot of work. Signed-off-by: Salvatore Mesoraca <s.mesoraca16@gmail.com> vulkan : fix build for GGML_VULKAN_RUN_TESTS, add TFLOPS to log (ggml/961) vulkan : multithread pipeline creation (ggml/963) vulkan : mul_mat: fix UB with small warps (ggml/952) When the device's warp size is less than 16, it is possible for loadstride_a (mul_mm.comp:114) and loadstride_b (mul_mm.comp:115) to be set to 0. Because they are calculated as: the workgroup size, multiplied by LOAD_VEC_* (which can be 1) and divided by 16. And the workgroup size is set to be the same as the warp/subgroup size. The loadstride_* variables are used as increments in the loops that populate the buffers used for the multiplication. When they are 0 they cause an infinite loop. But infinite loops without side-effects are UB and the values of loadstride_* are known at compile time. So, the compiler quietly optimizes all the loops away. As a consequence, the buffers are not populated and the multiplication result is just a matrix with all elements set to 0. We prevent the UB by making sure that the workgroup size will never be less than 16, even if our device has a smaller warp size (e.g. 8). Signed-off-by: Salvatore Mesoraca <s.mesoraca16@gmail.com> vulkan : retry allocation with fallback flags (whisper/2451) Co-authored-by: Samuel Morris <samuel.morris@artlist.io> vulkan : improve ggml_vk_create_buffer error handling (ggml-org#9898) vulkan: Fix newly added tests for permuted mul_mat and 1D im2col (ggml-org#10226) vulkan: Throttle the number of shader compiles during the build step. (ggml-org#10222) Fixes ggml-org#9582 Spawning too many concurrent copies of glslc leads to "Failed to create pipes" errors on Linux. This change applies the same throttling we use for multithreaded pipeline creation. # Conflicts: # ggml/src/vulkan-shaders/vulkan-shaders-gen.cpp vulkan: Optimize contiguous copies (ggml-org#10254) * tests: Fix memory bandwidth calculation for perf tests Add a flops calculation for flash attention. Add one GGML_OP_CPY perf test. * vulkan: Optimize contiguous copies Add a variant of the copy shader for when the tensors are contiguous. Avoid the complex addressing calculations, and do four elements per invocation to hide some other overhead. Apply similar changes to the scale shader, since scale is always contiguous. Add a "progress bar" for shader compiles. # Conflicts: # tests/test-backend-ops.cpp vulkan: Use macros to make the mat mul pipeline creation more concise (ggml-org#10259) Also add vk_matmul_pipeline2 to hold f16/f32 accumulator versions of a pipeline. This isn't really used yet. vulkan: Optimize binary ops (ggml-org#10270) Reuse the index calculations across all of src0/src1/dst. Add a shader variant for when src0/src1 are the same dimensions and additional modulus for src1 aren't needed. Div/mod are slow, so add "fast" div/mod that have a fast path when the calculation isn't needed or can be done more cheaply. # Conflicts: # ggml/src/ggml-vulkan.cpp # ggml/src/vulkan-shaders/acc.comp ggml : vulkan logs (whisper/2547) vulkan: Optimize some mat-vec mul quant shaders (ggml-org#10296) Compute two result elements per workgroup (for Q{4,5}_{0,1}). This reuses the B loads across the rows and also reuses some addressing calculations. This required manually partially unrolling the loop, since the compiler is less willing to unroll outer loops. Add bounds-checking on the last iteration of the loop. I think this was at least partly broken before. Optimize the Q4_K shader to vectorize most loads and reduce the number of bit twiddling instructions. Vulkan: Fix device info output format specifiers (ggml-org#10366) * Vulkan: Fix device info output format specifiers * Vulkan: Use zu printf specifier for size_t instead of ld vulkan: remove use of null initializer (ggml-org#10372) Seems like this isn't working for vulkan-over-metal when the array is sized by a spec constant. Maybe a spirv-cross limitation? vulkan: Optimize soft_max (ggml-org#10301) * vulkan: Optimize soft_max Large soft_max could already saturate memory, but small/medium sizes were pretty slow. The bulk of the gains for them comes from using a smaller workgroup size, and making the workgroup size match the subgroup size also makes the barriers much cheaper. Cache some values in locals to avoid refetching/recomputing. And stamp out a few "template instantiations" so smaller cases will fully unroll. Add a missing early return for OOB rows. This happens when there are more than 512 rows and the dispatch is 512 x H. * vulkan: Further soft_max optimizations Restore the workgroup size of 512 case, use it for >1024. Use unrollable loops for more iteration counts. vulkan: further optimize mul_mat_vec using larger loads (ggml-org#10387) * vulkan: Use pipeline_robustness to disable robustness in mul_mat_vec. Add some early returns for nonexistent rows in mul_mat_vec shaders. These can only be hit when dispatching a 2D grid of workgroups. Fix the logic for the 2D grid of workgroups to round up. Enable the pipeline robustness extension if it's available, and use it to disable robustness for these pipelines. The instructions to do the bounds checking contend for the same ALU resources as the bit twiddling dequant instructions. * vulkan: Add GLSL structure aliases for quant types to allow larger loads In Vulkan it's not possible to cast pointer types, so instead you have to declare an aliased binding for the memory with a different type. This commit adds aliases for the quant formats using 16b ints, and in a few places where the struct size is a multiple of 4 also using 32b ints. Currently only q4_k's aliases are used, but others will be used in subsequent commits. * vulkan: use larger loads in q5_k and q6_k shaders. Similar to the optimization I did in q4_k recently, this vectorizes some loads and reduces the number of bit twiddling instructions. * vulkan: use larger K step per iteration in mul_mat_vec. Add vec4 dequantization functions, and use them to do K=8 per iteration in mul_mat_vec. This uses 16b loads for the quant values and 128b loads for B which helps reduce the load on the memory system. The K_PER_ITER==2 logic is still there, just for F16/F32, and really only because they support unaligned sizes. Tweak the num_iters/unrolling logic to be simpler and catch a couple missed unrolling opportunities. vulkan: copy iq4_nl LUT into shared memory (ggml-org#10409) vulkan: predicate max operation in soft_max shaders/soft_max (ggml-org#10437) Fixes ggml-org#10434 vulkan: Fix a vulkan-shaders-gen arugment parsing error (ggml-org#10484) The vulkan-shaders-gen was not parsing the --no-clean argument correctly. Because the previous code was parsing the arguments which have a value only and the --no-clean argument does not have a value, it was not being parsed correctly. This commit can now correctly parse arguments that don't have values. vulkan: fix group_norm (ggml-org#10496) Fix bad calculation of the end of the range. Add a backend test that covers the bad case (taken from stable diffusion). Fixes leejet/stable-diffusion.cpp#439. # Conflicts: # ggml/src/ggml-vulkan.cpp vulkan: optimize Q2_K and Q3_K mul_mat_vec (ggml-org#10459) vulkan: skip integer div/mod in get_offsets for batch_idx==0 (ggml-org#10506) vulkan: further optimize q5_k mul_mat_vec (ggml-org#10479) vulkan: Handle GPUs with less shared memory (ggml-org#10468) There have been reports of failure to compile on systems with <= 32KB of shared memory (e.g. ggml-org#10037). This change makes the large tile size fall back to a smaller size if necessary, and makes mul_mat_id fall back to CPU if there's only 16KB of shared memory. vulkan: define all quant data structures in types.comp (ggml-org#10440) vulkan: get the first command buffer submitted sooner (ggml-org#10499) This is an incremental improvement over ggml-org#9118 to get work to the GPU a bit sooner. The first part is to start with a smaller number of nodes before the first submit, and ramp it up to the current 100 nodes/submit. The second part is to reduce the dryrun overhead for all the nodes that just need to request descriptor space. With these changes I get around 1-2% speedup on RTX 4070 combined with my old Haswell-era CPU. vulkan: Dynamic subgroup size support for Q6_K mat_vec (ggml-org#10536) * subgroup 64 version with subgroup add. 15% faster scalable version tested for subgroup sizes 16-128 * check for subgroup multiple of 16 and greater than 16 * subgroup sizes are always a power of 2 (KhronosGroup/GLSL#45) * force 16 sequential threads per block * make 16 subgroup size a constant vulkan: optimize and reenable split_k (ggml-org#10637) Use vector loads when possible in mul_mat_split_k_reduce. Use split_k when there aren't enough workgroups to fill the shaders. vulkan: Implement "fast divide" (mul+shift) for unary ops like copy (ggml-org#10642) vulkan: Add VK_NV_cooperative_matrix2 support for mul_mat and flash attention (ggml-org#10206) # Conflicts: # ggml/src/vulkan-shaders/dequant_funcs_cm2.comp # ggml/src/vulkan-shaders/flash_attn_cm2.comp # ggml/src/vulkan-shaders/mul_mm_cm2.comp Vulkan: VK_KHR_cooperative_matrix support to speed up prompt processing (ggml-org#10597) * Vulkan: Implement VK_KHR_cooperative_matrix support in the matrix matrix multiplication shader * Improve performance with better q4_k and q5_k dequant and store unrolling * Add Vulkan MUL_MAT and MUL_MAT_ID accumulator precision selection * Rework mulmat shader selection and compilation logic, avoid compiling shaders that won't get used by device * Vulkan: Implement accumulator switch for specific mul mat mat shaders * Vulkan: Unroll more loops for more mul mat mat performance * Vulkan: Add VK_AMD_shader_core_properties2 support to read Compute Unit count for split_k logic * Disable coopmat support on AMD proprietary driver * Remove redundant checks * Add environment variable GGML_VK_DISABLE_COOPMAT to disable VK_KHR_cooperative_matrix support * Fix rebase typo * Fix coopmat2 MUL_MAT_ID pipeline selection # Conflicts: # ggml/src/ggml-vulkan.cpp vulkan: compile a test shader in cmake to check for coopmat2 support (ggml-org#10713) # Conflicts: # ggml/src/ggml-vulkan.cpp # ggml/src/ggml-vulkan/CMakeLists.txt # ggml/src/vulkan-shaders/test_coopmat2_support.comp Vulkan: fix NaN in tanh.comp with AMD proprietary driver on Windows (ggml-org#10723) * Vulkan: fix NaN in tanh.comp * Faster NaN-free tanh vulkan: fix compile warnings (ggml-org#10731) vulkan: disable spirv-opt for coopmat shaders (ggml-org#10763) There are some bugs in the 1.3.296 SDK, so disable this. It isn't strictly necessary anyway. Add missing dependency on vulkan-shaders-gen, so shaders get recompiled when it changes. Fix coopmat support reporting when glslc doesn't support NV_coopmat2. vulkan: dynamic subgroup size for the remaining k quants (ggml-org#10745) * q5_k q4_k q3_k q2_k q6_k multi row example * revert as multi row isnt faster for k quants vulkan: request round-to-even for fp16 in im2col/rope_head (ggml-org#10767) Vulkan doesn't mandate a specific rounding mode, but the shader_float_controls feature allows rounding mode to be requested if the implementation supports it. Vulkan: Add VK_EXT_subgroup_size_control support to ensure full subgroups for coopmats (ggml-org#10721) * Vulkan: Add VK_EXT_subgroup_size_control support to ensure full subgroups for coopmats * Fix subgroup size control extension support check Add accf32 and accf16 checks for coopmats * Also disable coopmats on amdvlk Vulkan: Use improved q4_k and q5_k dequant code in dequant shaders (ggml-org#10798) vulkan: small mul_mat_vec optimizations (ggml-org#10665) * double the number of rows per workgroup * Update ggml-vulkan.cpp * Vulkan: Add VK_EXT_subgroup_size_control support to ensure full subgroups for coopmats * only increase the number of rows for amd and subgroup size 64 * fix missing NUM_ROWS for mul_mat_vec_iq4_nl_f16_f32, untested * use subgroup min and max to check for gcn (requires ggml-org#10721) * manual merge ggml-vulkan.cpp * set min and max subgroup size in any case * Also double the number of rows for Intel GPUs Change Debug print name add GGML_ROPE_TYPE_MROPE rwkv6: add wkv6 support for Vulkan backend (ggml-org#10829) * rwkv_wkv6 vulkan shader * RWKV_WKV6 Vulkan op tests passed Signed-off-by: Molly Sophia <mollysophia379@gmail.com> * Apply code format changes Signed-off-by: Molly Sophia <mollysophia379@gmail.com> * add [[unroll]] and remove unnecessary conditions * add uma support * fix erros in EditorConfig Checker --------- Signed-off-by: Molly Sophia <mollysophia379@gmail.com> Co-authored-by: Molly Sophia <mollysophia379@gmail.com> # Conflicts: # ggml/src/ggml-vulkan.cpp # ggml/src/vulkan-shaders/wkv6.comp vulkan: bugfixes for small subgroup size systems + llvmpipe test (ggml-org#10809) * ensure mul mat shaders work on systems with subgroup size less than 32 more fixes add test * only s_warptile_mmq needs to be run with 32 threads or more # Conflicts: # .github/workflows/build.yml vulkan : fix soft_max.comp division by zero (whisper/2633) This change prevents a division by zero error when p.KY is 0. vulkan: optimize coopmat2 dequant functions (ggml-org#10855) Change the code to do 16b loads when possible and extract the appropriate component late, so the code is effectively decoding a pair of elements and then selecting one. This can allow more commoning to happen in the compiler when neighboring elements are loaded. vulkan: build fixes for 32b (ggml-org#10927) * vulkan: build fixes for 32b Should fix ggml-org#10923 * vulkan: initialize some buffer/offset variables examples, ggml : fix GCC compiler warnings (ggml-org#10983) Warning types fixed (observed under MSYS2 GCC 14.2.0): * format '%ld' expects argument of type 'long int', but argument has type 'size_t' * llama.cpp/ggml/src/ggml-vulkan/vulkan-shaders/vulkan-shaders-gen.cpp:81:46: warning: missing initializer for member '_STARTUPINFOA::lpDesktop' [-Wmissing-field-initializers] (emitted for all struct field except first) # Conflicts: # examples/export-lora/export-lora.cpp vulkan: multi-row k quants (ggml-org#10846) * multi row k quant shaders! * better row selection * more row choices * readjust row selection * rm_kq=2 by default vulkan: Use push constant offset to handle misaligned descriptors (ggml-org#10987) vulkan: im2col and matmul optimizations for stable diffusion (ggml-org#10942) * tests: Add im2col perf tests * vulkan: optimize im2col, more elements per thread * vulkan: increase small tile size for NV_coopmat2 * vulkan: change im2col to 512 elements per workgroup vulkan: optimize mul_mat for small values of N (ggml-org#10991) Make the mul_mat_vec shaders support N>1 (as a spec constant, NUM_COLS) where the batch_strides are overloaded to hold the row strides. Put the loads from the B matrix in the innermost loop because it should cache better. Share some code for reducing the result values to memory in mul_mat_vec_base. # Conflicts: # tests/test-backend-ops.cpp fix: Vulkan shader gen binary path (ggml-org#11037) Vulkan: Add device-specific blacklist for coopmat for the AMD proprietary driver (ggml-org#11074) * Vulkan: Add device-specific blacklist for coopmat for the AMD proprietary driver * Add (TM) to AMD name check fix lora print Disable GL_KHR_cooperative_matrix Vulkan extension if not available. (ggml-org#11117) * Disable GL_KHR_cooperative_matrix Vulkan extension if not available. * Perform Vulkan extensions checks in a more sensible order * Remove unnecessary #ifdef directive # Conflicts: # ggml/src/vulkan-shaders/test_coopmat_support.comp llama: add support for QRWKV6 model architecture (ggml-org#11001) Vulkan: Fix float16 use on devices without float16 support + fix subgroup_size_control validation error (ggml-org#11161) * Vulkan: Remove float16 use in shaders * Fix validation error about subgroup_size_control extension fix: ggml: fix vulkan-shaders-gen build (ggml-org#10448) * fix: ggml: fix vulkan-shaders-gen build The vulkan-shaders-gen target was not being built correctly in case of cross-compilation. Other outputs need to be built for the cross compile target, but vulkan-shaders-gen needs to be built for the host. * refactor: ggml: Improve vulkan-shaders-gen toolchain setup - Add GGML_SHADERS_GEN_TOOLCHAIN CMake option. - Auto-detect host toolchain if not set. * refactor: ggml: Improve vulkan-shaders-gen toolchain setup Use configure_file to generate host_toolchain.cmake from template * fix: ggml: Fix compile error Fix compile error not finding vulkan-shaders-gen * fix: vulkan-shaders-gen build and path handling Fix build issues with vulkan-shaders-gen: - Add target dependency for correct build order - Use CMAKE_HOST_SYSTEM_NAME for executable suffix - Fix MSVC output directory in host toolchain - Normalize path handling for cross-compilation * fix: improve host compiler detection in vulkan shader build Improve host compiler detection for vulkan shader generation: - Add NO_CMAKE_FIND_ROOT_PATH to all compiler searches - Consolidate compiler detection logic - Fix Windows-specific MSVC detection - Ensure correct compiler search in cross-compilation * refactor: Simplify CMake function for detecting host compiler Simplified the CMake function to improve the process of detecting the host compiler. * fix: Remove unnecessary Vulkan library linkage in CMakeLists.txt Since `vulkan-shader-gen.cpp` only requires the `glslc` executable and not the Vulkan headers or libraries, CMakeLists.txt needs to be corrected. (See: ecc93d0) * refactor: Rename host_toolchain.cmake.in - Rename host_toolchain.cmake.in to cmake/host-toolchain.cmake.in * refactor: GGML_VULKAN_SHADERS_GEN_TOOLCHAIN Rename the macro GGML_SHADERS_GEN_TOOLCHAIN to GGML_VULKAN_SHADERS_GEN_TOOLCHAIN # Conflicts: # ggml/src/ggml-vulkan/CMakeLists.txt vulkan: scale caching for k quants + misc fixes (ggml-org#11081) * q6_k scale caching * 16 bit unpack * q4_k test (slow) * revert it * q3_k * q2_k * little stuff * try precalculating products of a and q2_k scales * Revert "try precalculating products of a and q2_k scales" This reverts commit 65110b81f23f66331a50c6e889a7c1ab9470a86b. * unpack should be u16, add vim swap to gitignore (about time) * better q4_k scales * q5_k * better q6_k with separate paths for all threads and partial threads in use, plus some more optimizations * q2_k better dequant * q3_k optimizations * q3_k use hmask simd from cpu avx version * make the caches happy * q3_k separate out calculation * q2_k separate out * little stuff * use calc_superblock everywhere * q2_k optimize scale calculation * more barriers vulkan: optimize coopmat2 q2_k dequant function (ggml-org#11130) vulkan: optimize coopmat2 q4_k/q5_k dequant functions. (ggml-org#11206) Do masking on whole dwords, fetch all scales at once. vulkan: support copy from f32 to q4_0/q4_1/q5_0/q5_1/q8_0/iq4_nl (ggml-org#11166) * vulkan: support copy from f32 to q4_0/q4_1/q5_0/q5_1/q8_0/iq4_nl Shaders are based on cpy.cu. * vulkan: support copy from q4_0/q4_1/q5_0/q5_1/q8_0/iq4_nl to f32 * ggml: copy q->f32 assumes some contiguity in the destination # Conflicts: # ggml/src/ggml-cpu/ggml-cpu.c # ggml/src/vulkan-shaders/copy_from_quant.comp # ggml/src/vulkan-shaders/copy_to_quant.comp vulkan: fix coopmat2 flash attention for non-contiguous inputs (ggml-org#11281) Add code similar to mul_mm_cm2 to force alignment of strides, to avoid a performance regression. Add noncontiguous FA tests in test-backend-ops. Fixes ggml-org#11268. # Conflicts: # tests/test-backend-ops.cpp vulkan: fix coopmat2 validation failures (ggml-org#11284) mul mat and flash attention shaders were loading f32 types directly into A/B matrices, which happens to work but is technically invalid usage. For FA, we can load it as an Accumulator matrix and convert and this is not in the inner loop and is cheap enough. For mul mat, it's more efficient to do this conversion in a separate pass and have the input(s) be f16. coopmat2 requires SPIR-V 1.6 (related using to LocalSizeId). LocalSizeId requires maintenance4 be enabled, and SPIR-V 1.6 requires Vulkan 1.3. vulkan: fix diag_mask_inf (ggml-org#11323) With robustbufferaccess disabled, this shader was showing OOB stores. There is a bounds check in the code, but the workgrouop dimensions were reversed vs CUDA and it was running the wrong number of threads. So fix the workgroup dimensions and disable robustness for this pipeline. vulkan: sort shaders for more deterministic binary (ggml-org#11315) Fixes ggml-org#11306. Vulkan-run-test: fix mmq_wg_denoms (ggml-org#11343) There should be a copy-and-paste error here. *mmq_wg_denoms should be used together with *warptile_mmq, instead of wg_denoms. vulkan: compile shaders on-demand (ggml-org#11406) Reduce first-run startup time and memory consumption. Should fix ggml-org#11339. vulkan: Catch pipeline creation failure and print an error message (ggml-org#11436) * vulkan: Catch pipeline creation failure and print an error message Also, fix some warnings from my on-demand compile change. * vulkan: fix pipeline creation logging vulkan: implement initial support for IQ2 and IQ3 quantizations (ggml-org#11360) * vulkan: initial support for IQ3_S * vulkan: initial support for IQ3_XXS * vulkan: initial support for IQ2_XXS * vulkan: initial support for IQ2_XS * vulkan: optimize Q3_K by removing branches * vulkan: implement dequantize variants for coopmat2 * vulkan: initial support for IQ2_S * vulkan: vertically realign code * port failing dequant callbacks from mul_mm * Fix array length mismatches * vulkan: avoid using workgroup size before it is referenced * tests: increase timeout for Vulkan llvmpipe backend --------- Co-authored-by: Jeff Bolz <jbolz@nvidia.com> # Conflicts: # ggml/src/vulkan-shaders/dequant_iq2_s.comp # ggml/src/vulkan-shaders/dequant_iq2_xs.comp # ggml/src/vulkan-shaders/dequant_iq2_xxs.comp # ggml/src/vulkan-shaders/dequant_iq3_s.comp # ggml/src/vulkan-shaders/dequant_iq3_xxs.comp CUDA: non-contiguous (RMS) norm support (ggml-org#11659) vulkan: use smaller combined allocations to avoid fragmentation (ggml-org#11551) # Conflicts: # ggml/src/ggml-alloc.c vulkan: initial support for IQ4_XS quantization (ggml-org#11501) # Conflicts: # ggml/src/vulkan-shaders/dequant_iq4_xs.comp vulkan: optimize coopmat2 iq2/iq3 callbacks (ggml-org#11521) * vulkan: optimize coopmat2 iq2/iq3 callbacks * build: trigger CI on GLSL compute shader changes vulkan: print shared memory size (ggml-org#11719) # Conflicts: # ggml/src/ggml-vulkan.cpp vulkan: account for lookup tables when checking shared memory size (ggml-org#11502) # Conflicts: # ggml/src/ggml-vulkan.cpp vulkan: add environment variable GGML_VK_PREFER_HOST_MEMORY to avoid VRAM allocation (ggml-org#11592) vulkan: linux builds + small subgroup size fixes (ggml-org#11767) * mm subgroup size * upload vulkan x86 builds vulkan: initial support for IQ1_S and IQ1_M quantizations (ggml-org#11528) * vulkan: initial support for IQ1_S and IQ1_M quantizations * vulkan: define MMV kernels for IQ1 quantizations * devops: increase timeout of Vulkan tests again * vulkan: simplify ifdef for init_iq_shmem # Conflicts: # ggml/src/vulkan-shaders/dequant_iq1_m.comp # ggml/src/vulkan-shaders/dequant_iq1_s.comp # ggml/src/vulkan-shaders/mul_mat_vec_iq1_m.comp # ggml/src/vulkan-shaders/mul_mat_vec_iq1_s.comp vulkan: support multi/vision rope, and noncontiguous rope (ggml-org#11902) # Conflicts: # ggml/src/ggml-vulkan.cpp # ggml/src/vulkan-shaders/rope_multi.comp # ggml/src/vulkan-shaders/rope_vision.comp vulkan: implement several ops relevant for ggml_opt (ggml-org#11769) * vulkan: support memset_tensor * vulkan: support GGML_OP_SUM * vulkan: implement GGML_OP_ARGMAX * vulkan: implement GGML_OP_SUB * vulkan: implement GGML_OP_COUNT_EQUAL * vulkan: implement GGML_OP_OPT_STEP_ADAMW * vulkan: fix check_results RWKV_WKV6 crash and memory leaks * vulkan: implement GGML_OP_REPEAT_BACK * tests: remove invalid test-backend-ops REPEAT_BACK tests * vulkan: fix COUNT_EQUAL memset using a fillBuffer command # Conflicts: # ggml/src/ggml-vulkan.cpp # ggml/src/vulkan-shaders/argmax.comp # ggml/src/vulkan-shaders/count_equal.comp # ggml/src/vulkan-shaders/opt_step_adamw.comp # ggml/src/vulkan-shaders/repeat_back.comp # ggml/src/vulkan-shaders/sub.comp # tests/test-backend-ops.cpp vulkan: implement more backpropagation operators (ggml-org#11914) * vulkan: implement GGML_OP_ROPE_BACK * vulkan: implement GGML_OP_RMS_NORM_BACK * vulkan: implement GGML_OP_SILU_BACK * vulkan: implement GGML_OP_SOFTMAX_BACK # Conflicts: # ggml/src/vulkan-shaders/rms_norm_back.comp # ggml/src/vulkan-shaders/silu_back.comp # ggml/src/vulkan-shaders/soft_max_back.comp Add memset tensor in all backend interface SYCL: implement memset ggml backend buffer interface (ggml-org#12580) * SYCL: implement memset ggml backend buffer interface * use GGML_ABORT macro * Do not wait for all queues to finish for memset operation # Conflicts: # ggml/src/ggml-sycl.cpp add OP sigmoid (ggml-org#12056) Co-authored-by: Judd <foldl@boxvest.com> # Conflicts: # ggml/src/vulkan-shaders/sigmoid.comp vulkan: fix assertion when qy_needs_dequant (ggml-org#12068) Looks like a copy/paste bug from qx_needs_dequant. vulkan: improve im2col (ggml-org#11826) * vulkan: improve im2col performance vulkan: matmul dequantization improvements (ggml-org#12015) * faster dequant for old quants * dont use unpack for iq4_nl * vec2 unpack for q8 vulkan: add specific MMV kernels for IQ2 and IQ3 quants + optimizations (ggml-org#11595) * vulkan: implement specialized MMV kernels for IQ2 quantizations * vulkan: add MMV kernels for IQ3 quants * vulkan: Increase MMV batch size and unroll IQ LUT setup * vulkan: fix init_iq_shmem for WG sizes larger than tables * vulkan: common batch size for all I-quants # Conflicts: # ggml/src/vulkan-shaders/mul_mat_vec_iq2_s.comp # ggml/src/vulkan-shaders/mul_mat_vec_iq2_xs.comp # ggml/src/vulkan-shaders/mul_mat_vec_iq2_xxs.comp # ggml/src/vulkan-shaders/mul_mat_vec_iq3_s.comp # ggml/src/vulkan-shaders/mul_mat_vec_iq3_xxs.comp cuda/vulkan: specify fp32-only support for some operations in supports_op (ggml/1129) ggml-ci # Conflicts: # ggml/src/ggml-cuda.cu # tests/test-backend-ops.cpp mat vec double buffer (ggml-org#12188) vulkan: fix bug in coopmat1 mul_mat_id (ggml-org#12316) * tests: run mul_mat_id with a larger N * vulkan: fix bug in coopmat1 mul_mat_id Update build.yml for Windows Vulkan builder to use Vulkan 1.4.304 SDK for VK_NV_cooperative_matrix2 support (ggml-org#12301) vulkan: Adjust coopmat2 tile sizes and selection heuristic (ggml-org#12258) vulkan: Pad N dimension of B matrix for coopmat2 perf, to avoid bounds checking (ggml-org#12273) * vulkan: Pad N dimension of B matrix for coopmat2 perf, to avoid bounds checking vulkan: use fp32 in coopmat2 q4_k dequant function (ggml-org#12309) vulkan: subgroup size tuning (ggml-org#12087) * vulkan: subgroup size test * Vulkan: Add device architecture enum and logic to recognize AMD generations * vulkan: use new architecture logic to specify subgroup size * Initial vulkan subgroup size tuning for RDNA3 * vulkan: commonize RDNA subgroup tuning * vulkan: override subgroup size if required_subgroup_size = 0 * vulkan: disable warp 32 for RDNA3 * vulkan: fine tuned RDNA1 subgroup sizes * vulkan: adjusted subgroup size map * vulkan: fixed RDNA2 subgroup map --------- Co-authored-by: 0cc4m <picard12@live.de> vulkan: Add N/2 and N/4 optimized paths in coopmat2 shader (ggml-org#12312) ggml-vulkan: remove unused find_program(glslc) (ggml-org#12416) It's already found by FindVulkan.cmake in the parent CMakeLists Vulkan: Default to 1GB allocations instead of 4GB to avoid fragmentation and driver issues (ggml-org#12434) vulkan: Submit once enough matmul work has been recorded (ggml-org#12406) I've been seeing significantly worse performance for tg with flash attention enabled vs disabled, and it seems to be related to the submit heuristic. Change the heuristic to check how many bytes worth of weight matrix are used and flush every 100MB, and ramp up after the first few submits. This seems to resolve the issue, and also increases perf for non-FA a bit. vulkan: optimize iq1 coopmat2 dequant functions (ggml-org#12427) vulkan: workaround for AMD Windows driver 16 bit unpack8 bug (ggml-org#12472) Vulkan: RTE rounding for cpy to quant (ggml-org#12480) * Vulkan: RTE rounding for cpy to quant Co-Authored-By: Jeff Bolz <jbolz@nvidia.com> * remove trailing whitespace * avoid duplicating pipeline_cpy_f32_quant * fix copypasting issue * remove duplicated code --------- Co-authored-by: Jeff Bolz <jbolz@nvidia.com> vulkan: Optimize mul_mat_vec p021 and nc shaders (ggml-org#12505) * tests: add mul_mat perf/functional tests for p021/nc vulkan shaders * vulkan: Optimize mul_mat_vec p021 and nc shaders. These shaders are used in attention calculations, and when the KV cache grows large they start to dominate the run time. For the nc shader (which is called with large 'k' dimension), use unrolling and vector loads. For the p021 shader (which is called with large 'm' and small 'k' dimensions), take advantage of grouped query attention to reuse loads from the A matrix for the whole group, and reduce the number of workgroups (too much overhead from tiny dispatches). Using subgroupAdd in the p021 shader also helps, use that conditionally. # Conflicts: # tests/test-backend-ops.cpp vulkan: fix mul_mat_vec failure in backend tests (ggml-org#12529) The OOB calculation could be wrong if the last iteration was during one of the unrolled loops. Adjust the unrolling counts to avoid this. Add a couple new backend tests that hit this failure on NVIDIA GPUs. vulkan: fix coopmat shader generation when cross-compiling (ggml-org#12272) * vulkan: fix coopmat shader generation when cross-compiling Previously the status of coopmat{,2} support isn't passed to the vulkan-shaders-gen project building on the host, which leads to build failure because of the cross-compiling code expecting coopmat{,2} shaders that didn't get generated. Fix this by passing the coopmat{,2} support status to vulkan-shaders subproject. Signed-off-by: Icenowy Zheng <uwu@icenowy.me> * Only call coop-mat shaders once * Fix whitespace --------- Signed-off-by: Icenowy Zheng <uwu@icenowy.me> Co-authored-by: bandoti <141645996+bandoti@users.noreply.github.com> cmake: improve Vulkan cooperative matrix support checks (whisper/2966) Co-authored-by: Sandro Hanea <me@sandro.rocks> cmake : fix whitespace (#0) Vulkan: Add DP4A MMQ and Q8_1 quantization shader (ggml-org#12135) * Vulkan: Add DP4A MMQ and Q8_1 quantization shader * Add q4_0 x q8_1 matrix matrix multiplication support * Vulkan: Add int8 coopmat MMQ support * Vulkan: Add q4_1, q5_0 and q5_1 quants, improve integer dot code * Add GL_EXT_integer_dot_product check * Remove ggml changes, fix mmq pipeline picker * Remove ggml changes, restore Intel coopmat behaviour * Fix glsl compile attempt when integer vec dot is not supported * Remove redundant code, use non-saturating integer dot, enable all matmul sizes for mmq * Remove redundant comment * Fix integer dot check * Fix compile issue with unsupported int dot glslc * Update Windows build Vulkan SDK version # Conflicts: # ggml/src/ggml-vulkan.cpp # ggml/src/vulkan-shaders/mul_mmq.comp # ggml/src/vulkan-shaders/mul_mmq_funcs.comp # ggml/src/vulkan-shaders/quantize_q8_1.comp # ggml/src/vulkan-shaders/test_integer_dot_support.comp vulkan: fix build when glslc doesn't support coopmat (ggml-org#12683) Vulkan: Fix mmq int dot float cache size (ggml-org#12722) vulkan: Implement grouped query attention in the coopmat2 FA shader (ggml-org#12559) When adjacent batches of Q share the same batches of K/V, batch them into the same workgroup. For example, when: dst(128,32,1,1) = FA(q(128,1,32,1), k(128,16640,8,1), v(128,16640,8,1)) previously we would run 32 workgroups computing 1 result each, now we will run 8 workgroups computing 4 results each. This doesn't directly translate to better performance (at least when you have >=32 SMs), but in a subsequent change I'll enable split_k which will scale much better with 4x fewer workgroups. cmake: remove caching from vulkan coopmat checks (ggml-org#12719) vulkan: Implement split_k for coopmat2 flash attention. (ggml-org#12627) When using group query attention, we have one workgroup per KV batch and this can be very few workgroups (e.g. just 8 in some models). Enable split_k to spread the work across SMs. This helps a lot when the KV cache is large. # Conflicts: # ggml/src/vulkan-shaders/flash_attn_split_k_reduce.comp vulkan: Fix missing cmake logic for dot product extension (ggml-org#12721) vulkan: set cmake minimum and project name in vulkan-shaders (ggml-org#12744) vulkan: Hybrid waitForFences/getFenceStatus to reduce fence latency (ggml-org#12630) There seems to be a bubble waking up from waitForFences, which costs a few percent performance and also increased variance in performance. This change inserts an "almost_ready" fence when the graph is about 80% complete and we waitForFences for the almost_ready fence and then spin (with _mm_pauses) waiting for the final fence to be signaled. # Conflicts: # ggml/src/ggml-vulkan.cpp cmake: fix ggml-shaders-gen compiler paths containing spaces (ggml-org#12747) fixes error for compiler paths with spaces Vulkan: Tune Vulkan mmq int dot shader for performance (ggml-org#12767) vulkan: Use unclamped loads for flash attention mask (ggml-org#12720) nem1 must be a multiple of GGML_KQ_MASK_PAD, and GGML_KQ_MASK_PAD is a multiple of the number of rows in the matrix. The KV dim is a multiple of the number of columns for the aligned shader. vulkan: fix NaN issue in flash attention shader (ggml-org#12776) Use -FLT_MAX/2 rather than -inf as the initial value for computing the maximum. vulkan: Use fp16 for the flash attention P*V multiplication (ggml-org#12783) This is consistent with the ggml-cuda behavior and the mul_mat fallback. vulkan: In coopmat2 mmq, load q4_k/q5_k scales through shared memory (ggml-org#12833) q4_k and q5_k had a lot of redundant global loads where the same 16B of scale information is repeatedly loaded and decoded during each loop iteration. This change restructures the loops to more explicitly iterate over whole blocks in the outer loop (with unrolled inner loop) and to copy/decode the scale data into shared memory once at the start of each outer loop. The copy is pipelined so the scale load from global memory is relatively cheap. This improves q4_k/q5_k model prompt processing performance by around 5-7%. I briefly tried applying this to q6_k and q4_0, and it didn't help for q6_k and hurt for q4_0. The big "else" path in mul_mm_cm2.comp that had all the clamped/unclamped variants isn't used as often as it originally was (e.g. due to the padded_N change), so I trimmed it down to offset some of the new complexity of the semi-manual loop unrolling. vulkan: use aligned loads for flash attention mask (ggml-org#12853) Rewrite the stride logic for the mask tensor in the FA shader to force the stride to be aligned, to allow using more efficient loads. vulkan: enable coopmat2 FA gqa and split_k optimizations more often (ggml-org#12931) The grouped query attention optmization doesn't require a power of two ratio, the only thing relying on it was the modulo operation written as bitwise &. split_k need not depend on gqa_ratio - enable it any time there's only one workgroup in the X dimension. The shader gets the split index from the x coord, and multiple workgroups in the X dimension (pre-split) indicates a larger FA operation that wouldn't need splitting. vulkan: support noncontiguous rms_norm (ggml-org#13031) # Conflicts: # ggml/src/ggml-vulkan.cpp vulkan: matmul gcn tuning (ggml-org#13016) * tune matmul for gcn * this one is more power efficient * Update ggml/src/ggml-vulkan/ggml-vulkan.cpp Co-authored-by: 0cc4m <picard12@live.de> * disable this tune for the proprietary driver --------- Co-authored-by: 0cc4m <picard12@live.de> vulkan: use uint array index to avoid glslang bug (ggml-org#13193) vulkan: Handle src1 batch dimension in non-contiguous mat-vec-mul shader (ggml-org#13191) * vulkan: Handle src1 batch dimension in non-contiguous mat-vec-mul shader vulkan: Add bfloat16 support (ggml-org#12554) * vulkan: Add bfloat16 support This adds bfloat16 matrix multiply support based on VK_KHR_shader_bfloat16. The extension is required for coopmat multiply support, but matrix-vector multiply trivially promotes bf16 to fp32 and doesn't require the extension. The copy/get_rows shaders also don't require the extension. It's probably possible to fall back to non-coopmat and promote to fp32 when the extension isn't supported, but this change doesn't do that. The coopmat support also requires a glslc that supports the extension, which currently requires a custom build. * vulkan: Support bf16 tensors without the bf16 extension or coopmat support Compile a variant of the scalar mul_mm shader that will promote the bf16 values to float, and use that when either the bf16 extension or the coopmat extensions aren't available. * vulkan: bfloat16 fixes (really works without bfloat16 support now) * vulkan: fix spirv-val failure and reenable -O # Conflicts: # ggml/src/vulkan-shaders/test_bfloat16_support.comp vulkan: Additional type support for unary, binary, and copy (ggml-org#13266) Support f16->f32 copy. Support f16->f16 and f32->f32 unary ops. Support all combinations of f16/f32 for src0/src1/dst for add/sub/mul/div. # Conflicts: # ggml/src/ggml-vulkan.cpp vulkan: Allow up to 4096 elements for mul_mat_id row_ids (ggml-org#13326) This assert fired running Qwen_Qwen3-30B-A3B-Q2_K.gguf: GGML_ASSERT(nei0 * nei1 <= 3072); The tensor is 8 x 512. Increase this array size to accommodate. vulkan: scalar flash attention implementation (ggml-org#13324) * vulkan: scalar flash attention implementation * vulkan: always use fp32 for scalar flash attention * vulkan: use vector loads in scalar flash attention shader * vulkan: remove PV matrix, helps with register usage * vulkan: reduce register usage in scalar FA, but perf may be slightly worse * vulkan: load each Q value once. optimize O reduction. more tuning * vulkan: support q4_0/q8_0 KV in scalar FA * CI: increase timeout to accommodate newly-supported tests * vulkan: for scalar FA, select between 1 and 8 rows * vulkan: avoid using Float16 capability in scalar FA # Conflicts: # ggml/src/ggml-vulkan.cpp # ggml/src/vulkan-shaders/flash_attn.comp vulkan: workaround FA compile failures on macos (ggml-org#13517) vulkan: KHR_coopmat flash attention (ggml-org#13506) This shader uses coopmat1 to do the Q*K^T multiply. The P*V multiply is more difficult for various reasons so I haven't done it. Performance for this shader is around 2.5x better than for the scalar shader when doing prompt processing. Some of the benefit may be from other optimizations like staging through shared memory, or splitting by rows. # Conflicts: # ggml/src/vulkan-shaders/flash_attn_cm1.comp cmake: simplify vulkan shader test logic (ggml-org#13263) vulkan: use scalar FA rather than coopmat2 when N==1 (ggml-org#13554) Add pipeline_acc_f32 vulkan: move common FA code to flash_attn_base.comp (ggml-org#13556) * vulkan: move common FA code to flash_attn_base.comp * vulkan: move common FA index/stride setup code to flash_attn_base.comp * build fix # Conflicts: # ggml/src/vulkan-shaders/flash_attn_base.comp cmake: use the current build config for vulkan-shaders-gen (ggml-org#13595) * fix: use the current build config for `vulkan-shaders-gen` * fix: only pass a valid build type to `--config` Vulkan: Add f32 accumulator support to quantized mul mat to fix GLM4 32B incoherence (ggml-org#13607) # Conflicts: # ggml/src/ggml-vulkan.cpp vulkan: fix warnings (ggml-org#13626) * small fixes * remove ifdef use LOG_WARN to replace `std::cerr` (ggml-org#13657) vulkan: Disable coopmat/coopmat2/bfloat extensions if glslc doesn't support it (ggml-org#13696) vulkan: support CPY from any type to itself (ggml-org#13695) Reuse the f16/f32 copy shaders, and just scale the number of elements according to the type size. add GGML_LOG_WARN vulkan: mark IM2COL as supporting non-contig (ggml-org#13783) # Conflicts: # ggml/src/ggml-vulkan.cpp vulkan: use timestamp queries for GGML_VULKAN_PERF (ggml-org#13817) Also change it to be controlled by an env var rather than cmake flag vulkan : Remove unexpected ; (ggml/1253) vulkan: fix warnings in perf logger querypool code (ggml-org#13937) ggml-vulkan: adds support for op CONV_TRANSPOSE_1D (ggml-org#13813) * * ggml-vulkan: adds op CONV_TRANSPOSE_1D * test-backend-ops: adds more spohisticated tests for CONV_TRANSPOSE_1D * Missing barrier added to shader. Number of additional tests reduced to 108. * * Fixes typo in variable name. * Removes extra whitespaces. * Adds int64->int32 casts to prevent possible warnings. * Problem size reduced in tests to pass tests with llvmpipe. * supports_op condition moved from unintended position # Conflicts: # ggml/src/ggml-vulkan.cpp # ggml/src/vulkan-shaders/conv_transpose_1d.comp vulkan: Enable VK_KHR_cooperative_matrix extension for Intel Xe2 GPUs (ggml-org#14001) * allowing B580 and U9-288V * experimenting code to detect Xe2 * allowing coopmat only for Xe2 GPUs * fixed comment wording * fixed comment wording * removed unnecessary driver check Vulkan: Don't default to CPU device (like llvmpipe), even if no other device is available, to allow fallback to CPU backend (ggml-org#14099) # Conflicts: # ggml/src/ggml-vulkan.cpp vulkan: force device 0 in CI (ggml-org#14106) Add GGML_LOG_INFO vulkan: Track descriptor pools/sets per-context (ggml-org#14109) Use the same descriptor set layout for all pipelines (MAX_PARAMETER_COUNT == 8) and move it to the vk_device. Move all the descriptor pool and set tracking to the context - none of it is specific to pipelines anymore. It has a single vector of pools and vector of sets, and a single counter to track requests and a single counter to track use. vulkan: Better thread-safety for command pools/buffers (ggml-org#14116) This change moves the command pool/buffer tracking into a vk_command_pool structure. There are two instances per context (for compute+transfer) and two instances per device for operations that don't go through a context. This should prevent separate contexts from stomping on each other. # Conflicts: # ggml/src/ggml-vulkan.cpp vulkan: mutex around vkQueueSubmit (ggml-org#14127) This fixes the remaining crash in test-thread-safety on my system. cmake: clean up external project logic for vulkan-shaders-gen (ggml-org#14179) * Remove install step for vulkan-shaders-gen * Add install step to normalize msvc with make * Regenerate modified shaders at build-time # Conflicts: # .github/workflows/build.yml cmake: remove shader-gen step-targets from ggml-vulkan (ggml-org#14226) * Remove step-targets from vulkan-shaders-gen * Unset DESTDIR when building vulkan-shaders-gen Vulkan: Set device max size for host memory to avoid OOM warning and fallback to CPU buffer (ggml-org#14249) Add support for VK_EXT_debug_utils to add labels to Vulkan objects. (ggml-org#13792) * Add support for VK_EXT_debug_utils to add labels to Vulkan objects. In step 1 compute pipelines are getting labeled. * remove #ifdef for debug utils and add queue marker. # Conflicts: # ggml/src/ggml-vulkan.cpp vulkan: update windows SDK in CI (ggml-org#14334) vulkan: update windows SDK in release.yml (ggml-org#14344) # Conflicts: # .github/workflows/release.yml cmake: regen vulkan shaders when shaders-gen sources change (ggml-org#14398) * Add shaders-gen sources as target deps vulkan: Fix GGML_VULKAN_SHADER_DEBUG_INFO (ggml-org#14427) This setting needs to be passed through to vulkan-shaders-gen vulkan: lock accesses of pinned_memory vector (ggml-org#14333) vulkan: handle noncontig in the final case of ggml_vk_get_cpy_pipeline (ggml-org#14378) Fix cuda build error test * remove new cpu backend and yml files * remove new op and GGML_ROPE_TYPE_NEOX * fix build error * change cmake file to add matrix operation * remove coopmat2 check in flash attention * print gpu info for vulkan * disable fuse to recover vulkan performance --------- Co-authored-by: 0cc4m <picard12@live.de> Co-authored-by: firecoperana <firecoperana>

github-actions bot added Vulkan Issues specific to the Vulkan backend ggml changes relating to the ggml tensor library for machine learning labels Nov 30, 2024

jeffbolznv reviewed Nov 30, 2024

View reviewed changes

ggml/src/ggml-vulkan/ggml-vulkan.cpp Outdated Show resolved Hide resolved

ggml/src/ggml-vulkan/vulkan-shaders/mul_mm.comp Show resolved Hide resolved

jeffbolznv reviewed Dec 2, 2024

View reviewed changes

0cc4m mentioned this pull request Dec 3, 2024

Misc. bug: Inconsistent Vulkan segfault #10528

Open

0cc4m force-pushed the 0cc4m/vulkan-coopmat branch from cf8fcae to 7002d6c Compare December 3, 2024 20:23

jeffbolznv mentioned this pull request Dec 4, 2024

vulkan: Add VK_NV_cooperative_matrix2 support for mul_mat and FlashAttention2 #10206

Merged

4 tasks

0cc4m marked this pull request as ready for review December 4, 2024 16:05

0cc4m added 2 commits December 5, 2024 20:44

Disable coopmat support on AMD proprietary driver

3a58a01

Remove redundant checks

f54afb4

0cc4m force-pushed the 0cc4m/vulkan-coopmat branch from b654968 to f54afb4 Compare December 5, 2024 20:47

Add environment variable GGML_VK_DISABLE_COOPMAT to disable VK_KHR_co…

c66d008

…operative_matrix support

jeffbolznv reviewed Dec 5, 2024

View reviewed changes

ggml/src/ggml-vulkan/ggml-vulkan.cpp Show resolved Hide resolved

ggml/src/ggml-vulkan/ggml-vulkan.cpp Outdated Show resolved Hide resolved

ggml/src/ggml-vulkan/vulkan-shaders/mul_mm.comp Show resolved Hide resolved

jeffbolznv approved these changes Dec 5, 2024

View reviewed changes

Fix rebase typo

e1d17a2

Fix coopmat2 MUL_MAT_ID pipeline selection

eeaf0b9

0cc4m force-pushed the 0cc4m/vulkan-coopmat branch from 297f5ca to eeaf0b9 Compare December 7, 2024 07:41

0cc4m merged commit 3df784b into master Dec 7, 2024
44 checks passed

0cc4m deleted the 0cc4m/vulkan-coopmat branch December 7, 2024 09:24

oscarbg mentioned this pull request Dec 7, 2024

Feature Request: provide "Vulkan enabled" prebuilt ubuntu binaries in releases page (https://github.com/ggerganov/llama.cpp/releases) #10707

Closed

4 tasks

majunlichn mentioned this pull request Dec 9, 2024

Misc. bug: llama-server: Invalid Reply Generated with Vulkan Backend with VK_KHR_cooperative_matrix Support #10734

Closed

theaerotoad mentioned this pull request Dec 11, 2024

Compile bug: Vulkan build fails on GL_KHR_cooperative_matrix #10785

Closed

Vulkan: VK_KHR_cooperative_matrix support to speed up prompt processing #10597

Vulkan: VK_KHR_cooperative_matrix support to speed up prompt processing #10597

Uh oh!

Conversation

0cc4m commented Nov 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0cc4m commented Nov 30, 2024

RTX 3090

Intel A770 (Linux Mesa)

Uh oh!

qnixsynapse commented Nov 30, 2024

Uh oh!

0cc4m commented Nov 30, 2024

Uh oh!

jeffbolznv left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jeffbolznv commented Nov 30, 2024

Uh oh!

0cc4m commented Nov 30, 2024

Uh oh!

jeffbolznv commented Nov 30, 2024

Uh oh!

jeffbolznv commented Nov 30, 2024

Uh oh!

characharm commented Dec 1, 2024

Uh oh!

easyfab commented Dec 1, 2024

Uh oh!

0cc4m commented Dec 1, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jeffbolznv commented Dec 2, 2024

Uh oh!

qnixsynapse commented Dec 2, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

0cc4m commented Dec 2, 2024

Uh oh!

qnixsynapse commented Dec 2, 2024

Uh oh!

0cc4m commented Dec 2, 2024

Uh oh!

qnixsynapse commented Dec 2, 2024

Uh oh!

characharm commented Dec 2, 2024

Uh oh!

jeffbolznv commented Dec 3, 2024

Uh oh!

0cc4m commented Dec 3, 2024

Uh oh!

0cc4m commented Dec 4, 2024

Uh oh!

0cc4m commented Dec 4, 2024

Uh oh!

jeffbolznv commented Dec 5, 2024

Uh oh!

0cc4m commented Dec 5, 2024

Uh oh!

Uh oh!

0cc4m commented Nov 30, 2024 •

edited

Loading

qnixsynapse commented Dec 2, 2024 •

edited

Loading

airlied commented Dec 6, 2024 •

edited

Loading

oscarbg commented Dec 7, 2024 •

edited

Loading