ggml : implement op fusion, starting with REGLU/GEGLU/SWIGLU #14158

CISC · 2025-06-12T23:24:57Z

~~Implement REGLU/GEGLU/SWIGLU ops to avoid unnecessary tensor duplications and a little more efficient execution by combining ops in one.~~

Implement op fusion, starting with REGLU/GEGLU/SWIGLU for PoC.

~~Only CPU and CUDA right now, help needed to complete other backends!~~

ggerganov

I missed that these ops change the shape of the input tensor.

I think it would be better to introduce:

enum ggml_glu_op {
    GGML_GLU_OP_REGLU,
    GGML_GLU_OP_GEGLU,
    GGML_GLU_OP_SWIGLU,
};

// similar to ggml_unary()
GGML_API struct ggml_tensor * ggml_glu(
        struct ggml_context * ctx,
         struct ggml_tensor * a,
           enum ggml_glu_op   op);

// these simply call ggml_glu()
GGML_API struct ggml_tensor * ggml_reglu(
        struct ggml_context * ctx,
        struct ggml_tensor  * a);

GGML_API struct ggml_tensor * ggml_geglu(
        struct ggml_context * ctx,
        struct ggml_tensor  * a);

GGML_API struct ggml_tensor * ggml_swiglu(
        struct ggml_context * ctx,
        struct ggml_tensor  * a);

ggml/include/ggml.h

ggerganov

Hope we don't forget to implement these in the rest of the backends.

Adding @JohannesGaessler for review of the CUDA changes.

ggerganov · 2025-06-13T08:31:31Z

Only CPU and CUDA right now, help needed to complete other backends!

Yes, let's add the rest of the backends first before merging. At least Metal and Vulkan.

ggml/src/ggml-cuda/unary.cu

JohannesGaessler · 2025-06-13T09:14:48Z

More generally, I've been thinking that it would be useful to have something like a backend-specific graph optimization step in ggml. That way you could do things like fuse tensors only if the fused tensor is supported by the backend and only if using it makes sense given the tensor shapes.

CISC · 2025-06-13T10:29:17Z

Only CPU and CUDA right now, help needed to complete other backends!

Yes, let's add the rest of the backends first before merging. At least Metal and Vulkan.

Any suggestions on who could help with that?

ngxson · 2025-06-13T13:14:56Z

ggml/include/ggml.h

+            struct ggml_context * ctx,
+            struct ggml_tensor  * a);
+
+    GGML_API struct ggml_tensor * ggml_swiglu(


just want to note that I have been observing one variants of swiglu. it's used by ultravox, which sigmoid the second half of the vector instead of the first half

Oh, interesting, worth adding a parameter for, or best just handling in conversion?
https://huggingface.co/fixie-ai/ultravox-v0_5-llama-3_3-70b/blob/main/ultravox_model.py#L701-L704

I think it would be nice to have a param since the GGUFs are already on the internet. Haven't thought about permuting the FFN up tensor before, nice suggestion

Added swapped variants.

@ggerganov I didn't dare update metal code, so needs to be implemented there too. :)

JohannesGaessler · 2025-06-13T13:29:00Z

@0cc4m @jeffbolznv are either of you interested in a Vulkan implementation?

0cc4m · 2025-06-13T13:42:21Z

I can look into it tomorrow.

ggml/src/ggml-cuda/unary.cu

JohannesGaessler · 2025-06-13T17:47:38Z

CUDA performance test:

GPU	Model	Microbatch size	Test	t/s master	t/s cisc/unary-reglu-geglu-swiglu	Speedup
RTX 4090	chatglm 9B Q4_0	1	pp512	157.48	160.49	1.02
RTX 4090	chatglm 9B Q4_0	2	pp512	268.16	276.78	1.03
RTX 4090	chatglm 9B Q4_0	4	pp512	517.41	535.36	1.03
RTX 4090	chatglm 9B Q4_0	8	pp512	826.69	855.46	1.03
RTX 4090	chatglm 9B Q4_0	16	pp512	1407.13	1453.62	1.03
RTX 4090	chatglm 9B Q4_0	32	pp512	2545.45	2664.80	1.05
RTX 4090	chatglm 9B Q4_0	64	pp512	4414.61	4704.57	1.07
RTX 4090	chatglm 9B Q4_0	128	pp512	6467.60	7028.01	1.09
RTX 4090	chatglm 9B Q4_0	256	pp512	8670.62	9451.16	1.09
RTX 4090	chatglm 9B Q4_0	512	pp512	9842.99	10832.14	1.10

Also a plot of the same data using #14169 :

ggerganov · 2025-06-13T19:11:08Z

CUDA performance test:

Huh, I didn't expect the benefit to be that much. Interesting.

jeffbolznv · 2025-06-13T20:27:21Z

Huh, I didn't expect the benefit to be that much. Interesting.

Everything other than mat-mat mul is either bandwidth or small dispatch limited. Fusion is a big opportunity. We should reopen discussions about how to enable more types of fusion.

CISC · 2025-06-13T20:54:58Z

CUDA performance test:

Nice! Will be interesting to see numbers on other backends as well...

CISC · 2025-06-13T21:38:33Z

Hmmm, it just occurred to me that we should be able to (now that I pass along a pointer to the gate separately) perform these ops on models with separate ffn_up/gate tensors too by conditionally setting src[1].

ngxson · 2025-06-13T21:59:11Z

ggml/include/ggml.h

+            struct ggml_context * ctx,
+            struct ggml_tensor  * a);
+
+    GGML_API struct ggml_tensor * ggml_geglu(


Tbh I don't even know why geglu was added in the first place. It doesn't seem to be used by any models. And to make matter worse, the PR where it was added has no useful description: #14074

So I wonder if we actually need to implement it as a kernel. The current kernel use tanh approximation, but in practice, there can be many different approximations for gelu op.

Nvm, see: https://github.com/ggml-org/llama.cpp/pull/14014/files#r2146203459

I've seen several, and in fact we already support a few (Gemma, DeepSeekV1, Jina-Bert and T5), it's just that the gate is split (some at conversion because we didn't have the op).

So I wonder if we actually need to implement it as a kernel. The current kernel use tanh approximation, but in practice, there can be many different approximations for gelu op.

It's pretty easy adding different GLU ops (and in CUDA I even reuse the original op), adding GEGLU_ERF if necessary shouldn't be a problem.

0cc4m · 2025-06-14T10:14:36Z

I implemented Vulkan shaders for the new ops.

qnixsynapse · 2025-06-14T13:43:41Z

Interesting.. I tried implementing for SYCL, saw little improvement. When I saw the graph logs, it wasn't using the fused kernels for llama 3.2 3B.

[SYCL][OP] call ggml_sycl_mul_mat: dst='ffn_gate-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]	src0='blk.19.ffn_gate.weight':type=f16;ne=[3072, 8192, 1, 1];nb=[2, 6144, 50331648, 50331648]	src1='ffn_norm-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288]
[SYCL][OP] call ggml_sycl_op_dequantize_mul_mat_vec/to_fp16_sycl: dst='ffn_gate-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]	src0='blk.19.ffn_gate.weight':type=f16;ne=[3072, 8192, 1, 1];nb=[2, 6144, 50331648, 50331648]	src1='ffn_norm-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288] : converting src1 to fp16
[SYCL][OP] call ggml_sycl_op_dequantize_mul_mat_vec/to_fp16_sycl done
[SYCL][OP] call ggml_sycl_mul_mat done
[SYCL][OP] call ggml_sycl_silu: dst='ffn_silu-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]	src0='ffn_gate-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]
[SYCL][OP] call ggml_sycl_silu done
[SYCL][OP] call ggml_sycl_mul_mat: dst='ffn_up-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]	src0='blk.19.ffn_up.weight':type=f16;ne=[3072, 8192, 1, 1];nb=[2, 6144, 50331648, 50331648]	src1='ffn_norm-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288]
[SYCL][OP] call ggml_sycl_op_dequantize_mul_mat_vec/to_fp16_sycl: dst='ffn_up-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]	src0='blk.19.ffn_up.weight':type=f16;ne=[3072, 8192, 1, 1];nb=[2, 6144, 50331648, 50331648]	src1='ffn_norm-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288] : converting src1 to fp16
[SYCL][OP] call ggml_sycl_op_dequantize_mul_mat_vec/to_fp16_sycl done
[SYCL][OP] call ggml_sycl_mul_mat done
[SYCL][OP] call ggml_sycl_mul: dst='ffn_gate_par-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]	src0='ffn_silu-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]	src1='ffn_up-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]
[SYCL][OP] call ggml_sycl_mul done
[SYCL][OP] call ggml_sycl_mul_mat: dst='ffn_out-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288]	src0='blk.19.ffn_down.weight':type=f16;ne=[8192, 3072, 1, 1];nb=[2, 16384, 50331648, 50331648]	src1='ffn_gate_par-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768]
[SYCL][OP] call ggml_sycl_op_dequantize_mul_mat_vec/to_fp16_sycl: dst='ffn_out-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288]	src0='blk.19.ffn_down.weight':type=f16;ne=[8192, 3072, 1, 1];nb=[2, 16384, 50331648, 50331648]	src1='ffn_gate_par-19':type=f32;ne=[8192, 1, 1, 1];nb=[4, 32768, 32768, 32768] : converting src1 to fp16
[SYCL][OP] call ggml_sycl_op_dequantize_mul_mat_vec/to_fp16_sycl done
[SYCL][OP] call ggml_sycl_mul_mat done
[SYCL][OP] call ggml_sycl_add: dst='l_out-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288]	src0='ffn_out-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288]	src1='ffn_inp-19':type=f32;ne=[3072, 1, 1, 1];nb=[4, 12288, 12288, 12288]
[SYCL][OP] call ggml_sycl_add done

I am using from this branch.

qnixsynapse · 2025-06-21T04:24:33Z

Hi. I plan to rebase this branch with master today but I am seeing conflicts in metal backend code as well. I'll try my best but if something goes incorrect please do correct it in the next commit. Thank you.

ggml-ci

* implement GLU for split up/gate * add tests for ggml_glu_split * Vulkan: Implement glu_split logic and shader support * add split to logging [no ci] * SYCL: refactor element_size ops and add split up and gate support to gated kernels * SYCL: switch GEGLU to use tanh approximation --------- Co-authored-by: 0cc4m <picard12@live.de> Co-authored-by: Akarshan <akarshan@menlo.ai>

CISC · 2025-06-22T06:42:21Z

Hi. I plan to rebase this branch with master today but I am seeing conflicts in metal backend code as well. I'll try my best but if something goes incorrect please do correct it in the next commit. Thank you.

Thanks. :)

…nlining This commit refactors the SYCL element-wise operations to improve performance by: - Inlining unary operations (sgn, abs, elu, gelu, silu, etc.) to reduce kernel launch overhead. - Introducing helper functions `op_xxx` for each unary operation to encapsulate the logic. - Replacing direct kernel calls with calls to these inlined functions. - Using `__dpct_inline__` to encourage compiler inlining. - Minor code cleanup and consistency improvements. The changes aim to reduce kernel launch overhead and improve the overall efficiency of element-wise operations on SYCL devices.

CISC · 2025-06-22T16:52:00Z

@qnixsynapse The latest improvements are probably worth extracting into a separate PR instead of keeping them locked away here.

qnixsynapse · 2025-06-22T17:20:41Z

@qnixsynapse The latest improvements are probably worth extracting into a separate PR instead of keeping them locked away here.

This one's the last, to partially address one of the review comments here.

jeffbolznv · 2025-06-23T18:25:48Z

I did a quick proof of concept of how to detect fusion opportunities in jeffbolznv@2ed9fb4. This adds the use_count, reverts the llama-graph changes in this PR, and implements a peephole optimization that converts the views+conts+mul+silu into swiglu. I verified it successfully fuses and has the same perf as this PR.

When it fuses, it's modifying the node->op to the new GGML_OP_SWIGLU added in this change, and changing the other tensors to GGML_OP_NONE. I haven't thought through what's the best way to represent fused operations in the backend. Maybe it's GGML_OP_FUSED, or maybe not, but there may be additional storage needed to save certain information about the combined operation, if we don't want to regenerate it multiple times.

Given how complete the implementation of GGML_OP_GLU is, IMO we should go ahead with this PR as-is and not block it waiting on fusion support. This pattern is a bit trickier than the usual sequence of elementwise operations and isn't something I'd expect to cover in the initial implementation of fusion.

CISC · 2025-06-23T19:06:52Z

I did a quick proof of concept of how to detect fusion opportunities in jeffbolznv@2ed9fb4. This adds the use_count, reverts the llama-graph changes in this PR, and implements a peephole optimization that converts the views+conts+mul+silu into swiglu. I verified it successfully fuses and has the same perf as this PR.

Sweet, though I guess it gets a little more complicated for a proper implementation as this is (for PoC simplicity) done after graph split and does not take backend support into consideration. Very cool though!

CISC · 2025-06-23T19:13:58Z

When it fuses, it's modifying the node->op to the new GGML_OP_SWIGLU added in this change, and changing the other tensors to GGML_OP_NONE. I haven't thought through what's the best way to represent fused operations in the backend. Maybe it's GGML_OP_FUSED, or maybe not, but there may be additional storage needed to save certain information about the combined operation, if we don't want to regenerate it multiple times.

I wonder if it would be possible to make a parallel graph node that bridges the ops that can be fused with the fused op so that it can be an optional route while still keeping the unfused route.

Edit: This means you can have smaller fusions as well, if backends don't support a larger fusion op.

slaren · 2025-06-23T19:47:27Z

I did a quick proof of concept of how to detect fusion opportunities in jeffbolznv@2ed9fb4. This adds the use_count, reverts the llama-graph changes in this PR, and implements a peephole optimization that converts the views+conts+mul+silu into swiglu. I verified it successfully fuses and has the same perf as this PR.

I am not sure that doing the operation fusing in ggml_backend_sched would be good. It would require querying the backend for support of fused ops, and it would limit the number of possible fused operations that may be implemented by a backend. I also don't think we should add more functionality to ggml_backend_sched at this point, it needs to be refactored first.

What I was thinking about would be adding a function to check if operator fusion is possible that the backends can use to easily determine if they can fuse a sequence of operations. Roughly, something like this:

ggml_status graph_compute(ggml_backend_t backend, struct ggml_cgraph * cgraph) {
    for (int node = 0; node < cgraph->n_nodes; node++) {
        if (ggml_can_fuse(cgraph, node, { GGML_OP_MUL_MAT, GGML_OP_ADD, GGML_OP_UNARY }, 3)) {
            compute_fused_mul_mat_bias(...);
            node += 2;
        } else if (ggml_can_fuse(cgraph, node, { GGML_OP_MUL_MAT, GGML_OP_ADD }, 2)) {
            compute_fused_mul_mat_bias_act(...);
            node += 1;
        } else {
            // run unfused operation
            compute_node(...);
        }
    }
    return GGML_STATUS_SUCCESS;
}

This would not really work for something like SWIGLU, but I think that's complex enough that it deserves to have it own operation.

jeffbolznv · 2025-06-23T19:59:41Z

What I was thinking about would be adding a function to check if operator fusion is possible that the backends can use to easily determine if they can fuse a sequence of operations. Roughly, something like this:

Sure, I'm happy to do it that way. I mostly wanted to show what the peephole detection looks like, for a complex case like this.

CISC added the help wanted Extra attention is needed label Jun 12, 2025

CISC requested a review from ggerganov June 12, 2025 23:25

github-actions bot added testing Everything test related Nvidia GPU Issues specific to Nvidia GPUs ggml changes relating to the ggml tensor library for machine learning labels Jun 12, 2025

ggerganov reviewed Jun 13, 2025

View reviewed changes

ggml/include/ggml.h Outdated Show resolved Hide resolved

CISC changed the title ~~ggml : implement unary REGLU/GEGLU/SWIGLU ops~~ ggml : implement REGLU/GEGLU/SWIGLU ops Jun 13, 2025

CISC requested a review from ggerganov June 13, 2025 08:23

ggerganov approved these changes Jun 13, 2025

View reviewed changes

ggerganov requested a review from JohannesGaessler June 13, 2025 08:30

JohannesGaessler reviewed Jun 13, 2025

View reviewed changes

ggml/src/ggml-cuda/unary.cu Outdated Show resolved Hide resolved

ggml/src/ggml-cuda/unary.cu Show resolved Hide resolved

ggml/src/ggml-cuda/unary.cu Outdated Show resolved Hide resolved

github-actions bot added the Apple Metal https://en.wikipedia.org/wiki/Metal_(API) label Jun 13, 2025

ngxson reviewed Jun 13, 2025

View reviewed changes

JohannesGaessler reviewed Jun 13, 2025

View reviewed changes

ggml/src/ggml-cuda/unary.cu Outdated Show resolved Hide resolved

ngxson reviewed Jun 13, 2025

View reviewed changes

CISC mentioned this pull request Jun 14, 2025

llama: Attempt to add ModernBert #14014

Open

github-actions bot added the Vulkan Issues specific to the Vulkan backend label Jun 14, 2025

CISC mentioned this pull request Jun 14, 2025

ggml : implement GLU for split up/gate #14181

Merged

CISC and others added 18 commits June 22, 2025 10:37

implement unary REGLU/GEGLU/SWIGLU cpu ops

76c9bc1

relax constraints

5a490f0

duplicate shape of source

56c7993

fix ggml_vec_geglu_f16

21c4963

special case gated ops

bb2fda7

implement unary REGLU/GEGLU/SWIGLU cuda ops

a1a7b6d

tighten constraints again

f8c2080

refactor into GGML_GLU_OP

a341aa3

metal : add glu kernels

d9ddeb9

ggml-ci

add CUDA_GLU_BLOCK_SIZE [no ci]

cfa9c7a

more constraints and use 64bit ints

70e8b48

ggml-ci

64bit multiplication [no ci]

f8705a2

implement swapped variants (cpu/cuda)

0b2703f

update comment [no ci]

d593429

ggml-ci

Vulkan: Add GLU ops and shaders

34d1aed

SYCL: Implement fused kernel GEGLU, SWIGLU and REGLU for single up+gate

a9aedf4

GGML: increase OP count in assertion

a234e09

qnixsynapse force-pushed the cisc/unary-reglu-geglu-swiglu branch from c2af58b to a234e09 Compare June 22, 2025 05:09

jeffbolznv mentioned this pull request Jun 23, 2025

vulkan: Increase workgroup size for GLU, for performance #14345

Open

ggml : implement op fusion, starting with REGLU/GEGLU/SWIGLU #14158

Are you sure you want to change the base?

ggml : implement op fusion, starting with REGLU/GEGLU/SWIGLU #14158

Uh oh!

Conversation

CISC commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ggerganov left a comment

Choose a reason for hiding this comment

Uh oh!

ggerganov commented Jun 13, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JohannesGaessler commented Jun 13, 2025

Uh oh!

CISC commented Jun 13, 2025

Uh oh!

ngxson Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

CISC Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

ngxson Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

CISC Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

JohannesGaessler commented Jun 13, 2025

Uh oh!

0cc4m commented Jun 13, 2025

Uh oh!

Uh oh!

JohannesGaessler commented Jun 13, 2025

Uh oh!

ggerganov commented Jun 13, 2025

Uh oh!

jeffbolznv commented Jun 13, 2025

Uh oh!

CISC commented Jun 13, 2025

Uh oh!

CISC commented Jun 13, 2025

Uh oh!

ngxson Jun 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ngxson Jun 13, 2025

Choose a reason for hiding this comment

Uh oh!

CISC Jun 14, 2025

Choose a reason for hiding this comment

Uh oh!

CISC Jun 14, 2025

Choose a reason for hiding this comment

Uh oh!

0cc4m commented Jun 14, 2025

Uh oh!

qnixsynapse commented Jun 14, 2025

Uh oh!

qnixsynapse commented Jun 21, 2025

Uh oh!

CISC commented Jun 22, 2025

Uh oh!

CISC commented Jun 22, 2025

Uh oh!

qnixsynapse commented Jun 22, 2025

Uh oh!

jeffbolznv commented Jun 23, 2025

Uh oh!

CISC commented Jun 23, 2025

Uh oh!

CISC commented Jun 12, 2025 •

edited

Loading

ngxson Jun 13, 2025 •

edited

Loading

CISC commented Jun 23, 2025 •

edited

Loading

slaren commented Jun 23, 2025 •

edited

Loading