Apple Silicon GPU Support Possible?

[bfrasure]

The CUDA acceleration is very impressive. Does anyone know of any efforts to run this on the GPU cores of the M processors? I'd be willing to assist but I'd rather not start from scratch if something exists.

[evanmiller]

You can see the author's previous notes on attempting to use the M1 GPU on GPT-J:

https://github.com/ggerganov/ggml/tree/master/examples/gpt-j#attempt-to-use-the-m1-gpu

Basically he found that with Apple's unified memory architecture, the bottleneck is memory bandwidth rather than pure compute.

[ggerganov]

@evanmiller This info is outdated (and likely wrong) - I am working on offloading the full computation on the M1 GPU with custom kernels and hope to do it properly this time around.

[x4080]

Can't wait for this, @ggerganov

[iandennismiller]

I've been trying to understand more about MPS and I found a few resources that helped.

Philip Turner has been doing interesting work with Metal:

• https://discuss.pytorch.org/t/sequential-throughput-of-gpu-execution/156303
• https://github.com/philipturner/opencl-metal-stdlib
• https://github.com/philipturner/metal-float64
• https://github.com/philipturner/metal-benchmarks
• https://github.com/philipturner/openmm-metal
• https://github.com/philipturner/metal-experiment-1

Also, I learned a lot about the limitations of MPS from this pytorch thread, titled "MPS device appears much slower than CPU on M1 Mac Pro." It's an old thread but it still has current activity:

pytorch/pytorch#77799

I thought I would leave these links in case it helps someone else.

[philipturner]

Context size 2048, 512 tokens, LLaMa 6.7B (3.9 GB)

Layers on Accelerate	Layers on CLBlast	Sequential Throughput/Token	Bandwidth
32	0	40590 µs	96 GB/s
30	2	58150 µs	67 GB/s
28	4	76310 µs	51 GB/s
24	8	113460 µs	34 GB/s
16	16	198330 µs	20 GB/s
0	32	253050 µs	15 GB/s

Theoretically it should be able to utilize more bandwidth. I think I can make this an order of magnitude faster. It will require a ton of tuning to align memory transactions and utilize ~376 GB/s of the bandwidth. Use triangular FlashAttention for long context, with dynamic work redistribution to keep GPU cores fully utilized.

Variant	Layers on Accelerate	Sequential Throughput/Token	Bandwidth
6.7B	32	40590 µs	96 GB/s
13.0B	40	75370 µs	103 GB/s
32.5B	60	173540 µs	112 GB/s
65.2B	80	34842780 µs	1 GB/s

Also, has anyone considered lane compression, so I can run 65B-q4 on 32 GB RAM with reasonable speed?

[philipturner]

I'm trying to fill in this table. Next is the PyTorch MPS fork of LLaMa.cpp that's slower than CPU.

Latency per 512 tokens:

LLaMa	6.7B	13.0B	32.5B	65.2B
PyTorch
LLaMa.cpp	20.8 s	38.6 s	88.9 s	17839.5 s
MLC LLM
MPSGraph
Metal FlashAttention
Theoretical Lower Bound	4.4 s	8.6 s	21.6 s	43.3 s

[iandennismiller]

Very happy to see @philipturner in this thread. @ggerganov , philipturner is an expert with MPS, M1 architecture, and GPU computation in general. I believe there is a lot of hard-won esoteric knowledge when it comes to optimizing for this architecture. So excited to see where this leads.

[philipturner]

I want an honest comparison. Please make LLaMa.cpp as fast as possible; I think I can make something faster. An open-source successor to MPS.

optimizing for this architecture

Why not AMD and Intel too?

[x4080]

@philipturner have you take a look at https://github.com/mlc-ai/web-llm especially https://github.com/mlc-ai/mlc-llm ?
It can run apple silicon gpu pretty fast using webgpu. I tested it and it seems as fast as llama cpp but with no heat

[philipturner]

have you take a look at https://github.com/mlc-ai/web-llm

I tested it and it seems as fast as llama cpp but with no heat

Can you quantify how much faster?

[x4080]

@philipturner yes, mlc llm is as fast as llama cpp, like I said above

[philipturner]

I suspect it might be a slight bit slower, just like PyTorch MPS. It's one thing to use the GPU. It's another to use it right.

If you use Metal the way it's designed, it should not be 10% slower than CPU, but 300% faster.

This is why I asked specifically for a number. Is it 0% faster? 50% faster? 50% slower?

[x4080]

Its about the same 😄

I was surprised as well, because I alread read about your work, btw I'm using M2 pro 16GB, and latest ventura dont know if that make a difference, have you tried it and the result is slower than llama cpp ?

Using M1?

[philipturner]

I have not benchmarked it yet. What speed are you getting on CPU with LLaMa.cpp? Try reporting the ms/token with -c 2048 -n 512. Run about 10 times, and report the fastest trial. Also exclude trials where it finishes early (before 450 tokens).

After we can quantify LLaMa.cpp on your device, I'll find a way to get a numerical measurement from your MLC experience.

---

To explain the benefit of using Metal correctly, you're talking about jumps from the red curve to the cyan curve. LLMs are smaller problems (1000 atoms -> 1000 tokens) and memory/latency bounded. For big stuff like Stable Diffusion, you can grossly misuse the MTLCommandQueue _{(ahem, PyTorch)}, but the problem's so big, the compute bottleneck is even larger.

Most people use Metal like this:

MTLCommandQueue {
  MTLCommandBuffer {
    MTLComputeCommandEncoder {
      // command that adds two 1000-wide tensors
      // finishes in 1 microsecond
    }
  }
  // 10 microseconds latency
  MTLCommandBuffer {
    MTLComputeCommandEncoder {
      // command that multiplies two 1000-wide tensors
      // finishes in 1 microsecond
    }
  }
  // 10 microseconds latency
  MTLCommandBuffer {
    MTLComputeCommandEncoder {
      // command that exponentiates two 1000-wide tensors
      // finishes in 1 microsecond
    }
  }
  // 10 microseconds latency
  MTLCommandBuffer.waitUntilCompeted {
    // 200 microseconds latency
  }
}

How it's designed to be used:

MTLCommandQueue {
  MTLCommandBuffer {
    MTLComputeCommandEncoder {
      // command that adds two 1000-wide tensors
      // finishes in 1 microsecond
      // command that multiplies two 1000-wide tensors
      // finishes in 1 microsecond
      // command that exponentiates two 1000-wide tensors
      // finishes in 1 microsecond

      // some very clever GPGPU way to run the complex
      // logic on the GPU, to prevent synchronization
      // with the CPU
      // finishes in 5 microseconds
    }
  }
  // 10 microseconds latency
}

[x4080]

For llama.cpp, now i'm using 13b models, so its kinda slower for the model that I use for mlc (7b), yes the mlc dont have any speed displayed but it really brrr 😄

Before using the web version, I think its about >20 token/s