opencl: improve profiling

[lhez]

• Wait for profiling events and collect profiling data when model execution is done. This way, the displayed performance numbers are more close to the true performance.
• Generate a chrome trace in addition to csv.

[zhouwg]

sorry to bother you, how can I mark a specified PR as ready for review? thanks.
one more thing, can Qualcomm's expert help to review a PR of Qualcomm's QNN backend for llama.cpp from an independent programmer(here is me and I already bought Qualcomm's Snapdragon 8gen3 equipped Android phone for personal learning dev activity and plan to buy a Snapdragon 8 Elite equipped phone for further dev activity later)?I know your team already has a senior technical expert whom knows everything about ggml-qnn and I hope can get help from your colleague. thanks too much.

[max-krasnyansky]

Looks good, and reminds me that I wanted to integrate graph-profiler branch with opencl and cpu backends.

[max-krasnyansky]

sorry to bother you, how can I mark a specified PR as ready for review? thanks.

If you start a Draft PR there is a way to mark it ready. qnn-backend PR is not marked as draft.

one more thing, can Qualcomm's expert help to review a PR of Qualcomm's QNN backend for llama.cpp from an independent programmer(here is me and I already bought Qualcomm's Snapdragon 8gen3 equipped Android phone for personal learning dev activity and plan to buy a Snapdragon 8 Elite equipped phone for further dev activity later)?I know your team already has a senior technical expert whom knows everything about ggml-qnn and I hope can get help from your colleague. thanks too much.

I've been keep an eye on it. In general, I'd say QNN is not the right solution here but I'll take another look.

[zhouwg]

thanks for your valuable time and attention on ggml-qnn(Qualcomm's backend for llama.cpp through Qualcomm QNN SDK) and my third formal PR of ggml-qnn.

1. is "QNN is not the right solution here" means we can't utilize the Hexagon NPU maximally through Qualcomm's QNN SDK in llama.cpp? even through the "second tech approach(mapping the entire ggml cgraph to a single QNN graph)"?

2. that's why the GPU performance in that PR(or all similar PR) is so bad at the same time Intel's ggml-sycl is so good? accordingly, that's why Qualcomm provide ggml-opencl for Qualcomm's GPU backend(right solution for Qualcomm GPU backend in llama.cpp)?

3. that's why the NPU performance in that PR(or all similar PR) is so bad?

4. that's why Qualcomm didn't provide an official ggml-qnn PR through the general approach in Intel's ggml-sycl or the second tech approach(mapping the entire ggml cgraph to a single QNN graph) with QNN SDK currently?

5. from this diagram,
   

we can see the key-point of "how to utilize the Hexagon NPU maximally through QNN SDK" is that we should compose an ideal single QNN graph from an entire/complete ggml cgraph. as my personal understanding, that's exactly the core principle(Qualcomm's dedicated binary tool convert a specified LLM model to a prepared QNN graph) in QNN SampleApp or Genie stack.

I don't understand some tech questiones:

• why "QNN is not the right solution here"? is there any other AI software stack or SDK provided by Qualcomm for llama.cpp's Hexagon NPU backend?
• or the first tech approach(general approach in Intel's ggml-sycl) and the second tech approach(mapping a complete/entire ggml cgraph to a single QNN graph through QNN SDK) in that PR are both incorrect?
• or the second tech approach(mapping the entire/complete ggml cgraph to a single QNN graph through QNN SDK) is correct path but the practice in that PR is not exactly correct? I understand the shared-buffer or memory-pool also should be used in the second tech approach.

I or/and the entire llama.cpp community need your further guidance and help.

thanks so much again.

[zhouwg]

@max-krasnyansky, thanks so much for your valuable guidance/correction on direction.

I think I know something about the third tech approach of "utilize the Hexagon NPU maximally", in other words, the Hexagon DSP SDK should be used in the third tech approach, which is exactly similar to what your excellent engineering team did with ggml-opencl, or which is exactly similar to what I did with video decoding hardware acceleration many years ago(it's also a DSP chip).

my guess might-be not correct, accordingly, it's greatly appreciated that you can give me/the llama.cpp community a clear explanation or a roughly confirmation of the third tech approach.

[zhouwg]

sorry to bother you, how can I mark a specified PR as ready for review? thanks.

If you start a Draft PR there is a way to mark it ready. qnn-backend PR is not marked as draft.

one more thing, can Qualcomm's expert help to review a PR of Qualcomm's QNN backend for llama.cpp from an independent programmer(here is me and I already bought Qualcomm's Snapdragon 8gen3 equipped Android phone for personal learning dev activity and plan to buy a Snapdragon 8 Elite equipped phone for further dev activity later)?I know your team already has a senior technical expert whom knows everything about ggml-qnn and I hope can get help from your colleague. thanks too much.

I've been keep an eye on it. In general, I'd say QNN is not the right solution here but I'll take another look.

yes, you are absolutely correct, thanks so much. I'll try to test it on a rooted 8gen3 phone to get better NPU performance(closer to the default ggml backend or better then the default ggml backend).

[updated on 03/21/2025,23:54] offload to cDSP directly is really much faster than QNN-CPU/QNN-GPU/QNN-NPU backend. now GGML_OP_ADD works fine on cDSP and verified with llama-cli: the NPU performance is good.

ggml-hexagon-offload-to-cdsp-directly.mp4

I think this approach is exactly similar to ggml-opencl: we need write some "kernels" on cDSP and then offload specified ggml ops to cDSP directly. there is a question in this approach: build the entire source code is not easy because one of parts are running on AP(arm-cpu) and one of parst are running on cDSP(NPU), at the same time, opencl provide a mature mechanism to manage/compile "kernels" naturally.

hope mulmat can also works fine on cDSP and NPU performance is also good tomorrow, then that PR can be reviewed formally.

[updated on 03/22/2025,23:22] source code has been submitted in that PR. there is an unknown issue about mulmat on cDSP. another issue in that PR: I didn't find a general approach to compile/build the hexagon kernel functions in that PR at the moment and a manual method can works fine in my local dev envs.

[zhouwg]

[updated on 03/24/2025,15:32]mulmat performance between QNN-NPU and cDSP:

mulmat performance between QNN-NPU and cDSP

case-1:
mulmat with QNN-NPU:

mulmat with cDSP(with naive algorithm):

case-2:
mulmat with QNN-NPU:

mulmat with cDSP(with naive algorithm):

case-3:
mulmat with QNN-NPU:

mulmat with cDSP(with algorithm from ggml-dsp(a tiny customized ggml on cDSP), but the compute result is not correct):

mulmat with cDSP(with naive algorithm):

case-4:
mulmat with QNN-NPU:

mulmat with cDSP(with naive algorithm):

@max-krasnyansky, I think this testcase can verify what you said "QNN is not the right solution here" is absolutely correct if there is no misunderstanding or mistake in this testcase/experiment: we can clearly see that performance of mulmat with cDSP is much faster then mulmat with QNN-NPU.

unfortunately, this hand-written implementation of mulmat on cDSP is very naive(without any optimization) and can only pass various case in UT with self-made command line program, it can't works with llama-cli.

I think AI experts can have chance to improve the mulmat algorithm on cDSP if that PR can be approved: this approach is exactly what you pointed out, developers and AI experts can do a lot on the cDSP rather than based on the black box in the QNN SDK.

I'd like to make a little contribution to Qualcomm although I'm an independent programmer, thanks so much!