Skip to content

WIP: llama: Vulkan: Fix Adreno Q8_0 issues. #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 39 commits into from
Closed

WIP: llama: Vulkan: Fix Adreno Q8_0 issues. #11

wants to merge 39 commits into from

Conversation

infinitalo
Copy link

This MR is a work-in-progress.

The current commits are able to get inference working for Q8_0 on Adreno 830 (Samsung S25), but finetuning still crashes.

We're currently working on a fix for lora-finetuning on Adreno A830, but you can use this for testing in the meanwhile.

makaveli10 and others added 18 commits July 30, 2025 16:58
@makaveli10
Add lora finetuning from adapter
d8ba38a
@makaveli10
Add: create new lora adapter for target modules to finetune if no lor…
c8ea7b2 
…a is provided
@makaveli10
Fix identical loss over epochs; fix garbage lora initization
69c464a 
Signed-off-by: vineet <vineet.suryan@collabora.com>
@makaveli10
Remove lora training from finetune.cpp
ab0dae2 
Signed-off-by: vineet <vineet.suryan@collabora.com>
@makaveli10
Add adapter saving & other lora target modules
1b6e22d 
Signed-off-by: vineet <vineet.suryan@collabora.com>
@makaveli10
Add finetune-lora for lora finetuning in examples
c61a5af 
Signed-off-by: vineet <vineet.suryan@collabora.com>
@makaveli10
Add dequantization to out_prod cuda kernel
7799375 
Signed-off-by: vineet <vineet.suryan@collabora.com>
@makaveli10
Update README with finetune-lora
cbd4e1f 
Signed-off-by: vineet <vineet.suryan@collabora.com>
Vulkan: add support for fp32 OUT_PROD op
cde18d5
CPU: add support for fp16_fp32 OUT_PROD op
b6b30de
Vulkan: add support for f16_f32 OUT_PROD op
5d4d767
@zoq
Vulkan: Add Q4_0/Q8_0 OUT_PROD Vulkan support
e55225a
@makaveli10
vulkan: Add initial cross entropy loss backward shader
c81f689 
Signed-off-by: vineet <vineet.suryan@collabora.com>
@makaveli10
vulkan: Fix cross-entropy-loss-back dispatch size and wg denominator
fc3ae66 
Signed-off-by: vineet <vineet.suryan@collabora.com>
@makaveli10
vulkan: Change uint32 cast to int32 for outprod; allows android compi…
055e30f 
…lation

Signed-off-by: vineet <vineet.suryan@collabora.com>
@makaveli10
vulkan: Deallocate memory after destroying buffer
ae5ae40
@makaveli10
vulkan: Set specialization constants to { 0 } for out_prod
794c37f 
This fixes the vkDeviceLostError on Mali
@makaveli10
vulkan: Set out_prod pipeline disable_robustness to true
36cfb76
@infinitalo infinitalo force-pushed the italo/tether/adreno_q8_inference branch 2 times, most recently from cbea88f to 208747f Compare September 1, 2025 14:13
@infinitalo
Copy link
Author

infinitalo commented Sep 1, 2025 •
edited
Loading

Steps to run the backend-ops test suite:

  1. Set up your Android environment for testing llama.cpp. You can use this comment as a reference if you haven't built it already: Add initial LoRA finetuning support; vulkan OUT_PROD; vulkan cross-entropy-backward #5 (comment)
  2. Configure your build with: cmake -B build -DGGML_VULKAN=1 -DCMAKE_BUILD_TYPE=Debug -DBUILD_TESTING=ON
  3. Build llama.cpp: cmake --build build --config Debug -j2
  4. Run the backend-ops tests: ./build/bin/test-backend-ops
  5. You can also run tests for specific operators with the -o option, for example: ./build/bin/test-backend-ops -o MUL_MAT

This PR has a commit disabling several tests for quantized datatypes that are not currently working properly on Adreno 830.

If you run the test suite as described above with this branch, it should say 2/2 backends passing at the end, with no failing tests on A830, as the attached file shows.

test_adreno_q8_inf2.txt

jpgaribotti pushed a commit that referenced this pull request Sep 10, 2025
@ggerganov @ngxson @slaren
llama : add gpt-oss (ggml-org#15091)
fd1234c 
* oai moe

* compat with new checkpoint

* add attn sink impl

* add rope scaling yarn

* logits match with latest transformers code

* wip chat template

* rm trailing space

* use ggml_scale_bias

* rm redundant is_swa_all

* convert interleaved gate_up

* graph : fix activation function to match reference (#7)

* vocab : handle o200k_harmony special tokens

* ggml : add attention sinks support (#1)

* llama : add attn sinks

* ggml : add attn sinks

* cuda : add attn sinks

* vulkan : add support for sinks in softmax

remove unnecessary return

* ggml : add fused swiglu_oai op (#11)

* ggml : add fused swiglu_oai op

* Update ggml/src/ggml-cpu/ops.cpp

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

* update CUDA impl

* cont : metal impl

* add vulkan impl

* test-backend-ops : more test cases, clean up

* llama : remove unfused impl

* remove extra lines

---------

Co-authored-by: Georgi Gerganov <ggerganov@gmail.com>

---------

Co-authored-by: slaren <slarengh@gmail.com>

* repack mxfp4 upon conversion

* clean up a bit

* enable thinking

* add quick hack to render only some special tokens

* fix bf16 conversion

* remove vocab hack

* webui ok

* support chat parsing for gpt-oss

* fix webui

* direct mapping mxfp4, FINALLY

* force using mxfp4

* properly use lazy tensor

* ggml : add mxfp4

ggml : use e8m0 conversion instead of powf

Co-authored-by: Diego Devesa <slarengh@gmail.com>

change kvalues_mxfp4 table to match e2m1 (#6)

metal : remove quantization for now (not used)

cuda : fix disabled CUDA graphs due to ffn moe bias

vulkan : add support for mxfp4

cont : add cm2 dequant

* ggml : add ggml_add_id (#13)

* ggml : add ggml_add_id

* add cuda impl

* llama : add weight support check for add_id

* perf opt

* add vulkan impl

* rename cuda files

* add metal impl

* allow in-place ggml_add_id

* llama : keep biases on CPU with --cpu-moe

* llama : fix compile error

ggml-ci

* cuda : add fallback for __nv_cvt_e8m0_to_bf16raw

ggml-ci

* cleanup

ggml-ci

* sycl : fix supports_op for MXFP4

ggml-ci

* fix Unknown reasoning format

* ggml-cpu : fix AVX build

ggml-ci

* fix hip build

ggml-ci

* cuda : add mxfp4 dequantization support for cuBLAS

ggml-ci

* ggml-cpu : fix mxfp4 fallback definitions for some architectures

ggml-ci

* cuda : fix version required for __nv_cvt_e8m0_to_bf16raw

---------

Co-authored-by: Xuan Son Nguyen <son@huggingface.co>
Co-authored-by: slaren <slarengh@gmail.com>
Italo Nicola and others added 2 commits September 24, 2025 11:07
wip vulkan crash fix
741f459
@0cc4m
Vulkan: Fix minor debug mode issues (ggml-org#14899)
c893d44 
* vulkan: fix debug mode issues

* vulkan: remove broken check_results GGML_OP_SET_ROWS support
@infinitalo
Copy link
Author

In this current version, the environment variable GGML_TILING_ENABLE=1 should be used when training. Wee'll remove it when cleaning up.

Italo Nicola added 13 commits September 26, 2025 09:05
@infinitalo infinitalo force-pushed the italo/tether/adreno_q8_inference branch from 1049722 to d12255c Compare September 29, 2025 15:59
@infinitalo infinitalo force-pushed the italo/tether/adreno_q8_inference branch from d12255c to c5b7162 Compare September 30, 2025 17:05
@infinitalo
Copy link
Author

Closing this, see #34 for the new version

@infinitalo infinitalo closed this Oct 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

examples ggml Nvidia GPU testing Vulkan

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@infinitalo @makaveli10 @zoq @0cc4m