Eval bug: ggml_cuda_compute_forward: MUL_MAT failed (ROCM 6.3.1)

[itterative]

Name and Version

$ llama-cli --version
ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
version: 6795 (ee09828)
built with cc (GCC) 15.2.1 20250808 (Red Hat 15.2.1-1) for x86_64-redhat-linux

Operating systems

Linux

GGML backends

HIP

Hardware

Ryzen 7 5700X + Radeon RX 7900 XTX

Models

OpenGVLab_InternVL3_5-30B-A3B-Q5_K_M.gguf

Problem description & steps to reproduce

I tried using the latest master commit since I was on a older version, but models do not load anymore.

Here are my build commands:

HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
  cmake -S . -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1100 -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/user/.local/

cmake --build build --config Release -- -j 8

CMAKE_INSTALL_PREFIX=/home/user/.local/ cmake --build build --config Release --target install

Here is the command I use to run, with the error log:

$ llama-server --host 127.0.0.1 --port 5001 -c 8096 --model OpenGVLab_InternVL3_5-30B-A3B-Q5_K_M.gguf --mmproj mmproj-OpenGVLab_InternVL3_5-30B-A3B-bf16.gguf

The issue was introduced in ee09828. I checked the previous commit (e56abd2) and that one loads the model correctly.

First Bad Commit

ee09828

Relevant log output

ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 1 ROCm devices:
  Device 0: AMD Radeon RX 7900 XTX, gfx1100 (0x1100), VMM: no, Wave Size: 32
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 7900 XTX (RADV NAVI31) (radv) | uma: 0 | fp16: 1 | bf16: 0 | warp size: 64 | shared memory: 65536 | int dot: 1 | matrix cores: KHR_coopmat
build: 6795 (ee09828cb) with cc (GCC) 15.2.1 20250808 (Red Hat 15.2.1-1) for x86_64-redhat-linux
system info: n_threads = 8, n_threads_batch = 8, total_threads = 16

system_info: n_threads = 8 (n_threads_batch = 8) / 16 | ROCm : NO_VMM = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | LLAMAFILE = 1 | OPENMP = 1 | REPACK = 1 | 

main: binding port with default address family
main: HTTP server is listening, hostname: 127.0.0.1, port: 5001, http threads: 15
main: loading model
srv    load_model: loading model 'OpenGVLab_InternVL3_5-30B-A3B-Q5_K_M.gguf'
llama_model_load_from_file_impl: skipping device Vulkan0 (AMD Radeon RX 7900 XTX (RADV NAVI31)) with id 0000:2d:00.0 - already using device ROCm0 (AMD Radeon RX 7900 XTX) with the same id
llama_model_load_from_file_impl: using device ROCm0 (AMD Radeon RX 7900 XTX) (0000:2d:00.0) - 24524 MiB free
llama_model_loader: loaded meta data with 46 key-value pairs and 579 tensors from OpenGVLab_InternVL3_5-30B-A3B-Q5_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3moe
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = InternVL3_5 30B A3B
llama_model_loader: - kv   3:                           general.basename str              = InternVL3_5
llama_model_loader: - kv   4:                         general.size_label str              = 30B-A3B
llama_model_loader: - kv   5:                            general.license str              = apache-2.0
llama_model_loader: - kv   6:                   general.base_model.count u32              = 1
llama_model_loader: - kv   7:                  general.base_model.0.name str              = InternVL3_5 30B A3B MPO
llama_model_loader: - kv   8:          general.base_model.0.organization str              = OpenGVLab
llama_model_loader: - kv   9:              general.base_model.0.repo_url str              = https://huggingface.co/OpenGVLab/Inte...
llama_model_loader: - kv  10:                      general.dataset.count u32              = 1
llama_model_loader: - kv  11:                     general.dataset.0.name str              = MMPR v1.2
llama_model_loader: - kv  12:                  general.dataset.0.version str              = v1.2
llama_model_loader: - kv  13:             general.dataset.0.organization str              = OpenGVLab
llama_model_loader: - kv  14:                 general.dataset.0.repo_url str              = https://huggingface.co/OpenGVLab/MMPR...
llama_model_loader: - kv  15:                               general.tags arr[str,3]       = ["internvl", "custom_code", "image-te...
llama_model_loader: - kv  16:                          general.languages arr[str,1]       = ["multilingual"]
llama_model_loader: - kv  17:                       qwen3moe.block_count u32              = 48
llama_model_loader: - kv  18:                    qwen3moe.context_length u32              = 40960
llama_model_loader: - kv  19:                  qwen3moe.embedding_length u32              = 2048
llama_model_loader: - kv  20:               qwen3moe.feed_forward_length u32              = 6144
llama_model_loader: - kv  21:              qwen3moe.attention.head_count u32              = 32
llama_model_loader: - kv  22:           qwen3moe.attention.head_count_kv u32              = 4
llama_model_loader: - kv  23:                    qwen3moe.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  24:  qwen3moe.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  25:                 qwen3moe.expert_used_count u32              = 8
llama_model_loader: - kv  26:              qwen3moe.attention.key_length u32              = 128
llama_model_loader: - kv  27:            qwen3moe.attention.value_length u32              = 128
llama_model_loader: - kv  28:                      qwen3moe.expert_count u32              = 128
llama_model_loader: - kv  29:        qwen3moe.expert_feed_forward_length u32              = 768
llama_model_loader: - kv  30:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  31:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  32:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  33:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  34:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  35:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  36:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  37:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  38:               tokenizer.ggml.add_eos_token bool             = false
llama_model_loader: - kv  39:                    tokenizer.chat_template str              = {% for message in messages %}{{'<|im_...
llama_model_loader: - kv  40:               general.quantization_version u32              = 2
llama_model_loader: - kv  41:                          general.file_type u32              = 17
llama_model_loader: - kv  42:                      quantize.imatrix.file str              = /models_out/InternVL3_5-30B-A3B-GGUF/...
llama_model_loader: - kv  43:                   quantize.imatrix.dataset str              = /training_dir/calibration_datav5.txt
llama_model_loader: - kv  44:             quantize.imatrix.entries_count u32              = 384
llama_model_loader: - kv  45:              quantize.imatrix.chunks_count u32              = 818
llama_model_loader: - type  f32:  241 tensors
llama_model_loader: - type q8_0:   48 tensors
llama_model_loader: - type q5_K:  241 tensors
llama_model_loader: - type q6_K:   49 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q5_K - Medium
print_info: file size   = 20.25 GiB (5.70 BPW) 
load: printing all EOG tokens:
load:   - 151643 ('<|endoftext|>')
load:   - 151645 ('<|im_end|>')
load:   - 151662 ('<|fim_pad|>')
load:   - 151663 ('<|repo_name|>')
load:   - 151664 ('<|file_sep|>')
load: special tokens cache size = 35
load: token to piece cache size = 0.9311 MB
print_info: arch             = qwen3moe
print_info: vocab_only       = 0
print_info: n_ctx_train      = 40960
print_info: n_embd           = 2048
print_info: n_layer          = 48
print_info: n_head           = 32
print_info: n_head_kv        = 4
print_info: n_rot            = 128
print_info: n_swa            = 0
print_info: is_swa_any       = 0
print_info: n_embd_head_k    = 128
print_info: n_embd_head_v    = 128
print_info: n_gqa            = 8
print_info: n_embd_k_gqa     = 512
print_info: n_embd_v_gqa     = 512
print_info: f_norm_eps       = 0.0e+00
print_info: f_norm_rms_eps   = 1.0e-06
print_info: f_clamp_kqv      = 0.0e+00
print_info: f_max_alibi_bias = 0.0e+00
print_info: f_logit_scale    = 0.0e+00
print_info: f_attn_scale     = 0.0e+00
print_info: n_ff             = 6144
print_info: n_expert         = 128
print_info: n_expert_used    = 8
print_info: causal attn      = 1
print_info: pooling type     = 0
print_info: rope type        = 2
print_info: rope scaling     = linear
print_info: freq_base_train  = 1000000.0
print_info: freq_scale_train = 1
print_info: n_ctx_orig_yarn  = 40960
print_info: rope_finetuned   = unknown
print_info: model type       = 30B.A3B
print_info: model params     = 30.53 B
print_info: general.name     = InternVL3_5 30B A3B
print_info: n_ff_exp         = 768
print_info: vocab type       = BPE
print_info: n_vocab          = 151936
print_info: n_merges         = 151387
print_info: BOS token        = 11 ','
print_info: EOS token        = 151645 '<|im_end|>'
print_info: EOT token        = 151645 '<|im_end|>'
print_info: PAD token        = 151643 '<|endoftext|>'
print_info: LF token         = 198 'Ċ'
print_info: FIM PRE token    = 151659 '<|fim_prefix|>'
print_info: FIM SUF token    = 151661 '<|fim_suffix|>'
print_info: FIM MID token    = 151660 '<|fim_middle|>'
print_info: FIM PAD token    = 151662 '<|fim_pad|>'
print_info: FIM REP token    = 151663 '<|repo_name|>'
print_info: FIM SEP token    = 151664 '<|file_sep|>'
print_info: EOG token        = 151643 '<|endoftext|>'
print_info: EOG token        = 151645 '<|im_end|>'
print_info: EOG token        = 151662 '<|fim_pad|>'
print_info: EOG token        = 151663 '<|repo_name|>'
print_info: EOG token        = 151664 '<|file_sep|>'
print_info: max token length = 256
load_tensors: loading model tensors, this can take a while... (mmap = true)
load_tensors: offloading 48 repeating layers to GPU
load_tensors: offloading output layer to GPU
load_tensors: offloaded 49/49 layers to GPU
load_tensors:        ROCm0 model buffer size = 20527.43 MiB
load_tensors:   CPU_Mapped model buffer size =   204.02 MiB
....................................................................................................
llama_init_from_model: model default pooling_type is [0], but [-1] was specified
llama_context: constructing llama_context
llama_context: n_seq_max     = 1
llama_context: n_ctx         = 8096
llama_context: n_ctx_per_seq = 8096
llama_context: n_batch       = 2048
llama_context: n_ubatch      = 512
llama_context: causal_attn   = 1
llama_context: flash_attn    = auto
llama_context: kv_unified    = false
llama_context: freq_base     = 1000000.0
llama_context: freq_scale    = 1
llama_context: n_ctx_per_seq (8096) < n_ctx_train (40960) -- the full capacity of the model will not be utilized
llama_context:  ROCm_Host  output buffer size =     0.58 MiB
llama_kv_cache:      ROCm0 KV buffer size =   768.00 MiB
llama_kv_cache: size =  768.00 MiB (  8192 cells,  48 layers,  1/1 seqs), K (f16):  384.00 MiB, V (f16):  384.00 MiB
llama_context: Flash Attention was auto, set to enabled
llama_context:      ROCm0 compute buffer size =   300.75 MiB
llama_context:  ROCm_Host compute buffer size =    20.01 MiB
llama_context: graph nodes  = 2983
llama_context: graph splits = 2
common_init_from_params: added <|endoftext|> logit bias = -inf
common_init_from_params: added <|im_end|> logit bias = -inf
common_init_from_params: added <|fim_pad|> logit bias = -inf
common_init_from_params: added <|repo_name|> logit bias = -inf
common_init_from_params: added <|file_sep|> logit bias = -inf
common_init_from_params: setting dry_penalty_last_n to ctx_size = 8192
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
ggml_cuda_compute_forward: MUL_MAT failed
ROCm error: invalid device function
  current device: 0, in function ggml_cuda_compute_forward at /home/user/Repos/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:2540
  err
/home/user/Repos/llama.cpp/ggml/src/ggml-cuda/ggml-cuda.cu:88: ROCm error
/home/user/.local/lib64/libggml-base.so(+0x3565) [0x7fe826e48565]
/home/user/.local/lib64/libggml-base.so(ggml_print_backtrace+0x1eb) [0x7fe826e4892b]
/home/user/.local/lib64/libggml-base.so(ggml_abort+0x11f) [0x7fe826e48aaf]
/home/user/.local/lib64/libggml-hip.so(+0x195ed2) [0x7fe824595ed2]
/home/user/.local/lib64/libggml-hip.so(+0x19cd39) [0x7fe82459cd39]
/home/user/.local/lib64/libggml-base.so(ggml_backend_sched_graph_compute_async+0x7f3) [0x7fe826e62383]
/home/user/.local/lib64/libllama.so(_ZN13llama_context13graph_computeEP11ggml_cgraphb+0xa0) [0x7fe826c370d0]
/home/user/.local/lib64/libllama.so(_ZN13llama_context14process_ubatchERK12llama_ubatch14llm_graph_typeP22llama_memory_context_iR11ggml_status+0xe2) [0x7fe826c38d42]
/home/user/.local/lib64/libllama.so(_ZN13llama_context6decodeERK11llama_batch+0x3af) [0x7fe826c3da8f]
/home/user/.local/lib64/libllama.so(llama_decode+0xe) [0x7fe826c3e8ce]
llama-server() [0x5c9058]
llama-server() [0x4a2fd9]
llama-server() [0x439e09]
/lib64/libc.so.6(+0x3575) [0x7fe821011575]
/lib64/libc.so.6(__libc_start_main+0x88) [0x7fe821011628]
llama-server() [0x43bc25]
Aborted (core dumped)

[itterative]

After checking the commit with the issue, it appears that the core issue is rather that the cmake flag has been changed (AMDGPU_TARGETS becomes GPU_TARGETS). So either changing the build documentation or reverting the flag change should suffice.

Using these commands work as expected:

HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
  cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1100 -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/user/.local/

cmake --build build --config Release -- -j 8

CMAKE_INSTALL_PREFIX=/home/user/.local/ cmake --build build --config Release --target install

[am17an]

cc: @JohannesGaessler

[IMbackK]

The flag cant be changed back, as it has been changed by upstream not us, we are just following the change. The documentation needs to be updated. -DGPU_TARGETS probably generally should not be set by users at all as when it is not set the hip cmake implementation will simply build for all targets required for the current system, which is the correct thing to do for local builds.

We could also error out when AMDGPU_TARGETS is set to make it clear that its no longer supported.

[JohannesGaessler]

I was unable to reproduce the issue. Can you check whether #16803 works correctly with AMDGPU_TARGETS? The intended behavior is that the option should still work but print a deprecation warning.

[itterative]

The flag cant be changed back, as it has been changed by upstream not us, we are just following the change. The documentation needs to be updated. -DGPU_TARGETS probably generally should not be set by users at all as when it is not set the hip cmake implementation will simply build for all targets required for the current system, which is the correct thing to do for local builds.

We could also error out when AMDGPU_TARGETS is set to make it clear that its no longer supported.

@IMbackK I tried running the same command without the targets specified in the command and that does work. I cannot see which target it uses in the cmake logs, but it seems like it reuses the previous ccache build, so I assume it still targets gfx1100 for me.

I was unable to reproduce the issue. Can you check whether #16803 works correctly with AMDGPU_TARGETS? The intended behavior is that the option should still work but print a deprecation warning.

@JohannesGaessler I checked out your branch and it does work now, although I do not see any warnings when running the first cmake command.

$ HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
  cmake -S . -B build -DGGML_HIP=ON -DAMDGPU_TARGETS=gfx1100 -DCMAKE_BUILD_TYPE=Release -DCMAKE_INSTALL_PREFIX=/home/user/.local/
CMAKE_BUILD_TYPE=Release
-- ccache found, compilation results will be cached. Disable with GGML_CCACHE=OFF.
-- CMAKE_SYSTEM_PROCESSOR: x86_64
-- GGML_SYSTEM_ARCH: x86
-- Including CPU backend
-- x86 detected
-- Adding CPU backend variant ggml-cpu: -march=native 
-- HIP and hipBLAS found
-- Including HIP backend
-- Vulkan found
-- GL_KHR_cooperative_matrix supported by glslc
-- GL_NV_cooperative_matrix2 supported by glslc
-- GL_EXT_integer_dot_product supported by glslc
-- GL_EXT_bfloat16 supported by glslc
-- Including Vulkan backend
-- ggml version: 0.9.4
-- ggml commit:  df23f75ef
-- Configuring done (0.9s)
-- Generating done (0.3s)
-- Build files have been written to: /home/user/Repos/llama.cpp/build

In any case, the PR does fix the issue for me. Thank you very much!

[IMbackK]

The warning only occurs when you use the cmake integration from rocm 7 on rocm 6.3 there will be no warning, this is correct behavior.

[itterative]

One unrelated question regarding the original issue: I noticed I could not build without having vulkan development packages installed. Is this intended? It appears that ggml-vulkan marks it as required, but I was wondering if that is required for hip builds.