bug: ValueError: Architecture qwen3 not supported

[JohnConnor123]

Name and Version

(venv) calibri@devtest:~/experiments/quantization_benchmark$ ./llama.cpp/build/bin/llama-cli
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
  Device 1: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes
build: 5215 (5f5e39e1) with cc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0 for x86_64-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_model_load_from_file_impl: using device CUDA0 (NVIDIA GeForce RTX 3090) - 23872 MiB free
llama_model_load_from_file_impl: using device CUDA1 (NVIDIA GeForce RTX 3090) - 23870 MiB free
gguf_init_from_file: failed to open GGUF file 'models/7B/ggml-model-f16.gguf'
llama_model_load: error loading model: llama_model_loader: failed to load model from models/7B/ggml-model-f16.gguf

llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model 'models/7B/ggml-model-f16.gguf'
main: error: unable to load model

Operating systems

Linux

GGML backends

CUDA

Hardware

AMD Ryzen 7 5800X 8-Core Processor + 2x rtx 3090

Models

Qwen/Qwen3-0.6B

Problem description & steps to reproduce

I used wrapper upon llama.cpp to get qwen3 quantizations (https://github.com/JohnConnor123/quantization-benchmark), but got error. But I found that there are GGUF for this model in hugging face hub

First Bad Commit

No response

Relevant log output

Traceback (most recent call last):
  File "/home/calibri/experiments/quantization_benchmark/quantization_benchmark.py", line 340, in <module>
    results_df = compare_quantizations(
  File "/home/calibri/experiments/quantization_benchmark/quantization_benchmark.py", line 285, in compare_quantizations
    result = evaluate_quantization(
  File "/home/calibri/experiments/quantization_benchmark/quantization_benchmark.py", line 225, in evaluate_quantization
    model, tokenizer = load_model_and_tokenizer(quantized_path)
  File "/home/calibri/experiments/quantization_benchmark/quantization_benchmark.py", line 199, in load_model_and_tokenizer
    model = AutoModelForCausalLM.from_pretrained(model_dir, gguf_file=gguf_file, **common_kwargs)
  File "/home/calibri/experiments/quantization_benchmark/venv/lib/python3.10/site-packages/transformers/models/auto/auto_factory.py", line 526, in from_pretrained
    config, kwargs = AutoConfig.from_pretrained(
  File "/home/calibri/experiments/quantization_benchmark/venv/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 1021, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/calibri/experiments/quantization_benchmark/venv/lib/python3.10/site-packages/transformers/configuration_utils.py", line 590, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/home/calibri/experiments/quantization_benchmark/venv/lib/python3.10/site-packages/transformers/configuration_utils.py", line 681, in _get_config_dict
    config_dict = load_gguf_checkpoint(resolved_config_file, return_tensors=False)["config"]
  File "/home/calibri/experiments/quantization_benchmark/venv/lib/python3.10/site-packages/transformers/modeling_gguf_pytorch_utils.py", line 326, in load_gguf_checkpoint
    raise ValueError(f"Architecture {architecture + model_size} not supported")
ValueError: Architecture qwen3 not supported

[the-veloper]

#12828

looks like support is there, since 3 weeks ago, we just need to upgrade to a more recent version.

yes. i just tested with a version from 2 weeks ago and it works fine. please try that on your own and report if you have any problems

[gercamjr]

I tried pulling the latest changes from the master branch, building, and running llama-server with qwen3 model and I get the same error:

llama-server -m Qwen3-14B-Q8_0.gguf --host 0.0.0.0 --port 8080 --flash-attn --mlock -ngl 99
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = AMD Radeon RX 6900 XT (AMD proprietary driver) | uma: 0 | fp16: 1 | warp size: 64 | matrix cores: none
build: 4591 (7919256c) with cc.exe (Rev2, Built by MSYS2 project) 14.2.0 for x86_64-w64-mingw32
system info: n_threads = 12, n_threads_batch = 12, total_threads = 24

system_info: n_threads = 12 (n_threads_batch = 12) / 24 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

main: HTTP server is listening, hostname: 0.0.0.0, port: 8080, http threads: 23
main: loading model
srv    load_model: loading model 'Qwen3-14B-Q8_0.gguf'
llama_model_load_from_file_impl: using device Vulkan0 (AMD Radeon RX 6900 XT) - 16112 MiB free
llama_model_loader: loaded meta data with 32 key-value pairs and 443 tensors from Qwen3-14B-Q8_0.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen3-14B
llama_model_loader: - kv   3:                           general.basename str              = Qwen3-14B
llama_model_loader: - kv   4:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   5:                         general.size_label str              = 14B
llama_model_loader: - kv   6:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv   7:                          qwen3.block_count u32              = 40
llama_model_loader: - kv   8:                       qwen3.context_length u32              = 40960
llama_model_loader: - kv   9:                     qwen3.embedding_length u32              = 5120
llama_model_loader: - kv  10:                  qwen3.feed_forward_length u32              = 17408
llama_model_loader: - kv  11:                 qwen3.attention.head_count u32              = 40
llama_model_loader: - kv  12:              qwen3.attention.head_count_kv u32              = 8
llama_model_loader: - kv  13:                       qwen3.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  14:     qwen3.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  15:                 qwen3.attention.key_length u32              = 128
llama_model_loader: - kv  16:               qwen3.attention.value_length u32              = 128
llama_model_loader: - kv  17:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  18:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  19:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  20:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  21:                      tokenizer.ggml.merges arr[str,151387]  = ["─á ─á", "─á─á ─á─á", "i n", "─á t",...
llama_model_loader: - kv  22:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  23:            tokenizer.ggml.padding_token_id u32              = 151654
llama_model_loader: - kv  24:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  25:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  26:               general.quantization_version u32              = 2
llama_model_loader: - kv  27:                          general.file_type u32              = 7
llama_model_loader: - kv  28:                      quantize.imatrix.file str              = Qwen3-14B-GGUF/imatrix_unsloth.dat
llama_model_loader: - kv  29:                   quantize.imatrix.dataset str              = unsloth_calibration_Qwen3-14B.txt
llama_model_loader: - kv  30:             quantize.imatrix.entries_count i32              = 280
llama_model_loader: - kv  31:              quantize.imatrix.chunks_count i32              = 32
llama_model_loader: - type  f32:  161 tensors
llama_model_loader: - type q8_0:  282 tensors
print_info: file format = GGUF V3 (latest)
print_info: file type   = Q8_0
print_info: file size   = 14.61 GiB (8.50 BPW)
llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen3'
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model 'Qwen3-14B-Q8_0.gguf'
srv    load_model: failed to load model, 'Qwen3-14B-Q8_0.gguf'
main: exiting due to model loading error

[JohnConnor123]

#12828

looks like support is there, since 3 weeks ago, we just need to upgrade to a more recent version.

yes. i just tested with a version from 2 weeks ago and it works fine. please try that on your own and report if you have any problems

A already did new git pull before posting issue

[JohnConnor123]

@ggerganov

[dmoskovenko]

I get the same error on apple m1

./llama-server -m "Qwen3-0.6B-Q5_K_M.gguf" -t 8 --mlock -fa --temp 0.7 --repeat-penalty 1.10 --repeat-last-n 1 --top-k 20 --top-p 0.80 --min-p 0 -c 32768
build: 4064 (4b3a9212) with Apple clang version 15.0.0 (clang-1500.3.9.4) for arm64-apple-darwin23.6.0
system info: n_threads = 8, n_threads_batch = 8, total_threads = 8

system_info: n_threads = 8 (n_threads_batch = 8) / 8 | AVX = 0 | AVX_VNNI = 0 | AVX2 = 0 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | AVX512_BF16 = 0 | AMX_INT8 = 0 | FMA = 0 | NEON = 1 | SVE = 0 | ARM_FMA = 1 | F16C = 0 | FP16_VA = 1 | RISCV_VECT = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 0 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LLAMAFILE = 1 |

main: HTTP server is listening, hostname: 127.0.0.1, port: 8080, http threads: 7
main: loading model
llama_load_model_from_file: using device Metal (Apple M1) - 10922 MiB free
llama_model_loader: loaded meta data with 32 key-value pairs and 310 tensors from ./models/Qwen3-0.6B-Q5_K_M.gguf (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen3
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen3-0.6B
llama_model_loader: - kv   3:                           general.basename str              = Qwen3-0.6B
llama_model_loader: - kv   4:                       general.quantized_by str              = Unsloth
llama_model_loader: - kv   5:                         general.size_label str              = 0.6B
llama_model_loader: - kv   6:                           general.repo_url str              = https://huggingface.co/unsloth
llama_model_loader: - kv   7:                          qwen3.block_count u32              = 28
llama_model_loader: - kv   8:                       qwen3.context_length u32              = 40960
llama_model_loader: - kv   9:                     qwen3.embedding_length u32              = 1024
llama_model_loader: - kv  10:                  qwen3.feed_forward_length u32              = 3072
llama_model_loader: - kv  11:                 qwen3.attention.head_count u32              = 16
llama_model_loader: - kv  12:              qwen3.attention.head_count_kv u32              = 8
llama_model_loader: - kv  13:                       qwen3.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  14:     qwen3.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  15:                 qwen3.attention.key_length u32              = 128
llama_model_loader: - kv  16:               qwen3.attention.value_length u32              = 128
llama_model_loader: - kv  17:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  18:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  19:                      tokenizer.ggml.tokens arr[str,151936]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  20:                  tokenizer.ggml.token_type arr[i32,151936]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  21:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  22:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  23:            tokenizer.ggml.padding_token_id u32              = 151654
llama_model_loader: - kv  24:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  25:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  26:               general.quantization_version u32              = 2
llama_model_loader: - kv  27:                          general.file_type u32              = 17
llama_model_loader: - kv  28:                      quantize.imatrix.file str              = Qwen3-0.6B-GGUF/imatrix_unsloth.dat
llama_model_loader: - kv  29:                   quantize.imatrix.dataset str              = unsloth_calibration_Qwen3-0.6B.txt
llama_model_loader: - kv  30:             quantize.imatrix.entries_count i32              = 196
llama_model_loader: - kv  31:              quantize.imatrix.chunks_count i32              = 32
llama_model_loader: - type  f32:  113 tensors
llama_model_loader: - type q5_K:  168 tensors
llama_model_loader: - type q6_K:   29 tensors
llama_model_load: error loading model: error loading model architecture: unknown model architecture: 'qwen3'
llama_load_model_from_file: failed to load model
common_init_from_params: failed to load model './models/Qwen3-0.6B-Q5_K_M.gguf'
srv    load_model: failed to load model, './models/Qwen3-0.6B-Q5_K_M.gguf'
main: exiting due to model loading error

[ggerganov]

Your versions are very old:

• build: 4591 (7919256) - this is from January
• build: 4064 (4b3a921) - this is from last year

Update to latest version.

[JohnConnor123]

Your versions are very old:

• build: 4591 (7919256) - this is from January
• build: 4064 (4b3a921) - this is from last year

Update to latest version.

why does it install an old version when using git clone ? I reinstalled the repository twice the other day before opening this issue :(

[CISC]

You are most likely running some old build in your path/venv instead of the new one...

[JohnConnor123]

You are most likely running some old build in your path/venv instead of the new one...

But I didn't do export command. And before git clone I deleted current folder llama.cpp, and after downloading I recompiled llama.cpp.

Maybe I can do some commands and attach their output to help you understand reason of cloning old version?

[gercamjr]

I downloaded the latest prebuilt binaries for Windows + Vulkan support, and I run into the same error when trying to load Qwen3 in llama-server **unknown model architecture: 'qwen3' **

[JohnConnor123]

Your versions are very old:

• build: 4591 (7919256) - this is from January
• build: 4064 (4b3a921) - this is from last year

Update to latest version.

@ggerganov My version is build: 5215 (5f5e39e1). Why I got error?

[reneleonhardt]

That version is a week "old", i.e. Qwen3 model type sizes came later: #13175
Maybe you could try a more current build?

[JohnConnor123]

That version is a week "old", i.e. Qwen3 model type sizes came later: #13175
Maybe you could try a more current build?

Ok, i'll try. But wasn't qwen3 support in llama.cpp merged more than a month ago?