Fix GGUF loader for Qwen3 MoE.

[Gh0u1L5]

Purpose

Despite upstream repositories (gguf-py, transformers) having added support for Qwen3 MoE GGUF quantization, vLLM GGUF loading is still broken.
This PR aims to fix the GGUF loader and a mismatch in the Qwen3 MoE model's embed_tokens layer.

Test Plan

wget https://huggingface.co/unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF/resolve/main/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf
vllm serve ./Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf

Test Result

Successfully loaded the GGUF model.

References

• [Feature]: Qwen3 Models GGUF Support #21511
• Support loading Qwen3 MoE GGUF huggingface/transformers#39638
• Fix Qwen3 MoE GGUF architecture mismatch huggingface/transformers#39976

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[gemini-code-assist]

Code Review

This pull request correctly adds support for Qwen2 and Qwen3 MoE models in the GGUF loader and fixes an embed_tokens layer initialization issue. The changes appear to be correct and well-tested. My main feedback is regarding code duplication in the GGUF loader, which could be refactored to improve long-term maintainability.

[gemini-code-assist]

This block for handling Qwen MoE models introduces significant code duplication with the existing logic for DeepSeek MoE models just above. While functionally correct, this pattern makes the code harder to maintain. If a change is needed for expert weight mapping, it would have to be applied in multiple places. Please consider refactoring to share the common logic between these model types.

[Isotr0py]

Thanks!

[yanmindi]

请问你用的版本是什么 我们试过都没法正常加载

[Isotr0py]

请问你用的版本是什么 我们试过都没法正常加载

Try using the nightly wheel, this PR hasn't been included in newest release.

pip install -U vllm \
    --pre \
    --extra-index-url https://wheels.vllm.ai/nightly

[adonishong]

请问你用的版本是什么 我们试过都没法正常加载

Try using the nightly wheel, this PR hasn't been included in newest release.

pip install -U vllm \
    --pre \
    --extra-index-url https://wheels.vllm.ai/nightly

we were using latest main branch code, pip install -e .; just now, I tried, "pip remove vllm", then "pip install -U vllm
--pre
--extra-index-url https://wheels.vllm.ai/nightly"

still no luck in load qwen3 mode gguf, both 30B version and 480B version

both are terminated in this error
"vllm/transformers_utils/config.py", line 623, in get_sentence_transformer_tokenizer_config
if not encoder_dict and not model.startswith("/"):
^^^^^^^^^^^^^^^^
AttributeError: 'PosixPath' object has no attribute 'startswith'"

and this error info could be observed at very beginning

"ERROR 08-18 05:01:32 [config.py:133] Error retrieving safetensors: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/dev/shm/Qwen3-30B-A3B-Q4_K_M.gguf'. Use repo_type argument if needed., retrying 1 of 2
ERROR 08-18 05:01:34 [config.py:131] Error retrieving safetensors: Repo id must be in the form 'repo_name' or 'namespace/repo_name': '/dev/shm/Qwen3-30B-A3B-Q4_K_M.gguf'. Use repo_type argument if needed."

[Isotr0py]

we were using latest main branch code

AttributeError: 'PosixPath' object has no attribute 'startswith'"

Perhaps you forgot to pull the main branch? This bug has been fixed by #21579 three weeks ago...

[yanmindi]

Purpose

Despite upstream repositories (gguf-py, transformers) having added support for Qwen3 MoE GGUF quantization, vLLM GGUF loading is still broken. This PR aims to fix the GGUF loader and a mismatch in the Qwen3 MoE model's embed_tokens layer.

Test Plan

wget https://huggingface.co/unsloth/Qwen3-30B-A3B-Instruct-2507-GGUF/resolve/main/Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf
vllm serve ./Qwen3-30B-A3B-Instruct-2507-Q4_K_M.gguf

Test Result

Successfully loaded the GGUF model.

References

• [Feature]: Qwen3 Models GGUF Support #21511
• Support loading Qwen3 MoE GGUF huggingface/transformers#39638
• Fix Qwen3 MoE GGUF architecture mismatch huggingface/transformers#39976

再请教个问题 我的也运行起来 gguf 格式了 但是推理的时候 报错乱码是什么原因？ 你得推理输出正常吗？ 换了好多版本的gguf还是不行

[yanmindi]

就是输出的内容是乱码 一些无意义的东西 或者就是不停的输出 iiiiiiiiiiiiiii 服务器端没报错退出