[Feature]: GLM-4-32B-0414

[aicodex]

🚀 The feature, motivation and pitch

期待支持 THUDM/GLM-4-32B-0414。看起来性能不错，以及THUDM/GLM-Z1-32B-0414.

Alternatives

No response

Additional context

No response

[fengxu-sz]

我把vllm_ascend/ops/rotary_embedding.py中倒数第4行RotaryEmbedding.forward_oot = rope_forward_oot注释掉就可以运行THUDM/GLM-4-32B-0414和THUDM/GLM-Z1-32B-0414了

[Yikun]

@fengxu-sz What's the specific issue raised when using THUDM/GLM-4-32B-0414 without forward_oot patch?

[fengxu-sz]

@fengxu-sz What's the specific issue raised when using THUDM/GLM-4-32B-0414 without forward_oot patch?

vllm的日志显示：
Exception raised from OperationSetup at build/third_party/op-plugin/op_plugin/CMakeFiles/op_plugin_atb.dir/compiler_depend.ts:131 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0xb8 (0xffff9a9ec908 in /xxx/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, char const*) + 0x70 (0xffff9a99b4e0 in /xxx/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #2: atb::OperationSetup(atb::VariantPack, atb::Operation*, atb::Context*) + 0x8c (0xffff102146bc in /xxx/lib/python3.10/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame #3: + 0x34d84 (0xffff10214d84 in /xxx/lib/python3.10/site-packages/torch_npu/lib/libop_plugin_atb.so)
frame #4: + 0x1644484 (0xffff8e8c4484 in /xxx/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #5: + 0x78d244 (0xffff8da0d244 in /xxx/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #6: + 0x78da58 (0xffff8da0da58 in /xxx/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #7: + 0x78a1cc (0xffff8da0a1cc in /xxx/lib/python3.10/site-packages/torch_npu/lib/libtorch_npu.so)
frame #8: + 0x4c9e4c (0xffff9aa29e4c in /xxx/lib/python3.10/site-packages/torch/lib/libc10.so)
frame #9: + 0x7d5c8 (0xffffa57ed5c8 in /lib/aarch64-linux-gnu/libc.so.6)
frame #10: + 0xe5edc (0xffffa5855edc in /lib/aarch64-linux-gnu/libc.so.6)
......
ERROR 04-28 03:37:31 [engine.py:448] RuntimeError: The Inner error is reported as above. The process exits for this inner error, and the current working operator name is RopeOperation.

atb日志显示：
[2025-04-28 03:37:31.869768] [error] [1093226] [rope_operation.cpp:94] RopeOperation_0 Wrong rotaryCoeff: 128
[2025-04-28 03:37:31.881527] [error] [1093226] [operation_base.cpp:513] RopeOperation_0 invalid param, setup check fail, error code: 1

环境：
pytorch_npu: 2.5.1
CANN: 8.1.0RC1.Alpha002

[aicodex]

我把vllm_ascend/ops/rotary_embedding.py中倒数第4行RotaryEmbedding.forward_oot = rod_forward_oot注释掉就可以运行THUDM/GLM-4-32B-0414和THUDM/GLM-Z1-32B-0414了

所以您这个方法，就是可以打个专门为GLM做的镜像对吧，用这个可能跑其他模型就报错了。有没有简单对比过去掉这以后用ascend硬件和nvidia部署的glm的差异呢，会不会有非常大的差异，比方说很容易胡言乱语或者无法结束生成。

[Eviannn]

The rotary dim of glm and llama-like models is different, rotary dim = head_dim // 2 in glm, maybe current npu rope can not support, i guess

[aicodex]

The rotary dim of glm and llama-like models is different, rotary dim = head_dim // 2 in glm, maybe current npu rope can not support, i guess

I have read the code of vllm and vllm-ascend, seem that vllm have a default value: "class RotaryEmbedding(CustomOp):" and in CustomOp class there have a function:

def forward_oot(self, *args, **kwargs):
    # By default, we assume that OOT ops are compatible with the
    # PyTorch-native implementation.
    return self.forward_native(*args, **kwargs)

，but some model have their special RotaryEmbedding, like DeepseekScalingRotaryEmbedding, Qwen2_5_VisionRotaryEmbedding。and in vllm_ascend/ops/rotary_embedding.py, they set the forward_oot function to vllm_ascend's “rope_forward_oot”。I guess this function only for they support models.

Maybe glm can use PyTorch-native implementation and work well by vllm's assume ? Delete RotaryEmbedding.forward_oot = rope_forward_oot in asend_vllm just let the default RotaryEmbedding.forward_oot function to vllm's "CustomOp" class

[aicodex]

can't use on vllm-asend:v0.8.4rc2

[whisper-bye]

GLM-Z1-Rumination-32B +1

[YQYANG2233]

我把vllm_ascend/ops/rotary_embedding.py中倒数第4行RotaryEmbedding.forward_oot = rope_forward_oot注释掉就可以运行THUDM/GLM-4-32B-0414和THUDM/GLM-Z1-32B-0414了

What's your vllm-ascend version that can run GLM-4-0414 through this way. I cannot run it on v0.9.0rc2