NPU基于Qwen2-VL-7B执行多卡QLoRA时卡住

[tbozhong]

Reminder

• I have read the above rules and searched the existing issues.

System Info

• llamafactory version: 0.9.1.dev0
• Platform: Linux-5.4.241-1-tlinux4-0017.7-x86_64-with-glibc2.28
• Python version: 3.9.16
• PyTorch version: 2.1.0+cpu (NPU)
• Transformers version: 4.45.2
• Datasets version: 2.21.0
• Accelerate version: 0.34.2
• PEFT version: 0.12.0
• TRL version: 0.9.6
• NPU type: Ascend910B2C
• CANN version: 8.0.RC3
• DeepSpeed version: 0.13.1+1d35db76
• Bitsandbytes version: 0.45.0.dev+7e6f865

Reproduction

单卡QLoRA运行成功，只有多卡时遇到问题。单、多卡LoRA同样可以运行成功。
卡住界面如下，随后会爆出通信超时的错误：

Others

期待您的回复~

[hiyouga]

cannot reproduce