[Bug]: MOE模型，2卡推理，报错AssertionError("Invalid device id")

[Elissa0723]

Your current environment

PyTorch version: 2.1.2+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.2 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.29.5
Libc version: glibc-2.35

Python version: 3.10.13 (main, Sep 11 2023, 13:44:35) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-3.10.0-1160.el7.x86_64-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A800-SXM4-80GB
GPU 1: NVIDIA A800-SXM4-80GB
GPU 2: NVIDIA A800-SXM4-80GB
GPU 3: NVIDIA A800-SXM4-80GB

Nvidia driver version: 470.199.02
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
架构： x86_64
CPU 运行模式： 32-bit, 64-bit
Address sizes: 46 bits physical, 57 bits virtual
字节序： Little Endian
CPU: 128
在线 CPU 列表： 0-127
厂商 ID： GenuineIntel
型号名称： Intel(R) Xeon(R) Platinum 8369B CPU @ 2.90GHz
CPU 系列： 6
型号： 106
每个核的线程数： 2
每个座的核数： 32
座： 2
步进： 6
CPU 最大 MHz： 3500.0000
CPU 最小 MHz： 800.0000
BogoMIPS： 5800.00
标记： fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb cat_l3 invpcid_single intel_pt ssbd mba ibrs ibpb stibp ibrs_enhanced tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local dtherm ida arat pln pts hwp hwp_act_window hwp_epp hwp_pkg_req avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq md_clear pconfig spec_ctrl intel_stibp flush_l1d arch_capabilities
虚拟化： VT-x
L1d 缓存： 3 MiB (64 instances)
L1i 缓存： 2 MiB (64 instances)
L2 缓存： 80 MiB (64 instances)
L3 缓存： 96 MiB (2 instances)
NUMA 节点： 2
NUMA 节点0 CPU： 0-31,64-95
NUMA 节点1 CPU： 32-63,96-127
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; Load fences, usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected

🐛 Describe the bug

在对Qwen2-57B-A14B-Instruct进行两卡推理的时候，运行如下指令：

os.environ['CUDA_VISIBLE_DEVICES'] = '0,1'
model_dir = '/publicdata/huggingface.co/Qwen/Qwen2-57B-A14B-Instruct'
model_type = ModelType.qwen2_57b_a14b_instruct

template_type = get_default_template_type(model_type)
llm_engine = get_vllm_engine(model_type, model_id_or_path=model_dir, tensor_parallel_size=2)
tokenizer = llm_engine.hf_tokenizer
template = get_template(template_type, tokenizer)

报错如下：
(RayWorkerVllm pid=2689517) INFO 06-14 11:52:26 selector.py:16] Using FlashAttention backend.
(RayWorkerVllm pid=2689517) ERROR 06-14 11:52:26 ray_utils.py:44] Error executing method init_device. This might cause deadlock in distributed execution.
(RayWorkerVllm pid=2689517) ERROR 06-14 11:52:26 ray_utils.py:44] Traceback (most recent call last):
(RayWorkerVllm pid=2689517) ERROR 06-14 11:52:26 ray_utils.py:44] File "/opt/conda/lib/python3.10/site-packages/vllm/engine/ray_utils.py", line 37, in execute_method
(RayWorkerVllm pid=2689517) ERROR 06-14 11:52:26 ray_utils.py:44] return executor(*args, **kwargs)
(RayWorkerVllm pid=2689517) ERROR 06-14 11:52:26 ray_utils.py:44] File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 93, in init_device
(RayWorkerVllm pid=2689517) ERROR 06-14 11:52:26 ray_utils.py:44] _check_if_gpu_supports_dtype(self.model_config.dtype)
(RayWorkerVllm pid=2689517) ERROR 06-14 11:52:26 ray_utils.py:44] File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 309, in _check_if_gpu_supports_dtype
(RayWorkerVllm pid=2689517) ERROR 06-14 11:52:26 ray_utils.py:44] compute_capability = torch.cuda.get_device_capability()
(RayWorkerVllm pid=2689517) ERROR 06-14 11:52:26 ray_utils.py:44] File "/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py", line 435, in get_device_capability
(RayWorkerVllm pid=2689517) ERROR 06-14 11:52:26 ray_utils.py:44] prop = get_device_properties(device)
(RayWorkerVllm pid=2689517) ERROR 06-14 11:52:26 ray_utils.py:44] File "/opt/conda/lib/python3.10/site-packages/torch/cuda/init.py", line 452, in get_device_properties
(RayWorkerVllm pid=2689517) ERROR 06-14 11:52:26 ray_utils.py:44] raise AssertionError("Invalid device id")
(RayWorkerVllm pid=2689517) ERROR 06-14 11:52:26 ray_utils.py:44] AssertionError: Invalid device id

[youkaichao]

can you try #5473 ? it should fix your error i think.

[Elissa0723]

can you try #5473 ? it should fix your error i think.

I pulled the latest vllm code and tried to install it
but it encountered some problems during installation

if I use pip install -e . ,it will get stuck here:

Looking in indexes: https://mirrors.aliyun.com/pypi/simple
Obtaining file:///workspace/huj11%40xiaopeng.com/code/vllm
Installing build dependencies ... |

if I use pip install --editable ./ --no-build-isolation, it will get stuck here:

Looking in indexes: https://mirrors.aliyun.com/pypi/simple
Obtaining file:///workspace/huj11%40xiaopeng.com/code/vllm
Checking if build backend supports build_editable ... done
Preparing editable metadata (pyproject.toml) ... done
Requirement already satisfied: cmake>=3.21 in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (3.29.5)
Requirement already satisfied: ninja in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (1.11.1.1)
Requirement already satisfied: psutil in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (5.9.7)
Requirement already satisfied: sentencepiece in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (0.1.99)
Requirement already satisfied: numpy in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (1.26.3)
Requirement already satisfied: requests in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (2.31.0)
Requirement already satisfied: py-cpuinfo in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (9.0.0)
Requirement already satisfied: transformers>=4.40.0 in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (4.41.2)
Requirement already satisfied: tokenizers>=0.19.1 in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (0.19.1)
Requirement already satisfied: fastapi in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (0.109.0)
Requirement already satisfied: openai in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (1.33.0)
Requirement already satisfied: uvicorn[standard] in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (0.27.0.post1)
Requirement already satisfied: pydantic>=2.0 in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (2.5.3)
Requirement already satisfied: prometheus-client>=0.18.0 in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (0.19.0)
Requirement already satisfied: prometheus-fastapi-instrumentator>=7.0.0 in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (7.0.0)
Requirement already satisfied: tiktoken==0.6.0 in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (0.6.0)
Requirement already satisfied: lm-format-enforcer==0.10.1 in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (0.10.1)
Requirement already satisfied: outlines==0.0.34 in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (0.0.34)
Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (4.9.0)
Requirement already satisfied: filelock>=3.10.4 in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (3.13.1)
Requirement already satisfied: ray>=2.9 in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (2.9.1)
Requirement already satisfied: nvidia-ml-py in /opt/conda/lib/python3.10/site-packages (from vllm==0.4.2) (12.555.43)
Collecting vllm-nccl-cu12<2.19,>=2.18 (from vllm==0.4.2)
Downloading https://mirrors.aliyun.com/pypi/packages/41/07/c1be8f4ffdc257646dda26470b803487150c732aa5c9f532dd789f186a54/vllm_nccl_cu12-2.18.1.0.4.0.tar.gz (6.2 kB)

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!