Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: Error when using tensor_parallel in v0.6.1.post1 or 0.6.2 #8937

Closed
1 task done
ruleGreen opened this issue Sep 29, 2024 · 21 comments
Closed
1 task done

[Bug]: Error when using tensor_parallel in v0.6.1.post1 or 0.6.2 #8937

ruleGreen opened this issue Sep 29, 2024 · 21 comments
Labels
bug Something isn't working

Comments

@ruleGreen
Copy link

Your current environment

Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 11 (bullseye) (x86_64)
GCC version: (Debian 10.2.1-6) 10.2.1 20210110
Clang version: Could not collect
CMake version: version 3.18.4
Libc version: glibc-2.31

Python version: 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110] (64-bit runtime)
Python platform: Linux-5.4.143.bsk.7-amd64-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A800-SXM4-80GB
GPU 1: NVIDIA A800-SXM4-80GB
GPU 2: NVIDIA A800-SXM4-80GB
GPU 3: NVIDIA A800-SXM4-80GB
GPU 4: NVIDIA A800-SXM4-80GB
GPU 5: NVIDIA A800-SXM4-80GB
GPU 6: NVIDIA A800-SXM4-80GB
GPU 7: NVIDIA A800-SXM4-80GB

Nvidia driver version: 535.161.08
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 57 bits virtual
CPU(s): 120
On-line CPU(s) list: 0-119
Thread(s) per core: 2
Core(s) per socket: 30
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 106
Model name: Intel(R) Xeon(R) Platinum 8336C CPU @ 2.30GHz
Stepping: 6
CPU MHz: 2294.608
BogoMIPS: 4589.21
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 2.8 MiB
L1i cache: 1.9 MiB
L2 cache: 75 MiB
L3 cache: 108 MiB
NUMA node0 CPU(s): 0-59
NUMA node1 CPU(s): 60-119
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves wbnoinvd arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid md_clear arch_capabilities

Versions of relevant libraries:
[pip3] byted-torch==2.1.0.post2
[pip3] byted-torch-monitor==0.0.1
[pip3] numpy==1.26.2
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.6.68
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pyzmq==26.2.0
[pip3] torch==2.4.0
[pip3] torchaudio==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.45.1
[pip3] triton==3.0.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.0@32e7db25365415841ebc7c4215851743fbb1bad1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV8 NV8 NV8 NV8 NV8 NV8 NV8 SYS PIX PIX NODE NODE SYS SYS SYS SYS 0-59 0N/A
GPU1 NV8 X NV8 NV8 NV8 NV8 NV8 NV8 SYS PIX PIX NODE NODE SYS SYS SYS SYS 0-59 0N/A
GPU2 NV8 NV8 X NV8 NV8 NV8 NV8 NV8 SYS NODE NODE PIX PIX SYS SYS SYS SYS 0-59 0N/A
GPU3 NV8 NV8 NV8 X NV8 NV8 NV8 NV8 SYS NODE NODE PIX PIX SYS SYS SYS SYS 0-59 0N/A
GPU4 NV8 NV8 NV8 NV8 X NV8 NV8 NV8 SYS SYS SYS SYS SYS PIX PIX NODE NODE 60-119 1N/A
GPU5 NV8 NV8 NV8 NV8 NV8 X NV8 NV8 SYS SYS SYS SYS SYS PIX PIX NODE NODE 60-119 1N/A
GPU6 NV8 NV8 NV8 NV8 NV8 NV8 X NV8 SYS SYS SYS SYS SYS NODE NODE PIX PIX 60-119 1N/A
GPU7 NV8 NV8 NV8 NV8 NV8 NV8 NV8 X SYS SYS SYS SYS SYS NODE NODE PIX PIX 60-119 1N/A
NIC0 SYS SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS
NIC1 PIX PIX NODE NODE SYS SYS SYS SYS SYS X PIX NODE NODE SYS SYS SYS SYS
NIC2 PIX PIX NODE NODE SYS SYS SYS SYS SYS PIX X NODE NODE SYS SYS SYS SYS
NIC3 NODE NODE PIX PIX SYS SYS SYS SYS SYS NODE NODE X PIX SYS SYS SYS SYS
NIC4 NODE NODE PIX PIX SYS SYS SYS SYS SYS NODE NODE PIX X SYS SYS SYS SYS
NIC5 SYS SYS SYS SYS PIX PIX NODE NODE SYS SYS SYS SYS SYS X PIX NODE NODE
NIC6 SYS SYS SYS SYS PIX PIX NODE NODE SYS SYS SYS SYS SYS PIX X NODE NODE
NIC7 SYS SYS SYS SYS NODE NODE PIX PIX SYS SYS SYS SYS SYS NODE NODE X PIX
NIC8 SYS SYS SYS SYS NODE NODE PIX PIX SYS SYS SYS SYS SYS NODE NODE PIX X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
NIC8: mlx5_8

Model Input Dumps

No response

🐛 Describe the bug

INFO 09-29 11:48:11 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager
INFO 09-29 11:48:12 multiproc_worker_utils.py:215] Worker ready; awaiting tasks
ERROR 09-29 11:48:12 multiproc_worker_utils.py:226] Exception in worker VllmWorkerProcess while processing method init_device: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method, Traceback (most recent call last):
ERROR 09-29 11:48:12 multiproc_worker_utils.py:226] File "/home/tiger/.local/lib/python3.9/site-packages/vllm/executor/multiproc_worker_utils.py", line 223, in _run_worker_process
ERROR 09-29 11:48:12 multiproc_worker_utils.py:226] output = executor(*args, **kwargs)
ERROR 09-29 11:48:12 multiproc_worker_utils.py:226] File "/home/tiger/.local/lib/python3.9/site-packages/vllm/worker/worker.py", line 166, in init_device
ERROR 09-29 11:48:12 multiproc_worker_utils.py:226] torch.cuda.set_device(self.device)
ERROR 09-29 11:48:12 multiproc_worker_utils.py:226] File "/home/tiger/.local/lib/python3.9/site-packages/torch/cuda/init.py", line 420, in set_device
ERROR 09-29 11:48:12 multiproc_worker_utils.py:226] torch._C._cuda_setDevice(device)
ERROR 09-29 11:48:12 multiproc_worker_utils.py:226] File "/home/tiger/.local/lib/python3.9/site-packages/torch/cuda/init.py", line 300, in _lazy_init
ERROR 09-29 11:48:12 multiproc_worker_utils.py:226] raise RuntimeError(
ERROR 09-29 11:48:12 multiproc_worker_utils.py:226] RuntimeError: Cannot re-initialize CUDA in forked subprocess. To use CUDA with multiprocessing, you must use the 'spawn' start method
ERROR 09-29 11:48:12 multiproc_worker_utils.py:226]

Before submitting a new issue...

  • Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
@ruleGreen ruleGreen added the bug Something isn't working label Sep 29, 2024
@DarkLight1337
Copy link
Member

Please check your vLLM installation. Your collect_env.py shows that you're still on v0.6.0:

Neuron SDK Version: N/A
vLLM Version: 0.6.0@32e7db25365415841ebc7c4215851743fbb1bad1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled

@ruleGreen ruleGreen mentioned this issue Sep 29, 2024
Closed
1 task
@ruleGreen
Copy link
Author

Please check your vLLM installation. Your collect_env.py shows that you're still on v0.6.0:

Neuron SDK Version: N/A
vLLM Version: 0.6.0@32e7db25365415841ebc7c4215851743fbb1bad1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled

sorry, should be this one.

Collecting environment information...
PyTorch version: 2.4.0+cu121
Is debug build: False
CUDA used to build PyTorch: 12.1
ROCM used to build PyTorch: N/A

OS: Debian GNU/Linux 11 (bullseye) (x86_64)
GCC version: (Debian 10.2.1-6) 10.2.1 20210110
Clang version: Could not collect
CMake version: version 3.18.4
Libc version: glibc-2.31

Python version: 3.9.2 (default, Feb 28 2021, 17:03:44) [GCC 10.2.1 20210110] (64-bit runtime)
Python platform: Linux-5.4.143.bsk.7-amd64-x86_64-with-glibc2.31
Is CUDA available: True
CUDA runtime version: 12.1.105
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A800-SXM4-80GB
GPU 1: NVIDIA A800-SXM4-80GB
GPU 2: NVIDIA A800-SXM4-80GB
GPU 3: NVIDIA A800-SXM4-80GB
GPU 4: NVIDIA A800-SXM4-80GB
GPU 5: NVIDIA A800-SXM4-80GB
GPU 6: NVIDIA A800-SXM4-80GB
GPU 7: NVIDIA A800-SXM4-80GB

Nvidia driver version: 535.161.08
cuDNN version: Probably one of the following:
/usr/lib/x86_64-linux-gnu/libcudnn.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_adv_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_cnn_train.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_infer.so.8.9.0
/usr/lib/x86_64-linux-gnu/libcudnn_ops_train.so.8.9.0
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Byte Order: Little Endian
Address sizes: 46 bits physical, 57 bits virtual
CPU(s): 120
On-line CPU(s) list: 0-119
Thread(s) per core: 2
Core(s) per socket: 30
Socket(s): 2
NUMA node(s): 2
Vendor ID: GenuineIntel
CPU family: 6
Model: 106
Model name: Intel(R) Xeon(R) Platinum 8336C CPU @ 2.30GHz
Stepping: 6
CPU MHz: 2294.608
BogoMIPS: 4589.21
Hypervisor vendor: KVM
Virtualization type: full
L1d cache: 2.8 MiB
L1i cache: 1.9 MiB
L2 cache: 75 MiB
L3 cache: 108 MiB
NUMA node0 CPU(s): 0-59
NUMA node1 CPU(s): 60-119
Vulnerability Itlb multihit: Not affected
Vulnerability L1tf: Not affected
Vulnerability Mds: Not affected
Vulnerability Meltdown: Not affected
Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling
Vulnerability Srbds: Not affected
Vulnerability Tsx async abort: Not affected
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology cpuid tsc_known_freq pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves wbnoinvd arat avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg avx512_vpopcntdq rdpid md_clear arch_capabilities

Versions of relevant libraries:
[pip3] byted-torch==2.1.0.post2
[pip3] byted-torch-monitor==0.0.1
[pip3] numpy==1.26.2
[pip3] nvidia-cublas-cu12==12.1.3.1
[pip3] nvidia-cuda-cupti-cu12==12.1.105
[pip3] nvidia-cuda-nvrtc-cu12==12.1.105
[pip3] nvidia-cuda-runtime-cu12==12.1.105
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.0.2.54
[pip3] nvidia-curand-cu12==10.3.2.106
[pip3] nvidia-cusolver-cu12==11.4.5.107
[pip3] nvidia-cusparse-cu12==12.1.0.106
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.20.5
[pip3] nvidia-nvjitlink-cu12==12.6.68
[pip3] nvidia-nvtx-cu12==12.1.105
[pip3] pyzmq==26.2.0
[pip3] torch==2.4.0
[pip3] torchaudio==2.4.0
[pip3] torchvision==0.19.0
[pip3] transformers==4.45.1
[pip3] triton==3.0.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.1.post1@acda0b35d00e733982aa4c1198f2bd381d368cb5
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 NIC8 CPU Affinity NUMA Affinity GPU NUMA ID
GPU0 X NV8 NV8 NV8 NV8 NV8 NV8 NV8 SYS PIX PIX NODE NODE SYS SYS SYS SYS 0-59 0N/A
GPU1 NV8 X NV8 NV8 NV8 NV8 NV8 NV8 SYS PIX PIX NODE NODE SYS SYS SYS SYS 0-59 0N/A
GPU2 NV8 NV8 X NV8 NV8 NV8 NV8 NV8 SYS NODE NODE PIX PIX SYS SYS SYS SYS 0-59 0N/A
GPU3 NV8 NV8 NV8 X NV8 NV8 NV8 NV8 SYS NODE NODE PIX PIX SYS SYS SYS SYS 0-59 0N/A
GPU4 NV8 NV8 NV8 NV8 X NV8 NV8 NV8 SYS SYS SYS SYS SYS PIX PIX NODE NODE 60-119 1N/A
GPU5 NV8 NV8 NV8 NV8 NV8 X NV8 NV8 SYS SYS SYS SYS SYS PIX PIX NODE NODE 60-119 1N/A
GPU6 NV8 NV8 NV8 NV8 NV8 NV8 X NV8 SYS SYS SYS SYS SYS NODE NODE PIX PIX 60-119 1N/A
GPU7 NV8 NV8 NV8 NV8 NV8 NV8 NV8 X SYS SYS SYS SYS SYS NODE NODE PIX PIX 60-119 1N/A
NIC0 SYS SYS SYS SYS SYS SYS SYS SYS X SYS SYS SYS SYS SYS SYS SYS SYS
NIC1 PIX PIX NODE NODE SYS SYS SYS SYS SYS X PIX NODE NODE SYS SYS SYS SYS
NIC2 PIX PIX NODE NODE SYS SYS SYS SYS SYS PIX X NODE NODE SYS SYS SYS SYS
NIC3 NODE NODE PIX PIX SYS SYS SYS SYS SYS NODE NODE X PIX SYS SYS SYS SYS
NIC4 NODE NODE PIX PIX SYS SYS SYS SYS SYS NODE NODE PIX X SYS SYS SYS SYS
NIC5 SYS SYS SYS SYS PIX PIX NODE NODE SYS SYS SYS SYS SYS X PIX NODE NODE
NIC6 SYS SYS SYS SYS PIX PIX NODE NODE SYS SYS SYS SYS SYS PIX X NODE NODE
NIC7 SYS SYS SYS SYS NODE NODE PIX PIX SYS SYS SYS SYS SYS NODE NODE X PIX
NIC8 SYS SYS SYS SYS NODE NODE PIX PIX SYS SYS SYS SYS SYS NODE NODE PIX X

Legend:

X = Self
SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
PIX = Connection traversing at most a single PCIe bridge
NV# = Connection traversing a bonded set of # NVLinks

NIC Legend:

NIC0: mlx5_0
NIC1: mlx5_1
NIC2: mlx5_2
NIC3: mlx5_3
NIC4: mlx5_4
NIC5: mlx5_5
NIC6: mlx5_6
NIC7: mlx5_7
NIC8: mlx5_8

@DarkLight1337
Copy link
Member

What is the command you used to run vLLM?

@DarkLight1337
Copy link
Member

By the way, please use Markdown syntax for code blocks, it makes reading them a lot easier.

@ruleGreen
Copy link
Author

What is the command you used to run vLLM?

from vllm import AsyncEngineArgs, AsyncLLMEngine, RequestOutput, SamplingParams, LLM, EngineArgs
self.model = LLM(**engine_args)

something like this, and engine_args below

image

@DarkLight1337
Copy link
Member

DarkLight1337 commented Sep 29, 2024 •
edited
Loading

DId you call torch.cuda-related functions (e.g. torch.cuda.is_available) before starting LLM? That may interfere with how vLLM uses CUDA. Try removing those functions.

@DarkLight1337
Copy link
Member

Also cc @youkaichao

@ruleGreen
Copy link
Author

I think this may also related to #8735 #7151

DId you call torch.cuda-related functions (e.g. torch.cuda.is_available) before starting LLM? That may interfere with how vLLM uses CUDA. Try removing those functions.

I do not have these functions before LLM

@ruleGreen
Copy link
Author

I think this may also related to #8735 #7151

DId you call torch.cuda-related functions (e.g. torch.cuda.is_available) before starting LLM? That may interfere with how vLLM uses CUDA. Try removing those functions.

I do not have these functions before LLM

maybe this could help

vllm_enforce_eager=True, vllm_maxlen=4096, vllm_gpu_util=0.9, adapter_name_or_path=None, vllm_max_lora_rank=8, visual_inputs=False, default_system=None, num_beams=1, max_new_tokens=4096, max_length=4096, repetition_penalty=1.0, temperature=0.0, top_p=0.95, top_k=50, length_penalty=1.0, model_revision='main', hf_hub_token=None, use_fast_tokenizer=True, split_special_tokens=False, new_special_tokens=None, resize_vocab=False, sample_num=1, cuda_ind=0, tensor_parallel=2, cuda_start=0, cuda_num=8, load_in_8bit=False, use_typewriter=0, min_new_tokens=1, cache_dir=None, worker_num=1, worker_id=0

I test my code, it goes well when tp = 1 on 1 node with 8 gpus to load 8b model, but it goes wrong when tp = 2 on same node to load 70b model.

@DarkLight1337
Copy link
Member

DarkLight1337 commented Sep 29, 2024 •
edited
Loading

TP = 1 means you only use one GPU. If you want to use all 8 GPUs on your node, you should instead set TP = 8. If one GPU can barely fit a 8B model, then you should use all 8 GPUs when loading the 70B version.

@ruleGreen
Copy link
Author

ruleGreen commented Sep 29, 2024 •
edited
Loading

TP = 1 means you only use one GPU. If you want to use all 8 GPUs on your node, you should instead set TP = 8. If one GPU can barely fit a 8B model, then you should use all 8 GPUs when loading the 70B version.

yep, I just want to use 2 of 8 gpus to load 70b model. also test tp=8 to load 70b model, same error

@DarkLight1337
Copy link
Member

Have you tried setting the multiprocessing method to spawn as suggested in the other threads?

@ruleGreen
Copy link
Author

Have you tried setting the multiprocessing method to spawn as suggested in the other threads?

yep, when i set spawn, it goes well in this machine, but in other machines, it just quits when loading models or like this #8735

@DarkLight1337
Copy link
Member

How is the environment different in "this machine" and "other machines"?

@DarkLight1337
Copy link
Member

If the other machines are using outdated vLLM version, then you should upgrade it.

@ruleGreen
Copy link
Author

How is the environment different in "this machine" and "other machines"?

I think same, also update vllm and other packages, but using more nodes and more gpus. so weird.

@DarkLight1337
Copy link
Member

How is the environment different in "this machine" and "other machines"?

I think same, also update vllm and other packages, but using more nodes and more gpus. so weird.

Can you also run collect_env.py on those machines and check whether they are consistent?

@ruleGreen
Copy link
Author

How is the environment different in "this machine" and "other machines"?

I think same, also update vllm and other packages, but using more nodes and more gpus. so weird.

Can you also run collect_env.py on those machines and check whether they are consistent?

will try it, thx

@ruleGreen
Copy link
Author

this issue seems can be solved by direct passing distributed_executor_backend as ray

@takagi97
Copy link

DId you call torch.cuda-related functions (e.g. torch.cuda.is_available) before starting LLM? That may interfere with how vLLM uses CUDA. Try removing those functions.

So sad. This is a serious problem that prevents me from integrating vllm into other neural network training systems, such as transformers.

@DarkLight1337
Copy link
Member

DId you call torch.cuda-related functions (e.g. torch.cuda.is_available) before starting LLM? That may interfere with how vLLM uses CUDA. Try removing those functions.

So sad. This is a serious problem that prevents me from integrating vllm into other neural network training systems, such as transformers.

Please open a new issue regarding this.

@takagi97 takagi97 mentioned this issue Oct 17, 2024
Open
1 task
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@ruleGreen @DarkLight1337 @takagi97