[Feature]: V1 support Xformers

[aabbccddwasd]

🚀 The feature, motivation and pitch

I tried to use V1 engine to load Qwen2.5-72B-Instruct-GPTQ-int4 on 4*2080ti 22g, and it raised an AssertionError. The error message shows assert is_fa_version_supported(self.fa_version) caused this AssertionError. It seems like the 2080ti does not support any version of FA. In the V0 engine, it uses the Xformers backend and works fine. However, in V1, it raises an error and stops working. So, I would like to request Xformers support for the V1 engine. I know the 2080ti is a bit outdated, but it is the only choice for getting a large GPU memory at an acceptable price. I really appreciate your help with this. It would mean a lot.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[aabbccddwasd]

full report:
Feb 04 08:47:05 server-2080ti sudo[477901]: INFO 02-04 08:47:05 core.py:45] Initializing a V1 LLM engine (v0.7.1) with config: model='/data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4', speculative_config=None, tokenizer='/data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=32678, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=4, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gptq, enforce_eager=False, kv_cache_dtype=auto, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=qwen, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=True, chunked_prefill_enabled=True, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"level":3,"custom_ops":["none"],"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"use_inductor":true,"compile_sizes":[],"use_cudagraph":true,"cudagraph_num_of_warmups":1,"cudagraph_capture_sizes":[512,504,496,488,480,472,464,456,448,440,432,424,416,408,400,392,384,376,368,360,352,344,336,328,320,312,304,296,288,280,272,264,256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":512} Feb 04 08:47:05 server-2080ti sudo[477901]: WARNING 02-04 08:47:05 multiproc_worker_utils.py:298] Reducing Torch parallelism from 28 threads to 1 to avoid unnecessary CPU contention. Set OMP_NUM_THREADS in the external environment to tune this value as needed. Feb 04 08:47:05 server-2080ti sudo[477901]: INFO 02-04 08:47:05 custom_cache_manager.py:17] Setting Triton cache manager to: vllm.triton_utils.custom_cache_manager:CustomCacheManager Feb 04 08:47:05 server-2080ti sudo[477901]: INFO 02-04 08:47:05 shm_broadcast.py:256] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[0, 1, 2, 3], buffer_handle=(4, 10485760, 10, 'psm_9e6b21c5'), local_subscribe_port=44291, remote_subscribe_port=None) Feb 04 08:47:05 server-2080ti sudo[477960]: (VllmWorker rank=0 pid=477960) INFO 02-04 08:47:05 shm_broadcast.py:256] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_4ffec3ce'), local_subscribe_port=46919, remote_subscribe_port=None) Feb 04 08:47:05 server-2080ti sudo[477975]: (VllmWorker rank=1 pid=477975) INFO 02-04 08:47:05 shm_broadcast.py:256] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_ca83cf9b'), local_subscribe_port=49753, remote_subscribe_port=None) Feb 04 08:47:06 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) INFO 02-04 08:47:06 shm_broadcast.py:256] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_c1f4574e'), local_subscribe_port=40745, remote_subscribe_port=None) Feb 04 08:47:06 server-2080ti sudo[478008]: (VllmWorker rank=3 pid=478008) INFO 02-04 08:47:06 shm_broadcast.py:256] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[0], buffer_handle=(1, 10485760, 10, 'psm_016fb16e'), local_subscribe_port=59379, remote_subscribe_port=None) Feb 04 08:47:07 server-2080ti sudo[477975]: (VllmWorker rank=1 pid=477975) INFO 02-04 08:47:07 utils.py:938] Found nccl from library libnccl.so.2 Feb 04 08:47:07 server-2080ti sudo[478008]: (VllmWorker rank=3 pid=478008) INFO 02-04 08:47:07 utils.py:938] Found nccl from library libnccl.so.2 Feb 04 08:47:07 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) INFO 02-04 08:47:07 utils.py:938] Found nccl from library libnccl.so.2 Feb 04 08:47:07 server-2080ti sudo[477975]: (VllmWorker rank=1 pid=477975) INFO 02-04 08:47:07 pynccl.py:67] vLLM is using nccl==2.21.5 Feb 04 08:47:07 server-2080ti sudo[477960]: (VllmWorker rank=0 pid=477960) INFO 02-04 08:47:07 utils.py:938] Found nccl from library libnccl.so.2 Feb 04 08:47:07 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) INFO 02-04 08:47:07 pynccl.py:67] vLLM is using nccl==2.21.5 Feb 04 08:47:07 server-2080ti sudo[478008]: (VllmWorker rank=3 pid=478008) INFO 02-04 08:47:07 pynccl.py:67] vLLM is using nccl==2.21.5 Feb 04 08:47:07 server-2080ti sudo[477960]: (VllmWorker rank=0 pid=477960) INFO 02-04 08:47:07 pynccl.py:67] vLLM is using nccl==2.21.5 Feb 04 08:47:07 server-2080ti sudo[477975]: (VllmWorker rank=1 pid=477975) WARNING 02-04 08:47:07 custom_all_reduce.py:134] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. Feb 04 08:47:07 server-2080ti sudo[478008]: (VllmWorker rank=3 pid=478008) WARNING 02-04 08:47:07 custom_all_reduce.py:134] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. Feb 04 08:47:07 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) WARNING 02-04 08:47:07 custom_all_reduce.py:134] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. Feb 04 08:47:07 server-2080ti sudo[477960]: (VllmWorker rank=0 pid=477960) WARNING 02-04 08:47:07 custom_all_reduce.py:134] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. Feb 04 08:47:07 server-2080ti sudo[477960]: (VllmWorker rank=0 pid=477960) INFO 02-04 08:47:07 shm_broadcast.py:256] vLLM message queue communication handle: Handle(connect_ip='127.0.0.1', local_reader_ranks=[1, 2, 3], buffer_handle=(3, 4194304, 6, 'psm_91d69185'), local_subscribe_port=54999, remote_subscribe_port=None) Feb 04 08:47:07 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) WARNING 02-04 08:47:07 config.py:975] MLA is not supported with gptq quantization. Disabling MLA. Feb 04 08:47:07 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) WARNING 02-04 08:47:07 registry.py:336]mm_limitshas already been set for model=/data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4, and will be overwritten by the new values. Feb 04 08:47:07 server-2080ti sudo[478008]: (VllmWorker rank=3 pid=478008) WARNING 02-04 08:47:07 config.py:975] MLA is not supported with gptq quantization. Disabling MLA. Feb 04 08:47:07 server-2080ti sudo[478008]: (VllmWorker rank=3 pid=478008) WARNING 02-04 08:47:07 registry.py:336]mm_limitshas already been set for model=/data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4, and will be overwritten by the new values. Feb 04 08:47:07 server-2080ti sudo[477960]: (VllmWorker rank=0 pid=477960) WARNING 02-04 08:47:07 config.py:975] MLA is not supported with gptq quantization. Disabling MLA. Feb 04 08:47:07 server-2080ti sudo[477960]: (VllmWorker rank=0 pid=477960) WARNING 02-04 08:47:07 registry.py:336]mm_limitshas already been set for model=/data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4, and will be overwritten by the new values. Feb 04 08:47:07 server-2080ti sudo[477975]: (VllmWorker rank=1 pid=477975) WARNING 02-04 08:47:07 config.py:975] MLA is not supported with gptq quantization. Disabling MLA. Feb 04 08:47:07 server-2080ti sudo[477975]: (VllmWorker rank=1 pid=477975) WARNING 02-04 08:47:07 registry.py:336]mm_limitshas already been set for model=/data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4, and will be overwritten by the new values. Feb 04 08:47:07 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) INFO 02-04 08:47:07 gpu_model_runner.py:843] Starting to load model /data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4... Feb 04 08:47:07 server-2080ti sudo[477960]: (VllmWorker rank=0 pid=477960) INFO 02-04 08:47:07 gpu_model_runner.py:843] Starting to load model /data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4... Feb 04 08:47:07 server-2080ti sudo[478008]: (VllmWorker rank=3 pid=478008) INFO 02-04 08:47:07 gpu_model_runner.py:843] Starting to load model /data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4... Feb 04 08:47:07 server-2080ti sudo[477975]: (VllmWorker rank=1 pid=477975) INFO 02-04 08:47:07 gpu_model_runner.py:843] Starting to load model /data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4... Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) INFO 02-04 08:47:08 cuda.py:163] Using Flash Attention backend on V1 engine. Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) ERROR 02-04 08:47:08 flash_attn.py:151] Cannot use FA version 2 is not supported due to FA3 is only supported on devices with compute capability >= 8 excluding 8.6 and 8.9 Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) WARNING 02-04 08:47:08 config.py:3401]torch.compileis turned on, but the model /data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4 does not support it. Please open an issue on GitHubif you want it to be supported. Feb 04 08:47:08 server-2080ti sudo[477960]: (VllmWorker rank=0 pid=477960) INFO 02-04 08:47:08 cuda.py:163] Using Flash Attention backend on V1 engine. Feb 04 08:47:08 server-2080ti sudo[477960]: (VllmWorker rank=0 pid=477960) ERROR 02-04 08:47:08 flash_attn.py:151] Cannot use FA version 2 is not supported due to FA3 is only supported on devices with compute capability >= 8 excluding 8.6 and 8.9 Feb 04 08:47:08 server-2080ti sudo[477960]: (VllmWorker rank=0 pid=477960) WARNING 02-04 08:47:08 config.py:3401]torch.compileis turned on, but the model /data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4 does not support it. Please open an issue on GitHubif you want it to be supported. Feb 04 08:47:08 server-2080ti sudo[477975]: (VllmWorker rank=1 pid=477975) INFO 02-04 08:47:08 cuda.py:163] Using Flash Attention backend on V1 engine. Feb 04 08:47:08 server-2080ti sudo[477975]: (VllmWorker rank=1 pid=477975) ERROR 02-04 08:47:08 flash_attn.py:151] Cannot use FA version 2 is not supported due to FA3 is only supported on devices with compute capability >= 8 excluding 8.6 and 8.9 Feb 04 08:47:08 server-2080ti sudo[477975]: (VllmWorker rank=1 pid=477975) WARNING 02-04 08:47:08 config.py:3401]torch.compileis turned on, but the model /data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4 does not support it. Please open an issue on GitHubif you want it to be supported. Feb 04 08:47:08 server-2080ti sudo[478008]: (VllmWorker rank=3 pid=478008) INFO 02-04 08:47:08 cuda.py:163] Using Flash Attention backend on V1 engine. Feb 04 08:47:08 server-2080ti sudo[478008]: (VllmWorker rank=3 pid=478008) ERROR 02-04 08:47:08 flash_attn.py:151] Cannot use FA version 2 is not supported due to FA3 is only supported on devices with compute capability >= 8 excluding 8.6 and 8.9 Feb 04 08:47:08 server-2080ti sudo[478008]: (VllmWorker rank=3 pid=478008) WARNING 02-04 08:47:08 config.py:3401]torch.compileis turned on, but the model /data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4 does not support it. Please open an issue on GitHubif you want it to be supported. Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) Process ForkProcess-1:3: Feb 04 08:47:08 server-2080ti sudo[477901]: CRITICAL 02-04 08:47:08 multiproc_executor.py:46] MulitprocExecutor got fatal signal from worker processes, shutting down. See stack trace above for root cause issue. Feb 04 08:47:08 server-2080ti sudo[477568]: CRITICAL 02-04 08:47:08 core_client.py:156] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue. Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) Traceback (most recent call last): Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) self.run() Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/multiprocessing/process.py", line 108, in run Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) self._target(*self._args, **self._kwargs) Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 308, in worker_main Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) worker = WorkerProc(*args, **kwargs) Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/v1/executor/multiproc_executor.py", line 241, in __init__ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) self.worker.load_model() Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/v1/worker/gpu_worker.py", line 142, in load_model Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) self.model_runner.load_model() Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/v1/worker/gpu_model_runner.py", line 845, in load_model Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) self.model = get_model(vllm_config=self.vllm_config) Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/model_executor/model_loader/__init__.py", line 12, in get_model Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) return loader.load_model(vllm_config=vllm_config) Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/model_executor/model_loader/loader.py", line 377, in load_model Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) model = _initialize_model(vllm_config=vllm_config) Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/model_executor/model_loader/loader.py", line 119, in _initialize_model Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) return model_class(vllm_config=vllm_config, prefix=prefix) Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/model_executor/models/qwen2.py", line 451, in __init__ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) self.model = Qwen2Model(vllm_config=vllm_config, Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/compilation/decorators.py", line 149, in __init__ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) old_init(self, vllm_config=vllm_config, prefix=prefix, **kwargs) Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/model_executor/models/qwen2.py", line 305, in __init__ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) self.start_layer, self.end_layer, self.layers = make_layers( Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) ^^^^^^^^^^^^ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/model_executor/models/utils.py", line 556, in make_layers Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) maybe_offload_to_cpu(layer_fn(prefix=f"{prefix}.{idx}")) Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/model_executor/models/qwen2.py", line 307, in <lambda> Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) lambda prefix: Qwen2DecoderLayer(config=config, Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/model_executor/models/qwen2.py", line 206, in __init__ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) self.self_attn = Qwen2Attention( Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) ^^^^^^^^^^^^^^^ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/model_executor/models/qwen2.py", line 158, in __init__ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) self.attn = Attention(self.num_heads, Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) ^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/attention/layer.py", line 114, in __init__ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) self.impl = impl_cls(num_heads, head_size, scale, num_kv_heads, Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) File "/home/aabbccddwasd/.conda/envs/vllm-latest/lib/python3.12/site-packages/vllm/v1/attention/backends/flash_attn.py", line 155, in __init__ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) assert is_fa_version_supported(self.fa_version) Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ Feb 04 08:47:08 server-2080ti sudo[477991]: (VllmWorker rank=2 pid=477991) AssertionError Feb 04 08:47:08 server-2080ti sudo[477549]: /data-HV-1t/AI-stuff/qwen-models/vllm-v2.5-72b-4bit.sh: line 16: 477568 Killed python -m vllm.entrypoints.openai.api_server --model /data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4 --tensor-parallel-size=4 --gpu-memory-utilization 0.9 --max-model-len 32678 --enable-prefix-caching --served-model-name qwen --swap-space 8 --disable-log-requests --quantization gptq --compilation-config 0

[NickLucche]

You can choose the backend with the environment variable VLLM_ATTENTION_BACKEND=XFORMERS

[aabbccddwasd]

You can choose the backend with the environment variable VLLM_ATTENTION_BACKEND=XFORMERS

seems not working，I will try is again

[aabbccddwasd]

`
#!/bin/bash
export CUDA_VISIBLE_DEVICES=2,3,4,5
export VLLM_ATTENTION_BACKEND=XFORMERS
export VLLM_USE_V1=1

source /usr/local/anaconda3/bin/activate /home/aabbccddwasd/.conda/envs/vllm-latest
python -m vllm.entrypoints.openai.api_server
--model /data-HV-1t/AI-stuff/qwen-models/Qwen2.5-72B-Instruct-GPTQ-Int4
--tensor-parallel-size=4
--gpu-memory-utilization 0.9
--max-model-len 32678
--enable-prefix-caching
--served-model-name qwen
--swap-space 8
--disable-log-requests
--quantization gptq
--compilation-config 0
`
this is my script, it still raise AssertionError, the log is exactly the same

[RawthiL]

Maybe its a really bad fix, but I managed to get my 2080ti to work on vLLM 7.1, by passing this parameter:
--kv-cache-dtype fp8

which triggers the the use of XFormers:

INFO 02-04 10:23:17 cuda.py:195] Cannot use FlashAttention-2 backend for FP8 KV cache.
WARNING 02-04 10:23:17 cuda.py:197] Please use FlashInfer backend with FP8 KV Cache for better performance by setting environment variable  VLLM_ATTENTION_BACKEND=FLASHINFER
INFO 02-04 10:23:17 cuda.py:232] Using XFormers backend.
(VllmWorkerProcess pid=190) INFO 02-04 10:23:17 cuda.py:195] Cannot use FlashAttention-2 backend for FP8 KV cache.
(VllmWorkerProcess pid=190) WARNING 02-04 10:23:17 cuda.py:197] Please use FlashInfer backend with FP8 KV Cache for better performance by setting environment variable  VLLM_ATTENTION_BACKEND=FLASHINFER
(VllmWorkerProcess pid=190) INFO 02-04 10:23:17 cuda.py:232] Using XFormers backend.

I tried using VLLM_ATTENTION_BACKEND=XFORMERS but it is not working or I'm not understanding how to set that properly.

[hmellor]

This snippet indicates that you either use V1 or xFormers:

vllm/vllm/platforms/cuda.py

Lines 157 to 174 in 18016a5

	if use_v1:
	logger.info("Using Flash Attention backend on V1 engine.")
	return "vllm.v1.attention.backends.flash_attn.FlashAttentionBackend"
	if use_mla:
	logger.info("Using Triton MLA backend.")
	return "vllm.attention.backends.triton_mla.TritonMLABackend"
	if selected_backend == _Backend.FLASHINFER:
	logger.info("Using FlashInfer backend.")
	return "vllm.attention.backends.flashinfer.FlashInferBackend"
	elif selected_backend == _Backend.XFORMERS:
	logger.info("Using XFormers backend.")
	return "vllm.attention.backends.xformers.XFormersBackend"
	elif selected_backend == _Backend.FLASH_ATTN:
	pass
	elif selected_backend:
	raise ValueError(
	f"Invalid attention backend for {cls.device_name}, "
	f"with use_v1: {use_v1} use_mla: {use_mla}")

Looking in https://github.com/vllm-project/vllm/tree/18016a5e627d2a4b69af599272a5aa8ce71b98c8/vllm/v1/attention/backends also indicates that currently, the only supported attention backend for V1 is Flash Attention.

[aabbccddwasd]

This snippet indicates that you either use V1 or xFormers:

vllm/vllm/platforms/cuda.py

Lines 157 to 174 in 18016a5

if use_v1:
logger.info("Using Flash Attention backend on V1 engine.")
return "vllm.v1.attention.backends.flash_attn.FlashAttentionBackend"
if use_mla:
logger.info("Using Triton MLA backend.")
return "vllm.attention.backends.triton_mla.TritonMLABackend"
if selected_backend == _Backend.FLASHINFER:
logger.info("Using FlashInfer backend.")
return "vllm.attention.backends.flashinfer.FlashInferBackend"
elif selected_backend == _Backend.XFORMERS:
logger.info("Using XFormers backend.")
return "vllm.attention.backends.xformers.XFormersBackend"
elif selected_backend == _Backend.FLASH_ATTN:
pass
elif selected_backend:
raise ValueError(
f"Invalid attention backend for {cls.device_name}, "
f"with use_v1: {use_v1} use_mla: {use_mla}")
Looking in https://github.com/vllm-project/vllm/tree/18016a5e627d2a4b69af599272a5aa8ce71b98c8/vllm/v1/attention/backends also indicates that currently, the only supported attention backend for V1 is Flash Attention.

ok I understand it, so will V1 support Xformers?

[aabbccddwasd]

Maybe its a really bad fix, but I managed to get my 2080ti to work on vLLM 7.1, by passing this parameter: --kv-cache-dtype fp8

which triggers the the use of XFormers:

INFO 02-04 10:23:17 cuda.py:195] Cannot use FlashAttention-2 backend for FP8 KV cache.
WARNING 02-04 10:23:17 cuda.py:197] Please use FlashInfer backend with FP8 KV Cache for better performance by setting environment variable  VLLM_ATTENTION_BACKEND=FLASHINFER
INFO 02-04 10:23:17 cuda.py:232] Using XFormers backend.
(VllmWorkerProcess pid=190) INFO 02-04 10:23:17 cuda.py:195] Cannot use FlashAttention-2 backend for FP8 KV cache.
(VllmWorkerProcess pid=190) WARNING 02-04 10:23:17 cuda.py:197] Please use FlashInfer backend with FP8 KV Cache for better performance by setting environment variable  VLLM_ATTENTION_BACKEND=FLASHINFER
(VllmWorkerProcess pid=190) INFO 02-04 10:23:17 cuda.py:232] Using XFormers backend.

I tried using VLLM_ATTENTION_BACKEND=XFORMERS but it is not working or I'm not understanding how to set that properly.

Maybe this way just disables V1 and drop back to V0

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!