RuntimeError: t == DeviceType::CUDA INTERNAL ASSERT FAILED about v0.2.7

[cocovoc]

I met this problem.

outputs = llm.generate(prompts, sampling_params)
File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/vllm/entrypoints/llm.py", line 165, in generate
return self._run_engine(use_tqdm)
File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/vllm/entrypoints/llm.py", line 185, in _run_engine
step_outputs = self.llm_engine.step()
File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 628, in step
all_outputs = self._run_workers(
File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/vllm/engine/llm_engine.py", line 795, in _run_workers
driver_worker_output = getattr(self.driver_worker,
File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/vllm/worker/worker.py", line 183, in execute_model
self.cache_swap(*block_swapping_info)
File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/vllm/worker/worker.py", line 139, in cache_swap
self.cache_engine.swap_in(blocks_to_swap_in)
File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/vllm/worker/cache_engine.py", line 131, in swap_in
self._swap(self.cpu_cache, self.gpu_cache, src_to_dst)
File "/home/ma-user/anaconda3/envs/py39/lib/python3.9/site-packages/vllm/worker/cache_engine.py", line 123, in _swap
cache_ops.swap_blocks(src_key_cache, dst_key_cache, src_to_dst)
RuntimeError: t == DeviceType::CUDA INTERNAL ASSERT FAILED at "/opt/hostedtoolcache/Python/3.9.18/x64/lib/python3.9/site-packages/torch/include/c10/cuda/impl/CUDAGuardImpl.h":25, please report a bug to PyTorch.

[iNeil77]

Hi guys, any solution to this problem? I have come across the same issue running humaneval on codellama 7b

[LiuXiaoxuanPKU]

Could you share the hardware you run (GPU type, CPU memory etc.)? We will try to reproduce it.

[iNeil77]

I am using an A6000 GPU along with 64GB of RAM. I was testing it out with the following script:

from vllm import (
    LLM, 
    SamplingParams
)
from vllm.model_executor.parallel_utils.parallel_state import destroy_model_parallel
from datasets import load_dataset

ds = load_dataset(
    "openai_humaneval", 
    split="test"
)
prompts = ds["prompt"]
llm = LLM(
    model="codellama/codellama-7b-hf", 
    tensor_parallel_size=1, 
    dtype="bfloat16",
    trust_remote_code=True,
    gpu_memory_utilization=0.95,
    swap_space=32,
)
samp = SamplingParams(
    n=200, 
    top_p=0.95, 
    temperature=0.8, 
    max_tokens=1024,
    stop=[
        "\nclass", 
        "\ndef", 
        "\n#", 
        "\n@", 
        "\nprint", 
        "\nif", 
        "\n```", 
        "<file_sep>"
    ]
)
outputs = llm.generate(
    prompts, 
    samp
)
destroy_model_parallel()

[minimario]

I also faced this problem. I don't have insight into why, but I switched to v0.2.6, it was solved. By the way, sometimes I also needed to implement the solution from this issue #1082.

[sh1ng]

77af974 introduced the error.

[sh1ng]

It seems it was there a long time and adding device_guard just makes it visible.

[sh1ng]

Potential fix #2583

[LiuXiaoxuanPKU]

Close because of the fix #2583, please reopen the issue if the error persists.

[RyanWMHI]

I faced this problem today.