为什么运行到一半出现oom的情况？

[AriesHaa]

运行代码为 export CUDA_VISIBLE_DEVICES=0,1,2,3 && python run.py --models vllm_qwen1_5_32b_chat_awq --datasets ceval_ppl --debug
然后运行到一半到这时就oom了
]
[rank0]: File "/root/opencompass/run.py", line 4, in
[rank0]: main()
[rank0]: File "/root/opencompass/opencompass/cli/main.py", line 298, in main
[rank0]: runner(tasks)
[rank0]: File "/root/opencompass/opencompass/runners/base.py", line 38, in call
[rank0]: status = self.launch(tasks)
[rank0]: File "/root/opencompass/opencompass/runners/local.py", line 116, in launch
[rank0]: task.run()
[rank0]: File "/root/opencompass/opencompass/tasks/openicl_infer.py", line 90, in run
[rank0]: self._inference()
[rank0]: File "/root/opencompass/opencompass/tasks/openicl_infer.py", line 140, in _inference
[rank0]: inferencer.inference(retriever,
[rank0]: File "/root/opencompass/opencompass/openicl/icl_inferencer/icl_ppl_inferencer.py", line 159, in inference
[rank0]: sub_res = self.model.get_ppl_from_template(sub_prompt_list).tolist()
[rank0]: File "/root/opencompass/opencompass/models/base.py", line 152, in get_ppl_from_template
[rank0]: return self.get_ppl(inputs, mask_length)
[rank0]: File "/root/opencompass/opencompass/models/vllm.py", line 117, in get_ppl
[rank0]: outputs = self.model.generate(inputs, sampling_kwargs)
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/vllm/utils.py", line 691, in inner
[rank0]: return fn(*args, **kwargs)
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 304, in generate
[rank0]: outputs = self._run_engine(use_tqdm=use_tqdm)
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/vllm/entrypoints/llm.py", line 556, in _run_engine
[rank0]: step_outputs = self.llm_engine.step()
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 776, in step
[rank0]: output = self.model_executor.execute_model(
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 76, in execute_model
[rank0]: return self._driver_execute_model(execute_model_req)
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/vllm/executor/multiproc_gpu_executor.py", line 84, in _driver_execute_model
[rank0]: return self.driver_worker.execute_model(
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/vllm/worker/worker.py", line 280, in execute_model
[rank0]: output = self.model_runner.execute_model(seq_group_metadata_list,
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
[rank0]: return func(*args, **kwargs)
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/vllm/worker/model_runner.py", line 765, in execute_model
[rank0]: output = self.model.sample(
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 345, in sample
[rank0]: next_tokens = self.sampler(logits, sampling_metadata)
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
[rank0]: return self._call_impl(*args, **kwargs)
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
[rank0]: return forward_call(*args, **kwargs)
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/vllm/model_executor/layers/sampler.py", line 112, in forward
[rank0]: prompt_logprobs, sample_logprobs = _get_logprobs(
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/vllm/model_executor/layers/sampler.py", line 776, in _get_logprobs
[rank0]: ranks = _get_ranks(
[rank0]: File "/data/conda-env/opencompass/lib/python3.10/site-packages/vllm/model_executor/layers/sampler.py", line 686, in get_ranks
[rank0]: return result.sum(1).add(1)
[rank0]: torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 1.07 GiB. GPU
ERROR 06-20 07:08:28 multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 2533865 died, exit code: -15
Fatal Python error: _enter_buffered_busy: could not acquire lock for <_io.BufferedWriter name=''> at interpreter shutdown, possibly due to daemon threads

配置文件如下：
from opencompass.models import VLLM

_meta_template = dict(
round=[
dict(role='HUMAN', begin='<|im_start|>user\n', end='<|im_end|>\n'),
dict(role='BOT', begin='<|im_start|>assistant\n', end='<|im_end|>\n', generate=True),
],
)

models = [
dict(
type=VLLM,
abbr='qwen1.5-32b-chat-vllm',
path='/data/Qwen1.5-32B-Chat-AWQ',
model_kwargs=dict(tensor_parallel_size=4),
meta_template=_meta_template,
max_out_len=100,
max_seq_len=1024,
batch_size=1,
generation_kwargs=dict(temperature=0),
stop_words=['<|im_end|>'],
run_cfg=dict(num_gpus=4, num_procs=1),
)
]

请问大家是如何解决的，谢谢！

[liushz]

Make sure you have about 100 GB GPU memory and try to add gpu_memory_utilization to the model config, like:
model_kwargs=dict(tensor_parallel_size=2, gpu_memory_utilization=0.7),

[AriesHaa]

添加 model_kwargs=dict(tensor_parallel_size=2, gpu_memory_utilization=0.7),后成功运行，谢谢！

[QiMingChina]

我添加了这两个参数，但是报错了...
TypeError: Qwen2ForCausalLM.init() got an unexpected keyword argument 'gpu_memory_utilization'
我的脚本：
MKL_SERVICE_FORCE_INTEL=0
MKL_THREADING_LAYER=GNU
python run.py
--hf-type chat
--hf-path /data/public/qim/model/Qwen1.5-4B-Chat
--model-kwargs gpu_memory_utilization=0.7
--tokenizer-path /data/public/qim/model/Qwen1.5-4B-Chat
--generation-kwargs do_sample=True
--accelerator vllm
--work-dir /data/public/qim/model/opencompass/outputs
--max-num-workers 10
--datasets Xsum_gen
--debug

[rationalspark]

请教一下，model_kwargs=dict(tensor_parallel_size=2, gpu_memory_utilization=0.7)是添加在哪里啊，我这样添加的
models = [
dict(
type=HuggingFaceBaseModel,
abbr='qwen2-7b-hf',
path='...',
max_out_len=1024,
batch_size=1,
run_cfg=dict(num_gpus=8, num_procs=1),
model_kwargs=dict(tensor_parallel_size=2, gpu_memory_utilization=0.7),
)
]

报错
TypeError: Qwen2ForCausalLM.init() got an unexpected keyword argument 'gpu_memory_utilization'