[Bug]: Disagreement and misalignment between supported models in documentation and actual testing

[manitadayon]

Your current environment

Tested under VLLM ==0.7.3 and 0.8.2

🐛 Describe the bug

I am using this model (after quantizing it to 4 bits):
nvidia/Llama-3_3-Nemotron-Super-49B-v1
https://huggingface.co/nvidia/Llama-3_3-Nemotron-Super-49B-v1
and according to the documentation here:
https://docs.vllm.ai/en/latest/models/supported_models.html
(This is the summary of the related section in the above website:
To determine whether a given model is natively supported, you can check the config.json file inside the HF repository. If the "architectures" field contains a model architecture listed below, then it should be natively supported.
)
The nvidia/Llama-3_3-Nemotron-Super-49B-v1 model architecture according to the HF is DeciLMForCausalLM
and according to the documentation above, DeciLMForCausalLM is listed as one of the supported architectures hence it should be natively supported. However loading the above model create these issues:

1- Under VLLM ==0.7.3 I get this error:
DeciLMConfig object has no attribute 'num_key_value_heads_per_layer'
2- Under VLLM==0.8.2 I get OOM and this error together.
Can someone explain if I am doing anything wrong, or if I am interpreting anything wrong.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

vLLM needs to be updated for the newest Nemotron models, see WIP #15008

[manitadayon]

@DarkLight1337 Can we reopen this? I still get the same error on latest VLLM. This error and OOM issue with max_num_seqs = 8

I am trying this on A100 GPU with 80GB memory. I can run LLama 3.3 70B with no problem though.

[DarkLight1337]

Can you show the full logs?

[manitadayon]

Sorry I cannot copy and paste the logs, so I typed some parts of it)

Fatal error: The Python process exit code 137 (SIGKILL:killed). There may have been caused by OOM error. 

Error: EngineCore hit an exception: Traceback:
File: vllm/v1/engine/core.py line 335, in run_engine_core

       engine_core = EngineCoreProc(*args, **kwargs)


File: vllm/v1/engine/core.py
Line 60 in __init__ 
       self.model_executor = executor_class(vllm_config)

File: vllm/executor/executor_base.py
Line 52, in __init__
          self._init_executor()

File: vllm/executor/uniproc_executor.py
Line 47, in _init_executor
          self.collective_rpc("load_model")

File: transformers/configuration_utils.py, line 224:

       return super().__getattribute__(key)


AttributeError: 'DeciLMConfig' object has no attribute 'num_key_value_heads_per_layer'

[DarkLight1337]

When you say "latest vLLM", are you referring to latest code (main branch) or latest release (v0.8.2)?

[manitadayon]

Good point: I meant latest release the 0.8.2 version.(I guess the fix has not been released yet)
The latest build on GitHub will produce this error for me:

Error: Could not build wheels for vllm, which is required to install pyproject.toml-based projects

-- The CUDA compiler identification is unknown.
-- failed to detect a default CUDA architecture.

Similar issue already exists here:

#1391

[manitadayon]

This is the error I get if I want to install using the latest version on main branch after lots of troubleshooting:

CMake Error: The following variables
- [ ] are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the  CMake files:

CUDA_nvrtc_LIBRARY 

Now I would like to edit the CMakeLists.txt or add CUDA_TOOLKIT_ROOT in the CMake command, however given my current environment setup I cannot do it.

If you think the PR fixed the issue, please feel free to close it, and I can wait until next release and reopen the case if I still get the error.

[DarkLight1337]

I suggest you use the Docker image for latest code on main branch. https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#pre-built-images

[DarkLight1337]

By the way, we refer to latest release as "stable" in the docs version selector.

[manitadayon]

I am not sure if we want to reopen this case or you want me to post this somewhere else:
I tried this model today on the latest release (0.8.3) and the error I get is

EOFError:
File: multiprocessing/connection.py:387 in Connection._recv(self, size, read)

If n == 0:
    If remaining== size:
         raise EOFError
    else:
         raise OSError(“got end of file during message”)

I thought this is OOM issue, so I reduce the gpu_memory_utilization to 0.7 and max_num_seqs = 8 but still the same error.