DeciLMConfig object has no attribute ‘num_key_value_heads_per_layer’

[manitadayon]

Your current environment

VLLM version: 0.7.3
Model: nvidia/Llama-3_3-Nemotron-Super-49B-v1

🐛 Describe the bug

I am trying to run the new Nvidia model Nemotron 49B-v1 using the VLLM 0.7.3 version but I got this error
DeciLMConfig object has no attribute ‘num_key_value_heads_per_layer’

I have two questions: I know there are PR such as
#15068
#15008

I am wondering about if the error would be resolved after adding the support for the model. Or the error is unrelated and something is wrong in my end?

[b8zhong]

Have you tried updating to the latest vLLM version (0.8.2)? @manitadayon

[manitadayon]

@b8zhong yes the latest version/release does not have the fix.
Still the same error. I think we need to wait for the next release,
Since you get an error if you want to install from
The source as well. Do you have any estimate on the next release?

[ameyanjarlekar]

@manitadayon it is working on 0.8.3

[manitadayon]

As I wrote here as well #15779

I would not say it works as the new release has issues with memory. If you have to use 8 A100 GPU for small model like that to make it to work, I would not say it is functioning.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!