[Feature]: Support for RTX 5090 (CUDA 12.8)

[rubentorresbonet]

🚀 The feature, motivation and pitch

Currently only nightlies from torch targeting 12.8 support blackwell such as the rtx 5090.
I tried using VLLM with a rtx 5090 and no dice. Vanilla vllm installation ends in:

NVIDIA GeForce RTX 5090 with CUDA capability sm_120 is not compatible with the current PyTorch installation.
The current PyTorch install supports CUDA capabilities sm_50 sm_60 sm_70 sm_75 sm_80 sm_86 sm_90.
If you want to use the NVIDIA GeForce RTX 5090 GPU with PyTorch, please check the instructions at https://pytorch.org/get-started/locally/
RuntimeError: CUDA error: no kernel image is available for execution on the device

Thanks

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[cyril23]

If anyone has any ideas, I can try it out on a RTX 5090.

Apparently:

• I guess you need https://github.com/vllm-project/vllm/pull/12702/files
• Adapt the Dockerfile:
  • Replace this with the nightly 12.8 (pip install --upgrade --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128, and maybe update "torch==2.7.0.dev20250121+cu126" "torchvision==0.22.0.dev20250121" to newer versions too?)
  • Update CUDA Version to some 12.8 e.g. here
• Adapt the requirements-build.txt torch version to the new one
• Adapt the requirements-cuda.txt too, torch, torchvision, torchaudio versions

But not sure how exactly to do it. Can anyone help me out? ideally open up a Pullrequest and I'll try to build it, report back any errors, until we get it to run some inference e.g.

[hxssgaa]

If anyone has any ideas, I can try it out on a RTX 5090.

Apparently:

• I guess you need https://github.com/vllm-project/vllm/pull/12702/files

• Adapt the Dockerfile:

  • Replace this with the nightly 12.8 (pip install --upgrade --pre torch torchvision torchaudio --index-url https://download.pytorch.org/whl/nightly/cu128, and maybe update "torch==2.7.0.dev20250121+cu126" "torchvision==0.22.0.dev20250121" to newer versions too?)
  • Update CUDA Version to some 12.8 e.g. here

• Adapt the requirements-build.txt torch version to the new one

• Adapt the requirements-cuda.txt too, torch, torchvision, torchaudio versions

But not sure how exactly to do it. Can anyone help me out? ideally open up a Pullrequest and I'll try to build it, report back any errors, until we get it to run some inference e.g.

The above method doesn't work for me because of a cutlass compilation error

[shahizat]

hello @rubentorresbonet, any updates? Were you able to run it?

[rubentorresbonet]

nope, switched to tabbyapi

[JiapengLi]

#14452 anyone tried this?

[bryantbiggs]

FYI for Blackwell - pytorch/pytorch#145949 (comment)

[g0t4]

BTW, cu128 nightly now has pytorch-triton 3.3.0 which fixes a failure I had using speculative decoding on a 5090:

'sm_120' is not a recognized processor for this target (ignoring processor)
LLVM ERROR: Cannot select: intrinsic %llvm.nvvm.shfl.sync.bfly.i32 

Here are my notes:

triton-lang/triton#5950 (comment)

[ssakar]

I have written a tutorial for setting up vLLM for the rtx 5090.

[cyril23]

#14452 anyone tried this?

• yes, the custom build presented in [Doc]: Steps to run vLLM on your RTX5080 or 5090! #14452 works for me. I can build and run VLLM like that.

I have written a tutorial for setting up vLLM for the rtx 5090.

• thanks! Just tried it out, it works for me, too.
• The solution is pretty similar to the one presented in [Doc]: Steps to run vLLM on your RTX5080 or 5090! #14452 (comment) but makes things a little bit easier, and you used better flags, like max jobs 4 etc. So maybe you post it in in [Doc]: Steps to run vLLM on your RTX5080 or 5090! #14452 too?
• edit: the great thing is you directly show how to run an OpenAI compatible server with vLLM, and not just how to build a Blackwell-compatible vllm package for python.
• Maybe in the https://github.com/ssakar/tutorial/tree/main/vllm#prerequisites you could give a hint for Ubuntu 22 users: they need to use a special repo in order to get podman 4+ which supports GPU etc., see Unable to install the latest podman on Ubuntu 22.04 by following the instruction on the website containers/podman#21249 (comment) - else one is stuck at podman 3

I would suggest a mod closes this issue here, and the discussion goes on in #14452 because that seems to be the main discussion now for 50XX Blackwell GPUs, and we don't need 2 separate issues?

[MrVolts]

I have written a tutorial for setting up vLLM for the rtx 5090.

Would this work with multi 5090 setups? I have two 5090's and am struggling with NCCL. Thanks

[ssakar]

Would this work with multi 5090 setups? I have two 5090's and am struggling with NCCL. Thanks

Yes, that should work. Only difference would be the additional vllm args to use more than one gpu.

[MrVolts]

Would this work with multi 5090 setups? I have two 5090's and am struggling with NCCL. Thanks

Yes, that should work. Only difference would be the additional vllm args to use more than one gpu.

I've been trying to get it working, and while it works on a single 5090 using AWQ or BNB format, it does not work on fp8 due to CUTLASS errors and GPU unless I install the latest NCCL. So I'm currently trying to get it to work with fp8, which the 5090 natively supports, and multi-GPU, which is proving incredibly tricky.