-
-
Notifications
You must be signed in to change notification settings - Fork 11.2k
Closed
Labels
feature requestNew feature or requestNew feature or request
Description
🚀 The feature, motivation and pitch
As it is coded in /vllm/config.py
if self.enable_sleep_mode and not current_platform.is_cuda():
raise ValueError("Sleep mode is only supported on CUDA devices.")While there are oot platforms, which will support sleep mode (Ascend NPU has already done this in v0.7.3 branch)
The above code makes it hard to support sleep mode in oot platforms.
A possible solution is to add a method to base class Platform: is_sleep_mode_available(), oot Platform classes can override this method to indicate whether the platform supports sleep mode.
Alternatives
No response
Additional context
No response
Before submitting a new issue...
- Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.
Metadata
Metadata
Assignees
Labels
feature requestNew feature or requestNew feature or request