Run error

[UncleFB]

from vllm import LLM, SamplingParams

prompts = [
    "Hello, my name is",
    "The president of the United States is",
    "The capital of France is",
    "The future of AI is",
]
sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

llm = LLM(model="facebook/opt-125m", tensor_parallel_size=8)

root@dell:/workspace# python test.py 
2023-08-10 17:15:34,725 ERROR services.py:1207 -- Failed to start the dashboard , return code 1
2023-08-10 17:15:34,726 ERROR services.py:1232 -- Error should be written to 'dashboard.log' or 'dashboard.err'. We are printing the last 20 lines for you. See 'https://docs.ray.io/en/master/ray-observability/ray-logging.html#logging-directory-structure' to find where the log file is.
2023-08-10 17:15:34,727 ERROR services.py:1276 -- 
The last 20 lines of /tmp/ray/session_2023-08-10_17-15-32_212987_31479/logs/dashboard.log (it contains the error message from the dashboard): 
    from ray.util.state.common import (
  File "/usr/local/lib/python3.8/dist-packages/ray/util/state/__init__.py", line 1, in <module>
    from ray.util.state.api import (
  File "/usr/local/lib/python3.8/dist-packages/ray/util/state/api.py", line 17, in <module>
    from ray.util.state.common import (
  File "/usr/local/lib/python3.8/dist-packages/ray/util/state/common.py", line 420, in <module>
    class ActorState(StateSchema):
  File "pydantic/dataclasses.py", line 224, in pydantic.dataclasses.dataclass.wrap
  File "pydantic/dataclasses.py", line 336, in pydantic.dataclasses._add_pydantic_validation_attributes
  File "pydantic/dataclasses.py", line 391, in pydantic.dataclasses.create_pydantic_model_from_dataclass
  File "pydantic/main.py", line 1026, in pydantic.main.create_model
  File "pydantic/main.py", line 198, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/fields.py", line 506, in pydantic.fields.ModelField.infer
  File "pydantic/fields.py", line 436, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 552, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 668, in pydantic.fields.ModelField._type_analysis
  File "/usr/lib/python3.8/typing.py", line 774, in __subclasscheck__
    return issubclass(cls, self.__origin__)
TypeError: issubclass() arg 1 must be a class
2023-08-10 17:15:34,734 WARNING services.py:1832 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 8589934592 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=10.24gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.
2023-08-10 17:15:34,900 INFO worker.py:1621 -- Started a local Ray instance.
[2023-08-10 17:15:36,902 E 31479 31479] core_worker.cc:201: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory

[iamsimha]

Facing the same error. This happens while using Ray distributed inference.

[iamsimha]

Duplicate #741

[thiagosalvatore]

Same thing happening here, I'm not able to use 4 GPUs to run inference

[thiagosalvatore]

I was able to fix it by downgrading the versions of typing-extensions

pip install typing-inspect==0.8.0 typing_extensions==4.5.0

There were other issues with the version of a nvidia lib that came, but this should fix this issue specifically

[ShawnALiu]

使用了这个后，pip install typing-inspect==0.8.0 typing_extensions==4.5.0，
又报错
File "/home/anaconda3/envs/vllm-env/lib/python3.9/site-packages/torch/serialization.py", line 17, in
from typing_extensions import TypeAlias # Python 3.10+
ModuleNotFoundError: No module named 'typing_extensions'

如果升级python3.10，直接崩了。

[hmellor]

Closing as duplicate of #741