Error while starting API server when using Ray

[iamsimha]

I am installing the vllm using

FROM nvcr.io/nvidia/pytorch:22.12-py3
RUN pip uninstall torch -y
WORKDIR /workspace
RUN pip install vllm
EXPOSE 8097

start vllm server using

docker run --gpus all --shm-size=20g -p 8097:8097 -it --rm --name vllm vllm_docker python3 -m vllm.entrypoints.api_server --port 8097 --host 0.0.0.0 --trust-remote-code --tensor-parallel-size 8

The last 20 lines of /tmp/ray/session_2023-08-11_20-00-52_804243_1/logs/dashboard.log (it contains the error message from the dashboard):
    from ray.util.state.common import (
  File "/usr/local/lib/python3.8/dist-packages/ray/util/state/__init__.py", line 1, in <module>
    from ray.util.state.api import (
  File "/usr/local/lib/python3.8/dist-packages/ray/util/state/api.py", line 17, in <module>
    from ray.util.state.common import (
  File "/usr/local/lib/python3.8/dist-packages/ray/util/state/common.py", line 420, in <module>
    class ActorState(StateSchema):
  File "pydantic/dataclasses.py", line 224, in pydantic.dataclasses.dataclass.wrap
  File "pydantic/dataclasses.py", line 336, in pydantic.dataclasses._add_pydantic_validation_attributes
  File "pydantic/dataclasses.py", line 391, in pydantic.dataclasses.create_pydantic_model_from_dataclass
  File "pydantic/main.py", line 1026, in pydantic.main.create_model
  File "pydantic/main.py", line 198, in pydantic.main.ModelMetaclass.__new__
  File "pydantic/fields.py", line 506, in pydantic.fields.ModelField.infer
  File "pydantic/fields.py", line 436, in pydantic.fields.ModelField.__init__
  File "pydantic/fields.py", line 552, in pydantic.fields.ModelField.prepare
  File "pydantic/fields.py", line 668, in pydantic.fields.ModelField._type_analysis
  File "/usr/lib/python3.8/typing.py", line 774, in __subclasscheck__
    return issubclass(cls, self.__origin__)
TypeError: issubclass() arg 1 must be a class
2023-08-11 20:00:54,710 INFO worker.py:1621 -- Started a local Ray instance.
[2023-08-11 20:00:56,259 E 1 1] core_worker.cc:201: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory

[thiagosalvatore]

I was able to fix it by downgrading the versions of typing-extensions

pip install typing-inspect==0.8.0 typing_extensions==4.5.0

There were other issues with the version of a nvidia lib that came, but this should fix this issue specifically

[iamsimha]

I was able to fix it by downgrading the versions of typing-extensions

pip install typing-inspect==0.8.0 typing_extensions==4.5.0

There were other issues with the version of a nvidia lib that came, but this should fix this issue specifically

This fixes my issue

[Amanda-Barbara]

this method was not fixed mine? why?

[hxujal]

use pip install typing-inspect==0.8.0 typing_extensions==4.5.0:

anaconda3/envs/vllm/lib/python3.10/site-packages/huggingface_hub/utils/_runtime.py:184: UserWarning: Pydantic is installed but cannot be imported. Please check your installation. huggingface_hub will default to not using Pydantic. Error message: '{e}'
warnings.warn(
Traceback (most recent call last):
File "/home/centific/anaconda3/envs/vllm/lib/python3.10/runpy.py", line 187, in _run_module_as_main
mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
File "/home/centific/anaconda3/envs/vllm/lib/python3.10/runpy.py", line 110, in _get_module_details
import(pkg_name)
File "/home/centific/hua-project/vllm/vllm/init.py", line 4, in
from vllm.engine.async_llm_engine import AsyncLLMEngine
File "/home/centific/hua-project/vllm/vllm/engine/async_llm_engine.py", line 12, in
from vllm.engine.llm_engine import LLMEngine
File "/home/centific/hua-project/vllm/vllm/engine/llm_engine.py", line 10, in
from vllm.core.scheduler import Scheduler, SchedulerOutputs
File "/home/centific/hua-project/vllm/vllm/core/scheduler.py", line 8, in
from vllm.core.interfaces import AllocStatus, BlockSpaceManager
File "/home/centific/hua-project/vllm/vllm/core/interfaces.py", line 5, in
from vllm.sequence import Sequence, SequenceGroup
File "/home/centific/hua-project/vllm/vllm/sequence.py", line 9, in
from vllm.sampling_params import SamplingParams
File "/home/centific/hua-project/vllm/vllm/sampling_params.py", line 8, in
from pydantic import conint
File "/home/centific/anaconda3/envs/vllm/lib/python3.10/site-packages/pydantic/init.py", line 383, in getattr
module = import_module(module_name, package=package)
File "/home/centific/anaconda3/envs/vllm/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "/home/centific/anaconda3/envs/vllm/lib/python3.10/site-packages/pydantic/types.py", line 33, in
from typing_extensions import Annotated, Literal, Protocol, TypeAlias, TypeAliasType, deprecated
ImportError: cannot import name 'TypeAliasType' from 'typing_extensions' (/home/centific/anaconda3/envs/vllm/lib/python3.10/site-packages/typing_extensions.py)