Plugin won't boot with vllm v0.8.1 #38
Description
Trying to boot with vllm 0.8.1 with VLLM_SPYRE_DYNAMO_BACKEND=eager and VLLM_USE_V1=1 gives the fun error:
INFO 03-19 22:31:45 [config.py:2595] Downcasting torch.float32 to torch.float16.
INFO 03-19 22:31:49 [config.py:583] This model supports multiple tasks: {'score', 'reward', 'classify', 'embed', 'generate'}. Defaulting to 'generate'.
Traceback (most recent call last):
File "/home/senuser/my-vllm/bin/vllm", line 8, in <module>
sys.exit(main())
^^^^^^
File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/entrypoints/cli/main.py", line 75, in main
args.dispatch_function(args)
File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/entrypoints/cli/serve.py", line 33, in cmd
uvloop.run(run_server(args))
File "/home/senuser/my-vllm/lib64/python3.11/site-packages/uvloop/__init__.py", line 105, in run
return runner.run(wrapper())
^^^^^^^^^^^^^^^^^^^^^
File "/usr/lib64/python3.11/asyncio/runners.py", line 118, in run
return self._loop.run_until_complete(task)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
File "/home/senuser/my-vllm/lib64/python3.11/site-packages/uvloop/__init__.py", line 61, in wrapper
return await main
^^^^^^^^^^
File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 1012, in run_server
async with build_async_engine_client(args) as engine_client:
File "/usr/lib64/python3.11/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 141, in build_async_engine_client
async with build_async_engine_client_from_engine_args(
File "/usr/lib64/python3.11/contextlib.py", line 210, in __aenter__
return await anext(self.gen)
^^^^^^^^^^^^^^^^^^^^^
File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/entrypoints/openai/api_server.py", line 161, in build_async_engine_client_from_engine_args
vllm_config = engine_args.create_engine_config(usage_context=usage_context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/engine/arg_utils.py", line 1215, in create_engine_config
if try_v1 and self._is_v1_supported_oracle(model_config):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/engine/arg_utils.py", line 1584, in _is_v1_supported_oracle
_raise_or_fallback(
File "/home/senuser/my-vllm/lib64/python3.11/site-packages/vllm/engine/arg_utils.py", line 1758, in _raise_or_fallback
raise NotImplementedError(
NotImplementedError: VLLM_USE_V1=1 is not supported with device type=cpu.