Fix local development for vllm==0.8.5

[wallashss]

The problem

The problem started happening when I pulled new code for vllm-spyre and it has updates for vllm. I got this stacktrace:

ERROR 05-05 13:57:01 [registry.py:355]   File "/Users/wallashss/develop/vllm-spyre/.pyenv/lib/python3.12/site-packages/torch/_dynamo/eval_frame.py", line 790, in _optimize
ERROR 05-05 13:57:01 [registry.py:355]     compiler_config=backend.get_compiler_config()
ERROR 05-05 13:57:01 [registry.py:355]                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 05-05 13:57:01 [registry.py:355]   File "/Users/wallashss/develop/vllm-spyre/.pyenv/lib/python3.12/site-packages/torch/__init__.py", line 2237, in get_compiler_config
ERROR 05-05 13:57:01 [registry.py:355]     from torch._inductor.compile_fx import get_patched_config_dict
ERROR 05-05 13:57:01 [registry.py:355]   File "/Users/wallashss/develop/vllm-spyre/.pyenv/lib/python3.12/site-packages/torch/_inductor/compile_fx.py", line 72, in <module>
ERROR 05-05 13:57:01 [registry.py:355]     from .fx_passes.joint_graph import joint_graph_passes
ERROR 05-05 13:57:01 [registry.py:355]   File "/Users/wallashss/develop/vllm-spyre/.pyenv/lib/python3.12/site-packages/torch/_inductor/fx_passes/joint_graph.py", line 19, in <module>
ERROR 05-05 13:57:01 [registry.py:355]     from ..pattern_matcher import (
ERROR 05-05 13:57:01 [registry.py:355]   File "/Users/wallashss/develop/vllm-spyre/.pyenv/lib/python3.12/site-packages/torch/_inductor/pattern_matcher.py", line 96, in <module>
ERROR 05-05 13:57:01 [registry.py:355]     from .lowering import fallback_node_due_to_unsupported_type
ERROR 05-05 13:57:01 [registry.py:355]   File "/Users/wallashss/develop/vllm-spyre/.pyenv/lib/python3.12/site-packages/torch/_inductor/lowering.py", line 6430, in <module>
ERROR 05-05 13:57:01 [registry.py:355]     from . import kernel
ERROR 05-05 13:57:01 [registry.py:355]   File "/Users/wallashss/develop/vllm-spyre/.pyenv/lib/python3.12/site-packages/torch/_inductor/kernel/__init__.py", line 1, in <module>
ERROR 05-05 13:57:01 [registry.py:355]     from . import mm, mm_common, mm_plus_mm, unpack_mixed_mm
ERROR 05-05 13:57:01 [registry.py:355]   File "/Users/wallashss/develop/vllm-spyre/.pyenv/lib/python3.12/site-packages/torch/_inductor/kernel/mm.py", line 16, in <module>
ERROR 05-05 13:57:01 [registry.py:355]     from torch._inductor.codegen.cpp_gemm_template import CppPackedGemmTemplate
ERROR 05-05 13:57:01 [registry.py:355]   File "/Users/wallashss/develop/vllm-spyre/.pyenv/lib/python3.12/site-packages/torch/_inductor/codegen/cpp_gemm_template.py", line 19, in <module>
ERROR 05-05 13:57:01 [registry.py:355]     from .cpp_micro_gemm import CppMicroGemmAMX, create_micro_gemm, LayoutType
ERROR 05-05 13:57:01 [registry.py:355]   File "/Users/wallashss/develop/vllm-spyre/.pyenv/lib/python3.12/site-packages/torch/_inductor/codegen/cpp_micro_gemm.py", line 16, in <module>
ERROR 05-05 13:57:01 [registry.py:355]     from .cpp_template_kernel import CppTemplateKernel
ERROR 05-05 13:57:01 [registry.py:355]   File "/Users/wallashss/develop/vllm-spyre/.pyenv/lib/python3.12/site-packages/torch/_inductor/codegen/cpp_template_kernel.py", line 20, in <module>
ERROR 05-05 13:57:01 [registry.py:355]     from .cpp_wrapper_cpu import CppWrapperCpu
ERROR 05-05 13:57:01 [registry.py:355]   File "/Users/wallashss/develop/vllm-spyre/.pyenv/lib/python3.12/site-packages/torch/_inductor/codegen/cpp_wrapper_cpu.py", line 30, in <module>
ERROR 05-05 13:57:01 [registry.py:355]     from .wrapper import EnterSubgraphLine, ExitSubgraphLine, WrapperCodeGen
ERROR 05-05 13:57:01 [registry.py:355]   File "/Users/wallashss/develop/vllm-spyre/.pyenv/lib/python3.12/site-packages/torch/_inductor/codegen/wrapper.py", line 46, in <module>
ERROR 05-05 13:57:01 [registry.py:355]     from ..runtime import triton_heuristics
ERROR 05-05 13:57:01 [registry.py:355]   File "/Users/wallashss/develop/vllm-spyre/.pyenv/lib/python3.12/site-packages/torch/_inductor/runtime/triton_heuristics.py", line 55, in <module>
ERROR 05-05 13:57:01 [registry.py:355]     from triton import Config
ERROR 05-05 13:57:01 [registry.py:355] ImportError: cannot import name 'Config' from 'triton' (unknown location)
ERROR 05-05 13:57:01 [registry.py:355] 

For newer vllm version (>=0.8.5), vLLM pokes the loaded module and it creates a placeholder for triton. This workaround can make pytorch mess up some code related to torch.compile.

My solution is just delete the triton module, and everything's can work ok again using Dyname Backend as eager in case we are developing on mac os.

Prashant:

PR for triton placeholder for reference: vllm-project/vllm#15099

[github-actions]

👋 Hi! Thank you for contributing to vLLM support on Spyre.
Just a reminder: Make sure that your code passes all the linting checks, otherwise your PR won't be able to be merged. To do so, first install the linting requirements, then run format.sh and commit the changes. This can be done with uv directly:

uv sync --frozen --group lint

Or this can be done with pip:

uv pip compile --group lint > requirements-lint.txt
pip install -r requirements-lint.txt
bash format.sh

Now you are good to go 🚀

[prashantgupta24]

yep I can confirm this works

[prashantgupta24]

I'm also curious to know what changed in 0.8.5 such that this Triton placeholder started behaving incorrectly? I'm assuming this placeholder existed in earlier versions too right?

Edit - I think I found the answer - https://github.com/vllm-project/vllm/pull/15099/files

So the TritonPlaceholder class was introduced 2 weeks ago, which means this did not exist earlier.

[wallashss]

I'm also curious to know what changed in 0.8.5 such that this Triton placeholder started behaving incorrectly? I'm assuming this placeholder existed in earlier versions too right?

Edit - I think I found the answer - https://github.com/vllm-project/vllm/pull/15099/files

So the TritonPlaceholder class was introduced 2 weeks ago, which means this did not exist earlier.

Yeah, this is the issue. I think a back contribution to the upstream would be a full fledge placeholder that every imports work, for instance to be able to import a dummy config; or a environment variable that prevents that, due the side effect it can cause on pytorch. Therefore, for cases like ours, and others it would be better to raise an exception than get an weird behavior in other dependencies like it does to the pytorch itself.