[Bug]: register model can't be found in v1 mode

[mmdzzh]

model registry in vllm 0.8.5.post1

model registry in vllm 0.8.5.post1 version has a bug in v1 mode

there is a bug in vllm v1 mode, the detail i found is that in v1 mode, my registered model name can't be found in  EngineCoreProc.run_engine_core

🐛 Describe the bug

there is a bug in vllm v1 mode, the detail i found is that vllm_config will be changed in vllm/v1/engine/core.py: def run_engine_core and vllm/v1/utils.py: self.proc: Process = context.Process(target=target_fn,, Theoretically, kwargs and process_kwargs should be identical, but in practice they differ. Specifically, the keys returned by kwargs.vllm_config.model_config.registry.models.keys() are missing the name of my custom-registered model.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

You have to register OOT models via plugins: https://docs.vllm.ai/en/latest/contributing/model/registration.html#out-of-tree-models

This is because V1 Engine runs its workers in separate processes. So, it's not sufficient to only register the model in your own code.

[mmdzzh]

so it means that my registered model didn't be found in load_general_plugins function? I need to write a setup function like this in my register file to make other processing to find my model?

setup(name='vllm_add_dummy_model',
      version='0.1',
      packages=[''],
      entry_points={
          'vllm.general_plugins':
          ["register_dummy_model = ModelClass"]
      })
ModelRegistry.register_model("ModelName", ModelClass)

You have to register OOT models via plugins: https://docs.vllm.ai/en/latest/contributing/model/registration.html#out-of-tree-models

This is because V1 Engine runs its workers in separate processes. So, it's not sufficient to only register the model in your own code.

[DarkLight1337]

You need to create your own plugin package, with a setup.py file to call setup and another Python file that acts as the entrypoint to the plugin

[mmdzzh]

You need to create your own plugin package, with a setup.py file to call setup and another Python file that acts as the entrypoint to the plugin

Sorry, I didn't quite understand. Do I need to write a setup.py and run python3 setup.py before import vLLM? Or
just import the setup file?

[DarkLight1337]

You need to create a small Python package. You can check out https://github.com/vllm-project/vllm/blob/cf069aa8aa38a9003c254f8434a29ec6a3070b08/tests/plugins/vllm_add_dummy_model as an example

[mmdzzh]

You need to create a small Python package. You can check out https://github.com/vllm-project/vllm/blob/cf069aa8aa38a9003c254f8434a29ec6a3070b08/tests/plugins/vllm_add_dummy_model as an example

Does this mean that I only need to write the setup.py file in my project folder and don't need to do anything

[DarkLight1337]

Apart from the setup.py file, you also need a separate file (the entrypoint) to register the model

[mmdzzh]

Apart from the setup.py file, you also need a separate file (the entrypoint) to register the model

Ok thank you for your patient explanation. , I'm just confused why this can be useful in plugin found. I know that in model register file there is a register function so the model can be found in vLLM, but why setup can do the things like ModelRegistry.register_model

[DarkLight1337]

You have to pip install the plugin package you created, just like any other Python library. The code in setup.py basically tells pip how to install the package.

[mmdzzh]

You have to pip install the plugin package you created, just like any other Python library. The code in setup.py basically tells pip how to install the package.

Ok, this can be closed, thx