[Model] Rename MiniCPMVQwen2 to MiniCPMV2.6

[jeejeelee]

I have completed the following modification:

• Rename MiniCPMVQwen2 to MiniCPMV2.6
• Slightly modified some logic
• Add MiniCPMV2.6 to list of supported models (FIX [New Model]:Is MiniCPM-V-2_6 supported? #7267)

ping @ywang96 @DarkLight1337 @HwwwwwwwH

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

• Comment /ready on the PR
• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

Could you also update the docs to include this in the list of supported models?

[HwwwwwwwH]

Sry for late. Thank you for your work! There're something still need attention. The example of minicpmv should be extended to contain three versions in example/offline_inference_vision_language.py. And three versions of minicpmv need different stop_token_ids. So if the user run example/offline_inference_vision_language.py, for example, for V2.5, they might got many <|eot|> at the end of outputs.
So the logic of offline_inference_vision_language.py may need a slight modification.

[jeejeelee]

Sry for late. Thank you for your work! There're something still need attention. The example of minicpmv should be extended to contain three versions in example/offline_inference_vision_language.py. And three versions of minicpmv need different stop_token_ids. So if the user run example/offline_inference_vision_language.py, for example, for V2.5, they might got many <|eot|> at the end of outputs. So the logic of offline_inference_vision_language.py many need a slight modification.

Ok, is that right？

  # 2.0
  stop_token_ids = [tokenizer.eos_id]
  # 2.5
  # stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id]
  #2.6
  # stop_tokens = ['<|im_end|>', '<|endoftext|>']
  # stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]

[HwwwwwwwH]

Ok, is that right？

  # 2.0
  stop_token_ids = [tokenizer.eos_id]
  # 2.5
  # stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id]
  #2.6
  # stop_tokens = ['<|im_end|>', '<|endoftext|>']
  # stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]

Yes

[jeejeelee]

Ok, is that right？

  # 2.0
  stop_token_ids = [tokenizer.eos_id]
  # 2.5
  # stop_token_ids = [tokenizer.eos_id, tokenizer.eot_id]
  #2.6
  # stop_tokens = ['<|im_end|>', '<|endoftext|>']
  # stop_token_ids = [tokenizer.convert_tokens_to_ids(i) for i in stop_tokens]

Yes

@DarkLight1337 I am not sure if I can add these in vl example

[DarkLight1337]

Feel free to update the example!

[DarkLight1337]

@HwwwwwwwH can you try running the example on your end? I'm outside right now.

[HwwwwwwwH]

@HwwwwwwwH can you try running the example on your end? I'm outside right now.

I'm running it now.

[HwwwwwwwH]

@HwwwwwwwH I'd like to know why you chose to use the LlamaModel rather than LlamaForCausalLM as the language model for minicpm2.5. This approach actually changes the model hierarchy, which is not conducive to the support of LoRA.

emmmm, at first I used xxxCausalLM, but then I found that all of the CausalLM deleted the inputs_embeds parameters while kept it in xxModel . And using xxModel is what other VLMs do, so I followed this way.

[HwwwwwwwH]

I got this error when running gguf. And I don't see any dependency in requirements files. Maybe you need add to this?

Traceback (most recent call last):
  File "/data1/a1/vllm_e/examples/offline_inference_vision_language.py", line 10, in <module>
    from vllm import LLM, SamplingParams
  File "/data1/a1/vllm_e/vllm/__init__.py", line 3, in <module>
    from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
  File "/data1/a1/vllm_e/vllm/engine/arg_utils.py", line 7, in <module>
    from vllm.config import (CacheConfig, DecodingConfig, DeviceConfig,
  File "/data1/a1/vllm_e/vllm/config.py", line 11, in <module>
    from vllm.model_executor.layers.quantization import QUANTIZATION_METHODS
  File "/data1/a1/vllm_e/vllm/model_executor/layers/quantization/__init__.py", line 16, in <module>
    from vllm.model_executor.layers.quantization.gguf import GGUFConfig
  File "/data1/a1/vllm_e/vllm/model_executor/layers/quantization/gguf.py", line 3, in <module>
    import gguf
ModuleNotFoundError: No module named 'gguf'

[DarkLight1337]

I got this error when running gguf. And I don't see any dependency in requirements files. Maybe you need add to this?

Traceback (most recent call last):
  File "/data1/a1/vllm_e/examples/offline_inference_vision_language.py", line 10, in <module>
    from vllm import LLM, SamplingParams
  File "/data1/a1/vllm_e/vllm/__init__.py", line 3, in <module>
    from vllm.engine.arg_utils import AsyncEngineArgs, EngineArgs
  File "/data1/a1/vllm_e/vllm/engine/arg_utils.py", line 7, in <module>
    from vllm.config import (CacheConfig, DecodingConfig, DeviceConfig,
  File "/data1/a1/vllm_e/vllm/config.py", line 11, in <module>
    from vllm.model_executor.layers.quantization import QUANTIZATION_METHODS
  File "/data1/a1/vllm_e/vllm/model_executor/layers/quantization/__init__.py", line 16, in <module>
    from vllm.model_executor.layers.quantization.gguf import GGUFConfig
  File "/data1/a1/vllm_e/vllm/model_executor/layers/quantization/gguf.py", line 3, in <module>
    import gguf
ModuleNotFoundError: No module named 'gguf'

It should be in the latest requirements.txt. Maybe need to sync this branch with main.

[DarkLight1337]

@HwwwwwwwH I'd like to know why you chose to use the LlamaModel rather than LlamaForCausalLM as the language model for minicpm2.5. This approach actually changes the model hierarchy, which is not conducive to the support of LoRA.

emmmm, at first I used xxxCausalLM, but then I found that all of the CausalLM deleted the inputs_embeds parameters while kept it in xxModel . And using xxModel is what other VLMs do, so I followed this way.

Yeah, the existing VLMs use the *Model class rather than *ForCausalLM. There have been efforts to change this though to use @Isotr0py 's recipe for loading inner models. (See #7153)

[HwwwwwwwH]

@HwwwwwwwH can you try running the example on your end? I'm outside right now.

It's ok with the examples.

[jeejeelee]

@HwwwwwwwH I'd like to know why you chose to use the LlamaModel rather than LlamaForCausalLM as the language model for minicpm2.5. This approach actually changes the model hierarchy, which is not conducive to the support of LoRA.

emmmm, at first I used xxxCausalLM, but then I found that all of the CausalLM deleted the inputs_embeds parameters while kept it in xxModel . And using xxModel is what other VLMs do, so I followed this way.

Yeah, the existing VLMs use the *Model class rather than *ForCausalLM. There have been efforts to change this though to use @Isotr0py 's recipe for loading inner models. (See #7153)

Sorry for deleting my previous comments by mistake. I will read these codes, thank you.