[Model] VLM2Vec, the first multimodal embedding model in vLLM

[DarkLight1337]

Support VLM2Vec embedding model by TIGER-Lab.

This is a low-hanging fruit as the model architecture is exactly the same as Phi3V.

Future works, in order of priority:

1. Add CLI option to specify whether to use a model for generation or embedding so we don't have to hardcode the model name when the same model architecture can be used for both.
2. Add multimodal embedding API for OpenAI-compatible server
3. More multimodal embedding models, e.g. E5-V which can be similarly supported with our existing LLaVA-NeXT implementation

@jeejeelee are you available to help provide LoRA support for this model after this PR?

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[jeejeelee]

Okay, I'd be happy to try supporting LoRA.

[DarkLight1337]

I'm preemptively moving them into the same file to be consistent with the upcoming BERT PR. (#9056)

[wenhuchen]

Nice work! There were some typos in our paper. We actually used last token instead eos token representation. I saw that you already used the last token as the representation, which is the correct implementation.

[Isotr0py]

LGTM. Especially the implementation correctness is checked by model vendors :)

[Isotr0py]

Ooops, seems that the new added vision embedding test didn't include in test-pipeline:

vllm/.buildkite/test-pipeline.yaml

Lines 337 to 349 in 7abba39

	- label: Other Models Test # 6min
	#mirror_hardwares: [amd]
	source_file_dependencies:
	- vllm/
	- tests/models/embedding/language
	- tests/models/encoder_decoder/language
	- tests/models/encoder_decoder/vision_language
	commands:
	- pytest -v -s models/embedding/language
	- pytest -v -s models/encoder_decoder/language
	- pytest -v -s models/encoder_decoder/vision_language

We might need to open another PR to include it.

[DarkLight1337]

Ooops, seems that the new added vision embedding test didn't include in test-pipeline:

vllm/.buildkite/test-pipeline.yaml

Lines 337 to 349 in 7abba39

	- label: Other Models Test # 6min
	#mirror_hardwares: [amd]
	source_file_dependencies:
	- vllm/
	- tests/models/embedding/language
	- tests/models/encoder_decoder/language
	- tests/models/encoder_decoder/vision_language
	commands:
	- pytest -v -s models/embedding/language
	- pytest -v -s models/encoder_decoder/language
	- pytest -v -s models/encoder_decoder/vision_language

We might need to open another PR to include it.

Nice catch, I have opened #9406

[jvlinsta]

What will it take / when would it be expected for online engine support for this? ^^

[DarkLight1337]

What will it take / when would it be expected for online engine support for this? ^^

Maybe sometime in the next two weeks.

[DarkLight1337]

What will it take / when would it be expected for online engine support for this? ^^

Maybe sometime in the next two weeks.

Quick heads-up that it's done now!

[wenhuchen]

When will the LoRA version be online? Actually, the LoRA version works better.

[DarkLight1337]

When will the LoRA version be online? Actually, the LoRA version works better.

Currently, vLLM only supports LoRA for the language backbone of VLMs - some re-arch work is necessary to extend this to the vision encoder. @jeejeelee do you have a timeframe regarding this?

[wenhuchen]

When will the LoRA version be online? Actually, the LoRA version works better.

Currently, vLLM only supports LoRA for the language backbone of VLMs - some re-arch work is necessary to extend this to the vision encoder. @jeejeelee do you have a timeframe regarding this?

I guess you meant "vLLM doesn't support LoRA" instead of "vLLM only support LoRA"?

[DarkLight1337]

I guess you meant "vLLM doesn't support LoRA" instead of "vLLM only support LoRA"?

I mean that vLLM supports LoRA on the language backbone, but not LoRA on the vision encoder of VLMs.

[wenhuchen]

Got it. Thanks for the explanation!