[Core] generate from input embeds

[Nan2018]

adds support for passing prompt_embeds to LLM.generate as

llm.generate({"prompt_embeds": input_embeds}, sampling_params)

or

llm.generate(
    [{"prompt_embeds": input_embeds} for input_embeds in inputs_embeds], sampling_params
)

this enables use cases when only the embedding layer is finetuned, and have the same model backend support multiple custom tuned embedding layers

FIX #416
FIX #8323

inspired by #1265 which is very outdated

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which consists a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of default ones by unblocking the steps in your fast-check build on Buildkite UI.

Once the PR is approved and ready to go, please make sure to run full CI as it is required to merge (or just use auto-merge).

To run full CI, you can do one of these:

• Comment /ready on the PR
• Add ready label to the PR
• Enable auto-merge.

🚀

[Nan2018]

@WoosukKwon @ywang96 @robertgshaw2-neuralmagic

the failed tests with
ValueError: Cannot use apply_chat_template() because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating
seems not related to my changes and I can't reproduce it locally.

other than that, this is ready for review

[ywang96]

@WoosukKwon @ywang96 @robertgshaw2-neuralmagic

the failed tests with ValueError: Cannot use apply_chat_template() because tokenizer.chat_template is not set and no template argument was passed! For information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at https://huggingface.co/docs/transformers/main/en/chat_templating seems not related to my changes and I can't reproduce it locally.

other than that, this is ready for review

This is due to a recent change from transformers of deprecating default chat template, and it should have been fixed by #7238. Can you merge your branch with main again?

[DarkLight1337]

if we want to support this, we might need to factor out something like self.process_input , just like we factor out self.sample() . the model runner needs to call the process_input directly, and the model's forward always see hidden_states as input.

Is the idea to call self.process_input outside of torch.compile so that the rest of the operations can still benefit from torch.compile?

[youkaichao]

Is the idea to call self.process_input outside of torch.compile so that the rest of the operations can still benefit from torch.compile?

yes, correct.

[DarkLight1337]

I have moved the .all().item() call outside of get_input_embeds, see if it can support torch.compile now.

[youkaichao]

this line is not compatible with compilation, the shape of inputs_embeds does not match hidden_states .

[DarkLight1337]

So basically, in-place operations aren't allowed?

[youkaichao]

in-place is fine, but we need all operations to have the same batchsize.

[youkaichao]

the requirement for compilation is:

during the lifespan of the model, the computation graph does not change. if hidden_states come from input_ids, it needs to always come from input_ids. it cannot occasionally come from input_ids for some requests, while using input_embeds for some other requests.

see #9946 for example of how to make vision language model compatible with compilation.

in vision-language model, this part of logic is not in compile-region. we only compile language tower.

[lzl-mt]

Excited to use this feature!

[DarkLight1337]

the requirement for compilation is:

during the lifespan of the model, the computation graph does not change. if hidden_states come from input_ids, it needs to always come from input_ids. it cannot occasionally come from input_ids for some requests, while using input_embeds for some other requests.

see #9946 for example of how to make vision language model compatible with compilation.

in vision-language model, this part of logic is not in compile-region. we only compile language tower.

Hmm, I see. I'm quite against changing the semantics of forward to only work on embedded inputs though since that would cause some confusion for developers who are used to working with HF models. Could we instead selectively torch.compile individual methods? (Not sure why the current decorator is hardcoded to compile forward method)

[OswaldoBornemann]

@DarkLight1337 It seems that this feature is almost ready to be used?

[DarkLight1337]

@DarkLight1337 It seems that this feature is almost ready to be used?

It is incompatible with the recent changes to vLLM internals. I'm waiting for the dust to settle for V2 engine before picking this back up.

[lzl-mt]

@DarkLight1337 It seems that this feature is almost ready to be used?

It is incompatible with the recent changes to vLLM internals. I'm waiting for the dust to settle for V2 engine before picking this back up.

Can I run it directly using your branch?

[DarkLight1337]

Can I run it directly using your branch?

No, the current branch is broken because I have already merged in those changes from main.

[DarkLight1337]

I think the latest working commit is 49fe3f7

[lzl-mt]

Can I run it directly using your branch?

No, the current branch is broken because I have already merged in those changes from main.

Umm.. Is there a working commit I can reset back to?

[Nan2018]

Hmm, I see. I'm quite against changing the semantics of forward to only work on embedded inputs though since that would cause some confusion for developers who are used to working with HF models. Could we instead selectively torch.compile individual methods? (Not sure why the current decorator is hardcoded to compile forward method)

Perhaps an option is to move the embedding logic to ModelForCausalLM.forward and compile the Model classes instead of the ModelForCausalLM classes?

[toilaluan]

@DarkLight1337 Do you guys have ETA of this feature?

[DarkLight1337]

@DarkLight1337 Do you guys have ETA of this feature?

No ETA yet, see when V2 re-arch is done.

[DarkLight1337]

I think that this PR will eventually be superseded by the follow-up work to #10374 which should make it easy to support embedding inputs.

[DaoD]

Hi there, if there is any update for this PR?

[DarkLight1337]

Hi there, if there is any update for this PR?

Please read the above comment.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Nan2018.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[CandiedCode]

@DarkLight1337 since #10374 has landed, is this pr going to be updated so that this work can be finished, or is there a new PR you can reference for tracking?

[DarkLight1337]

@DarkLight1337 since #10374 has landed, is this pr going to be updated so that this work can be finished, or is there a new PR you can reference for tracking?

Thanks for your interest. At the moment, V1 isn't stable enough to implement embedding inputs yet. I would point you to #8779 to check the progress, but that RFC is somewhat outdated. Instead, you can search for recent PRs with the [V1] tag.

[CandiedCode]

Thanks for your interest. At the moment, V1 isn't stable enough to implement embedding inputs yet. I would point you to #8779 to check the progress, but that RFC is somewhat outdated. Instead, you can search for recent PRs with the [V1] tag.

Thanks for the reference, @DarkLight1337 . Is it possible also to add this to the roadmap for additional visibility as well?

[ywang96]

Thanks for your interest. At the moment, V1 isn't stable enough to implement embedding inputs yet. I would point you to #8779 to check the progress, but that RFC is somewhat outdated. Instead, you can search for recent PRs with the [V1] tag.

Thanks for the reference, @DarkLight1337 . Is it possible also to add this to the roadmap for additional visibility as well?

Hey @CandiedCode! Thanks for following up on this.

IMO supporting embeddings as input as is does not have technical difficulty but we do want to be careful with the design to make it work with all other features we want to natively support on vLLM, especially now that we're going through re-architecture. I have discussed briefly with @WoosukKwon in #11032 (comment) about it.

In particular, some issues we still need to design and address:

• What happens if a batch has both token ids as input and embeddings as input?
• Prefix caching (Currently we use token ids as hash key)
• Spec decode (we assume draft models to output token id to be accepted by main model)