[Bugfix] add input embedding

[Bryce1010]

adds support for passing prompt_embeds to LLM.generate as

llm.generate({"prompt_embeds": input_embeds}, sampling_params)

or

llm.generate(
    [{"prompt_embeds": input_embeds} for input_embeds in inputs_embeds], sampling_params
)

this enables use cases when only the embedding layer is finetuned, and have the same model backend support multiple custom tuned embedding layers

FIX #416
FIX #8323
FIX #14621

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[DarkLight1337]

Thanks for opening this PR, can you explain what this PR is about and how this is related to #11375?

[Bryce1010]

Thanks for opening this PR, can you explain what this PR is about and how this is related to #11375?

@DarkLight1337 Sorry for referencing the incorrect issue number. Please refer to the following issues for
#416
#8323

[DarkLight1337]

There is a related PR #6869, but your solution can sidestep the problem of missing input token IDs (at the cost of being less efficient).

Can you update this PR with the unit tests in #6869 to see whether this solution works correctly?

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Bryce1010.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[OswaldoBornemann]

@Bryce1010 Hi, when I used your branch, I ran the following case

import torch
import numpy as np

test_prompts = [
    "Who are you",
]

test_embeds = torch.randn(len(test_prompts), 10, 892).to(dtype=torch.float32)  # [batch_size, seq_len, hidden_size]
llm.generate(
    {"prompt_embeds":test_embeds.to(device)}
)

But I came out of the error like RuntimeError: Error in model execution (input dumped to /tmp/err_execute_model_input_20250211-170512.pkl): expected scalar type Float but found BFloat16

[OswaldoBornemann]

@Bryce1010 I changed the test_embeds to test_embeds = torch.randn(len(test_prompts), 10, 896).to(dtype=torch.bfloat16) # [batch_size, seq_len, hidden_size] .

Now the error is AssertionError: Error in model execution (input dumped to /tmp/err_execute_model_input_20250211-172100.pkl):

[qthequartermasterman]

@Bryce1010 @DarkLight1337 Is there an expected time for when this will be updated/merged?

[sidhartha-roy]

I would love to see this feature merged. Is there a timeline for when you expect this to be reviewed and merged @njhill and @alexm-redhat ?

[Bryce1010]

@Bryce1010 do tell me when it's ready for another round of review. Seems that many people are interested in this.

@DarkLight1337 Thanks for your response! The code has been rebased. Could you please review it and let me know if any further changes are needed before merging into the main branch?

[DarkLight1337]

Can you fix the pre-commit errors?

[DarkLight1337]

There are still errors, please fix them

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @Bryce1010.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[Reek12138]

I would like to ask when this feature will be merged into the main branch. Once merged, will it support Python 3.8?

[DarkLight1337]

Support for Python 3.8 in general has been dropped a while ago as it has reached EOL.

[liangwythu]

Would love to see this one merged!

[qthequartermasterman]

@Bryce1010 @DarkLight1337 Since it has been a few weeks without an update, and I would love to have this feature, I took the initiative to rebase this PR onto main and fix the pre-commit errors as previously asked. I can't commit to this branch, since it's owned by @Bryce1010, so these updates are in #15428.

[DarkLight1337]

@Bryce1010 @DarkLight1337 Since it has been a few weeks without an update, and I would love to have this feature, I took the initiative to rebase this PR onto main and fix the pre-commit errors as previously asked. I can't commit to this branch, since it's owned by @Bryce1010, so these updates are in #15428.

That sounds good, thanks for picking this up!

[leonardtang]

@Bryce1010 @DarkLight1337 Since it has been a few weeks without an update, and I would love to have this feature, I took the initiative to rebase this PR onto main and fix the pre-commit errors as previously asked. I can't commit to this branch, since it's owned by @Bryce1010, so these updates are in #15428.

My goat

[DarkLight1337]

Closing in favor of #15428