Feature/vllm/input embedding completion api

[Nan2018]

adds support for passing prompt_embeds as b64 encoded bytes to the completions api.

Start the server with

VLLM_USE_V1=0 vllm serve HuggingFaceH4/zephyr-7b-beta

query example:

url = "http://localhost:8000/v1/completions"

prompt_embeds = []
for input_embeds in inputs_embeds: # inputs_embeds is list of 2d tensors of shape (seq_len, embed_dim)
    buff = io.BytesIO()
    torch.save(input_embeds.detach().cpu(), buff)
    prompt_embeds.append(b64encode(buff.getvalue()).decode("utf-8"))

resps = requests.post(
    url,
    json={
        "model": request_model,
        "prompt_embeds": prompt_embeds,
    },
)

# or with openai client

completions = client.completions.create(
    model=request_model,
    prompt="",
    extra_body={"prompt_embeds": prompt_embeds},
)

Note

this does not work with lora or prompt adapters

[Nan2018]

I spent the better part of the afternoon trying to pin point exactly where the failure is occuring. It seems to be only affecting vLLM instances launched via subprocess (in particular via tests.utils.RemoteOpenAIServer in the tests). Launching a vLLM instance normally with the exact same environment variables and arguments works perfectly...

@DarkLight1337 any ideas about this? Do you think it is a blocker for this pr?

[DarkLight1337]

I'm fine with not supporting LoRA for now, unless LoRA is a very important use case for this.

[DarkLight1337]

Can you add an example script to the documentation for both offline and online inference?

[qthequartermasterman]

I'm fine with not supporting LoRA for now, unless LoRA is a very important use case for this.

I don't think this is an important use case at this time. I think it only came up because the existing completion tests checked for LoRA compatibility and @Nan2018 tried to use both of them together.

Can you add an example script to the documentation for both offline and online inference?

I added the docs/source/serving/prompt_embeds.md file. I don't need to add anything to add the page to the sphinx site, correct? It'll automatically find it? I couldn't find anything in the sphinx documentation explicitly mentioning the other files in that directory.

[DarkLight1337]

Yeah they should be added automatically

[qthequartermasterman]

@DarkLight1337 It looks like docs build timed out. All of the fast checks are passing. I do think this PR is ready for review.

Thanks for your help with this!

[DarkLight1337]

Regarding the subprocess issue, it may be related to #18308 (comment)

[DarkLight1337]

Let's merge this first though

[CandiedCode]

@DarkLight1337 will this make it into the v0.9.0 release?

[DarkLight1337]

Yes

[Potabk]

What is this lora module used for?