[Usage]: How to use LLM.classify(...) through OpenAI endpoint?

[ccdv-ai]

Your current environment

vLLM 0.8.5

How would you like to use vllm

I need to run a classification model, extract probs and serve it through OpenAI API.
I know that it is possible to do offline:

from vllm import LLM

llm = LLM(model="jason9693/Qwen2.5-1.5B-apeach", task="classify")
(output,) = llm.classify("Hello, my name is")

probs = output.outputs.probs
print(f"Class Probabilities: {probs!r} (size={len(probs)})")

My task is very similar to a reranker (score) task but with multiple classes (query + document).
How do you do you this using OpenAI?

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[aarnphm]

You need to first run serve: VLLM_USE_V1=0 vllm serve jason9693/Qwen2.5-1.5B-apeach --task classify. (not sure if this is already support on v1, might need to check)

then you can access the fastapi app at localhost:8000/docs

then just send in a curl request at /v1/classify

edit: I'm wrong, there is a PR for this on server #17032