[Usage]: embeddings API when task is generate

[Harsh-Sensei]

Your current environment

Collecting environment information...
PyTorch version: 2.5.1+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Springdale Linux release 8.8 (Modena) (x86_64)
GCC version: (GCC) 8.5.0 20210514 (Red Hat 8.5.0-18)
Clang version: Could not collect
CMake version: version 3.20.2
Libc version: glibc-2.28

Python version: 3.12.8 | packaged by Anaconda, Inc. | (main, Dec 11 2024, 16:31:09) [GCC 11.2.0] (64-bit runtime)
Python platform: Linux-4.18.0-477.13.1.el8_8.x86_64-x86_64-with-glibc2.28
Is CUDA available: True
CUDA runtime version: 12.5.82
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA RTX A6000
GPU 1: NVIDIA RTX A6000
GPU 2: NVIDIA RTX A6000
GPU 3: NVIDIA RTX A6000
GPU 4: NVIDIA RTX A6000
GPU 5: NVIDIA RTX A6000
GPU 6: NVIDIA RTX A6000
GPU 7: NVIDIA RTX A6000

Nvidia driver version: 555.42.06
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:        x86_64
CPU op-mode(s):      32-bit, 64-bit
Byte Order:          Little Endian
CPU(s):              128
On-line CPU(s) list: 0-127
Thread(s) per core:  2
Core(s) per socket:  32
Socket(s):           2
NUMA node(s):        2
Vendor ID:           AuthenticAMD
CPU family:          23
Model:               49
Model name:          AMD EPYC 7502 32-Core Processor
Stepping:            0
CPU MHz:             3319.148
CPU max MHz:         2500.0000
CPU min MHz:         1500.0000
BogoMIPS:            4999.77
Virtualization:      AMD-V
L1d cache:           32K
L1i cache:           32K
L2 cache:            512K
L3 cache:            16384K
NUMA node0 CPU(s):   0-31,64-95
NUMA node1 CPU(s):   32-63,96-127
Flags:               fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd amd_ppin arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif v_spec_ctrl umip rdpid overflow_recov succor smca sme sev sev_es

Versions of relevant libraries:
[pip3] numpy==1.26.3
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-ml-py==12.560.30
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.2.0
[pip3] torch==2.5.1
[pip3] torch-model-archiver==0.12.0
[pip3] torch-workflow-archiver==0.2.15
[pip3] torchserve==0.12.0
[pip3] torchvision==0.20.1+cu121
[pip3] transformers==4.47.1
[pip3] triton==3.1.0
[conda] numpy                     1.26.3                   pypi_0    pypi
[conda] nvidia-cublas-cu12        12.4.5.8                 pypi_0    pypi
[conda] nvidia-cuda-cupti-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-nvrtc-cu12    12.4.127                 pypi_0    pypi
[conda] nvidia-cuda-runtime-cu12  12.4.127                 pypi_0    pypi
[conda] nvidia-cudnn-cu12         9.1.0.70                 pypi_0    pypi
[conda] nvidia-cufft-cu12         11.2.1.3                 pypi_0    pypi
[conda] nvidia-curand-cu12        10.3.5.147               pypi_0    pypi
[conda] nvidia-cusolver-cu12      11.6.1.9                 pypi_0    pypi
[conda] nvidia-cusparse-cu12      12.3.1.170               pypi_0    pypi
[conda] nvidia-ml-py              12.560.30                pypi_0    pypi
[conda] nvidia-nccl-cu12          2.21.5                   pypi_0    pypi
[conda] nvidia-nvjitlink-cu12     12.4.127                 pypi_0    pypi
[conda] nvidia-nvtx-cu12          12.4.127                 pypi_0    pypi
[conda] pyzmq                     26.2.0                   pypi_0    pypi
[conda] torch                     2.5.1                    pypi_0    pypi
[conda] torch-model-archiver      0.12.0                   pypi_0    pypi
[conda] torch-workflow-archiver   0.2.15                   pypi_0    pypi
[conda] torchserve                0.12.0                   pypi_0    pypi
[conda] torchvision               0.20.1+cu121             pypi_0    pypi
[conda] transformers              4.47.1                   pypi_0    pypi
[conda] triton                    3.1.0                    pypi_0    pypi
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.6.5
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV4     NODE    NODE    SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU1    NV4      X      NODE    NODE    SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU2    NODE    NODE     X      NV4     SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU3    NODE    NODE    NV4      X      SYS     SYS     SYS     SYS     0-31,64-95      0               N/A
GPU4    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE    32-63,96-127    1               N/A
GPU5    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE    32-63,96-127    1               N/A
GPU6    SYS     SYS     SYS     SYS     NODE    NODE     X      NV4     32-63,96-127    1               N/A
GPU7    SYS     SYS     SYS     SYS     NODE    NODE    NV4      X      32-63,96-127    1               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

LD_LIBRARY_PATH=/zfsauton2/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/cv2/../../lib64:/usr/local/cuda/lib64:
MKL_NUM_THREADS=1
OMP_NUM_THREADS=1
CUDA_MODULE_LOADING=LAZY

How would you like to use vllm

Is the v1/embeddings API supposed to work when the task is set to "generate". Upon "POST"ing request:
{
"model": "neuralmagic/Mistral-7B-Instruct-v0.3-quantized.w8a16",
"input": "Never have I ever",
}

when the vllm server is running with following command:
CUDA_VISIBLE_DEVICES=3 python -m vllm.entrypoints.openai.api_server --model "neuralmagic/Mistral-7B-Instruct-v0.3-quantized.w8a16" --quantization gptq --dtype half --gpu-memory-utilization 0.6 --chat-template mistral_chat_template.jinja

I get the following error:
ne 867, in check_generation_prompt
if data.get("continue_final_message") and data.get(
^^^^^^^^
AttributeError: 'bytes' object has no attribute 'get'

I suppose, I would have to run another vllm engine for embedding task?

Thanks!

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

Yes, you need to have a separate vLLM instance with --task embed.

[Harsh-Sensei]

Alright, and is there support for any huggingface model or just "intfloat/e5-mistral-7b-instruct" cause I am still getting the same error after running another vllm instance?

[DarkLight1337]

Please refer to the Supported Models page for a list of supported models. Since v0.6.5, vLLM attempts to convert models outside of this list for embedding task, but the model is not guaranteed to work properly. Can you show a stack trace of the error?

[Harsh-Sensei]

Yeah, here is the full error stack :

Traceback (most recent call last):
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/middleware/base.py", line 185, in __call__
    with collapse_excgroups():
         ^^^^^^^^^^^^^^^^^^^^
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
    raise exc
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/middleware/base.py", line 187, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/api_server.py", line 543, in add_request_id
    response = await call_next(request)
               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/middleware/base.py", line 163, in call_next
    raise app_exc
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/middleware/base.py", line 149, in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/fastapi/routing.py", line 291, in app
    solved_result = await solve_dependencies(
                    ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 666, in solve_dependencies
    ) = await request_body_to_args(  # body_params checked above
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 898, in request_body_to_args
    v_, errors_ = _validate_value_with_model_field(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 706, in _validate_value_with_model_field
    v_, errors_ = field.validate(value, values, loc=loc)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/fastapi/_compat.py", line 129, in validate
    self._type_adapter.validate_python(value, from_attributes=True),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/pydantic/type_adapter.py", line 412, in validate_python
    return self.validator.validate_python(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/home/hshah2/miniconda/envs/vllm/lib/python3.12/site-packages/vllm/entrypoints/openai/protocol.py", line 867, in check_generation_prompt
    if data.get("continue_final_message") and data.get(
       ^^^^^^^^
AttributeError: 'bytes' object has no attribute 'get'

[DarkLight1337]

Are you setting --task embed in the command to serve the model?

[Harsh-Sensei]

Are you setting --task embed in the command to serve the model?

Yep, here is the full command :

CUDA_VISIBLE_DEVICES=6 python -m vllm.entrypoints.openai.api_server --model "neuralmagic/Mistral-7B-Instruct-v0.3-quantized.w8a16" --quantization gptq --dtype half --gpu-memory-utilization 0.6 --chat-template mistral_chat_template.jinja --task embed --port 8001

[DarkLight1337]

I think you need to set the content type of the request to json. See #11171

[Harsh-Sensei]

Crap totally slipped out of my mind...thank you so much! Working now!