[Feature]: Improvements to front-end embedding response

[uasan]

🚀 The feature, motivation and pitch

I will describe here in more detail what was said here.

We use the OpenAI interface from vLLM in our ETL service, our main task is batch generation and vector matching.

When processing large batch responses, we notice the collector overhead and the resource consumption of receiving uncompressed JSON text, which contains a vector of values ​​in Base64 strings. This results in significant memory usage and garbage collection, and also puts a strain on the network.

I'd like to suggest several improvements.

1. Response compression if the client sends the 'accept-encoding' header: 'zstd, gzip'

2. This is quite radical, but would be very effective. Add a new request parameter, for example, is_binary_response. If true, return the response not as JSON, but simply as a binary tuple. We always know the tuple length when making a request, so it will be very easy to parse a binary response with this structure: Index: UInt16, tokens: UInt32, vector: FixedList

3. This is a small thing, but still, endian, our ETL service often doesn't work with the vector, it just saves it to another database, for example, in Postgres, whose binary protocol requires values ​​to always be transmitted in bigendian, we have to convert the vector to the required endian, this is not difficult, but it will be useful if there is a query parameter - endian, which will control the endian of response.

Thanks.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[noooop]

1. Response compression if the client sends the 'accept-encoding' header: 'zstd, gzip'

Use Response compression to transmit binary files that don't need base64, cool. base64 is very inefficient

I wouldn't think of this hacky method before seeing this issue.

cc @DarkLight1337 @maxdebayser

[lambdaq]

Add a new request parameter, for example, is_binary_response

There is already a header for that: Accept

why not take a leap forward, adopt binary serialization?

protobuf, cap n' proto or this

https://github.com/cloudflare/capnweb

it has ArrayBuffer with UInt32

Also consider other options like Arrow, which allows zero-copy benifits.

[maxdebayser]

In the production setups that I've seen, vLLM is typically not exposed directly to the user but is deployed behind a proxy of some kind. I think gzip compression should be handled at the proxy level, together with other concerns like https, WAF, routing, load balancing and so on.

But sending the arrays in a binary is a good idea I think, especially for token embedding where we'll be sending a lot of data over the wire.

[uasan]

The earlier you compress the data, the more beneficial it is.
vLLM, it's easier to send compressed data to a network socket because compressed data has fewer bytes and reduces network backpressure.

Sending an uncompressed response is best done only to unixsocket when the client is on a single host machine. This, by the way, is also a good production use case – it utilizes host resources well and reduces outgoing traffic.

[maxdebayser]

The earlier you compress the data, the more beneficial it is.

I'm not sure about that. Typically the datacenter network will be very fast while the connection from the ingress proxy to the client will be much slower. So it makes a lot of sense to compress data to send it over the network, but on a fast network it might actually take more time to compress. Especially because if you compress you can't do zero-copy transfers directly from the memory buffer to the NIC.

Also, the API server will use CPU for compression that it could be using for tokenization and other workloads. vLLM shouldn't be doing stuff that nginx, haproxy, envoy and others are heavily optimized to do. Unless there is very compelling evidence, of course.

[noooop]

In the production setups that I've seen, vLLM is typically not exposed directly to the user but is deployed behind a proxy of some kind. I think gzip compression should be handled at the proxy level, together with other concerns like https, WAF, routing, load balancing and so on.

But sending the arrays in a binary is a good idea I think, especially for token embedding where we'll be sending a lot of data over the wire.

+1

And float data are almost incompressible.

[uasan]

And float data are almost incompressible.

This is a strong argument not to compress, if this is so, then I agree, there is no point

[uasan]

@noooop When is the release of this feature planned?

[noooop]

vllm typically releases every four weeks, and now it's the third week of this release cycle.

If you can help test this feature before the release, that would be best.

Installing the dev version is just as simple as installing the official version.

https://docs.vllm.ai/en/latest/getting_started/installation/gpu.html#set-up-using-python-only-build-without-compilation

git clone https://github.com/vllm-project/vllm.git
cd vllm
VLLM_USE_PRECOMPILED=1 uv pip install --editable .

[noooop]

If you can provide an example of using binary embedding response with Node.js or any other language, that would be great.