[Performance]: V0.6.0 version of the model, benchmarking found that the number of successful responses accounted for half of the number of requests, which is why

[Amber-Believe]

Proposal to improve performance

No response

Report of performance regression

No response

Misc discussion on performance

No response

Your current environment (if you think it is necessary)

The output of `python collect_env.py`

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[Amber-Believe]

In the v0.6.0 model, the total number of requests is 1000, -- the request-rate is 100, 1) only half of the requests are successfully responded to, why do so many fail? At the same time, check the output length, found that the output length is only 502 non-empty replies, 2) That is to say, the successful response is also None, what is the reason.

[youkaichao]

cc @KuntaiDu

[youkaichao]

@Amber-Believe make sure you follow #8176

[Amber-Believe]

@Amber-Believe确保你关注了#8176

Hello, so the meaning here is that the number of successful responses is only half due to the problem of the multi-step scheduler (--num-scheduler-steps 10)?

[KuntaiDu]

IIRC vLLM's request buffer size is not large enough to hold 1000 requests. Try use 500 requests instead.

[KuntaiDu]

And also would be great if you could use latest vLLM version, there are several stabiliy-related bug fixes there.

[Amber-Believe]

Okay, I'll try the new version of vllm