[Feature]: Support similar API, such as /health_generate

🚀 The feature, motivation and pitch

Currently, there is a health check endpoint called /health. Could we add another endpoint /health_generate?
The purpose of this endpoint would be to have the model perform inference to generate a token, thereby confirming that the service can perform inference normally and avoiding situations where it appears alive but is actually unresponsive.

Health Check
/health: Check the health of the server.
/health_generate: Check the health of the server by generating one token.

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[xsank]

Add a engine core client rpc interface maybe is enough.

[lengrongfu]

Have you encountered situations where it appears alive but is actually unresponsive. ?

Last week, we encountered an issue in production: the product received HTTP 500 errors when invoking the model inference API. Inspection showed the container instances in a Running state. Upon investigation, we found that one instance was failing during inference due to a CUDA-related error. At the time, the /health endpoint was configured as the container probe; when invoked from inside the container, it returned HTTP 200, which is why the probe did not surface the problem and the instance remained marked healthy. This resulted in the incident. The deployment was running vLLM 0.7, and the exception occurred only once after the service had been running for an extended period. Even so, this suggests the health-check endpoint should be strengthened to detect such failures. We regret that, due to the need to urgently remediate the production incident at the time, the logs were not retained.

[lengrongfu]

This is usually caused by a temporary GPU failure. If you add a /health_generate to perform model inference at any time, even if only a token of one length is generated, it will consume resources.

[cjackal]

Have you encountered situations where it appears alive but is actually unresponsive. ?

In fact this happens in long-running server in pretty bad way, c.f. #18786. vLLM backend can be irresponsive while the FastAPI frontend is "healthy" (in terms of /health) due to kv cache garbage collect deadlock or workers silently disconnected in PD disaggregation.

I also add a readiness probe of OP's type on my server like the following:

readinessProbe:
  exec:
    command:
    - curl
    - -XPOST
    - https://localhost:8080/v1/completions
    - -H
    - "Content-Type: application/json"
    - -d
    - '{"prompt": "Hi.", "max_tokens": 5}'
  periodSeconds: 5

and it would be convenient if vllm has it battery-included. In production serving, making a short request every few seconds for an active healthcheck is worth the computing cost, without it the server can silently fail for a long time which ends up paying a lot of money and human effort.

[simon-mo]

I think it is useful indeed. I have seen similar production service using this approach.

One design concern is whether we just call it /health?generate=true

[lengrongfu]

I think it is useful indeed. I have seen similar production service using this approach.

One design concern is whether we just call it /health?generate=true

agree, because except generate, there are multi model、embed、score etc. so current we can only add generate operate.

/assign