[Feature]: Additional metrics to enable better autoscaling / load balancing of vLLM servers in Kubernetes

[achandrasekar]

🚀 The feature, motivation and pitch

vLLM provides some metrics on model performance and load today which are very useful. There are a few metrics that are missing today which if added can make it easier for any orchestrators like Kubernetes to provide better support for autoscaling vLLM servers or distribute load between multiple vLLM servers more efficiently. We have a proposal in the Kubernetes Serving WG to add these additional metrics to popular model servers. We want to add these to vLLM as well.

Google doc link to the proposal which has the set of metrics we want to add and the reasoning behind it - https://docs.google.com/document/d/1SpSp1E6moa4HSrJnS4x3NpLuj88sMXr2tbofKlzTZpk/edit?usp=sharing&resourcekey=0-ob5dR-AJxLQ5SvPlA4rdsg. (Please request access if you are not able to view it)

Listing the metrics that we've identified to include in vLLM:

Metric Name	Type	Unit
model_load_time	Counter	Seconds
time_per_output_token_per_batch_size	Histogram	Milliseconds
request_wait_time (total time - time spent on inference)	Histogram	Milliseconds
request_queue_time	Histogram	Milliseconds
max_token_capacity	Counter	Tokens
time_per_prefill_token	Histogram	Milliseconds
total_tokens_in_current_batch	Gauge	Tokens
estimated_max_prefill_tokens_per_second	Gauge	Tokens
estimated_max_batch_before_compute_saturation	Gauge	Tokens
request_input_length	Histogram	Tokens
request_output_length	Histogram	Tokens
request_with_evicted_tokens	Counter	Count
total_evicted_tokens	Counter	Tokens

It would be good to add these metrics both for observability as well as efficient orchestration.

Alternatives

No response

Additional context

cc @WoosukKwon @robertgshaw2-neuralmagic

[davidgxue]

+1 would be great to have these!!!

[simon-mo]

They do look useful to me! Looking forward to the contribution! Also adding @ywang96 for awareness.

[ywang96]

This is great - thank you @achandrasekar!

Note that a few metrics in the list (e.g, request_input_length, request_output_length) are already supported by vLLM, so it would be great to consolidate them in your upcoming contribution. I do think we're currently missing a metric related to queue time, which is very important to decide when to scale up inference services.

[robertgshaw2-redhat]

@ywang96 couple of these are implemented in a branch. Will triage and help get merged

[gyliu513]

@achandrasekar do you mind bring this to otel semantic convention team meeting and discuss there as well? We are working for LLM Semantic Convetion, and this is an area that we do not have now.

An related issue in otel semantic convention open-telemetry/semantic-conventions#1079

Here is the meeting info https://docs.google.com/document/d/1EKIeDgBGXQPGehUigIRLwAUpRGa7-1kXB736EaYuJ2M/edit#heading=h.ylazl6464n0c

[achandrasekar]

@gyliu513 yes, thanks for bringing this up. We discussed this in the last LLM Semantic Conventions meeting. I've created an issue(open-telemetry/semantic-conventions#1102) and an initial PR(open-telemetry/semantic-conventions#1103) to create the server metrics. Let's collaborate there. We can also discuss in the next semconv meeting.

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!

[github-actions]

This issue has been automatically closed due to inactivity. Please feel free to reopen if you feel it is still relevant. Thank you!

[achandrasekar]

A lot of the metrics mentioned here have been added by folks like:

• request_queue_time (vllm:time_in_queue_requests)
• request_input_length (vllm:request_prompt_tokens)
• request_output_length (vllm:request_generation_tokens)
• request_wait_time (vllm:request_queue_time_seconds)

A few more have been added which provide additional insights, but not exactly what we are looking for with respect to autoscaling / load balancing like:

• vllm:request_prefill_time_seconds (we need time_per_prefill_token instead of at the request level)
• vllm:iteration_tokens_total (this is a histogram of tokens processed per iteration (in the batch), but we need a gauge of this so we can make decisions on current batch state)

Updated table below on what we still need:

Metric Name	Type	Unit
model_load_time	Guage	Seconds
max_token_capacity	Gauge	Tokens
time_per_prefill_token	Histogram	Milliseconds
total_tokens_in_current_batch	Gauge	Tokens
total_tokens_in_queue (prefill + decode)	Gauge	Tokens
request_with_evicted_tokens	Counter	Count
total_evicted_tokens	Counter	Tokens

cc @annapendleton who is looking at some of this.