[MISC] Add lora requests to metrics #9477
Conversation
coolkp
requested review from
WoosukKwon,
alexm-redhat,
comaniac,
njhill,
youkaichao and
zhuohan123
as code owners
|
👋 Hi! Thank you for contributing to the vLLM project. Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can do one of these:
🚀 |
| max_lora_stat = "0" | ||
| if self.lora_config: | ||
| max_lora_stat = str(self.lora_config.max_loras) |
There was a problem hiding this comment.
This seems always fixed? In this case can we don't dump this value?
There was a problem hiding this comment.
across multiple deployments its hard to get this value, helps determine how many loras can be fitted on the server. You are right, its definitely static right now, initialised at runtime and thats it. I considered moving it to separate info metric like the cache config info. But I think in future there maybe value in enabling dynamic adjustment of max lora, like base_model which is static right now.
coolkp
changed the title
Co-authored-by: Kunjan Patel <kunjanp_google_com@vllm.us-central1-a.c.kunjanp-gke-dev-2.internal> Signed-off-by: Alvant <alvasian@yandex.ru>
Co-authored-by: Kunjan Patel <kunjanp_google_com@vllm.us-central1-a.c.kunjanp-gke-dev-2.internal> Signed-off-by: Amit Garg <mitgarg17495@gmail.com>
Co-authored-by: Kunjan Patel <kunjanp_google_com@vllm.us-central1-a.c.kunjanp-gke-dev-2.internal> Signed-off-by: qishuai <ferdinandzhong@gmail.com>
Co-authored-by: Kunjan Patel <kunjanp_google_com@vllm.us-central1-a.c.kunjanp-gke-dev-2.internal> Signed-off-by: Sumit Dubey <sumit.dubey2@ibm.com>
The current `vllm:lora_requests_info` Gauge is somewhat similar to an
Info metric (like cache_config_info) except the value is the current
wall-clock time, and is updated every iteration.
The label names used are:
- running_lora_adapters: a list of adapters with running requests,
formatted as a comma-separated string.
- waiting_lora_adapters: similar, except listing adapters with
requests waiting to be scheduled.
- max_lora - the static "max number of LoRAs in a single batch."
configuration.
It looks like this:
```
vllm:lora_requests_info{max_lora="1",running_lora_adapters="",waiting_lora_adapters=""} 1.7395575657589855e+09
vllm:lora_requests_info{max_lora="1",running_lora_adapters="test-lora",waiting_lora_adapters=""} 1.7395575723949368e+09
vllm:lora_requests_info{max_lora="1",running_lora_adapters="test-lora",waiting_lora_adapters="test-lora"} 1.7395575717647147e+09
```
I can't really make much sense of this. Encoding a running/waiting
status for multiple adapters in a comma-separated string seems quite
misguided - we should use labels to distinguish between per-adapter
counts instead:
```
vllm:num_lora_requests_running{lora_name="test-lora",model_name="meta-llama/Llama-3.1-8B-Instruct"} 8.0
vllm:num_lora_requests_waiting{lora_name="test-lora",model_name="meta-llama/Llama-3.1-8B-Instruct"} 7.0
```
This was added in vllm-project#9477 and there is at least one known user. If we
revisit this design and deprecate the old metric, we should reduce the
need for a significant deprecation period by making the change in v0
also and asking this project to move to the new metric.
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Kunjan Patel <kunjanp_google_com@vllm.us-central1-a.c.kunjanp-gke-dev-2.internal> Signed-off-by: LeiWang1999 <leiwang1999@outlook.com>
2 participants
This PR adds lora requests to log metrics. The metrics will be logged only when the lora is enabled. Here is an example:
We plan to leverage this information for routing decisions in https://github.com/kubernetes-sigs/llm-instance-gateway