Skip to content

Commit

Permalink
TGISStatLogger: fix stats usage (#25)
Browse files Browse the repository at this point in the history
Cherry-pick of fix commit 6100f4b from ODH:
#17

---------

Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Co-authored-by: Daniele Trifirò <dtrifiro@redhat.com>
  • Loading branch information
tjohnson31415 and dtrifiro authored May 8, 2024
1 parent c737a7a commit 06d9876
Showing 1 changed file with 8 additions and 10 deletions.
18 changes: 8 additions & 10 deletions vllm/tgis_utils/metrics.py
Original file line number Diff line number Diff line change
Expand Up @@ -116,21 +116,19 @@ def log(self, stats: Stats) -> None:
self._vllm_stat_logger.log(stats)

# Then log TGIS specific ones
self.tgi_queue_size.set(stats.num_waiting + stats.num_swapped)
self.tgi_batch_current_size.set(stats.num_running)
self.tgi_queue_size.set(stats.num_waiting_sys + stats.num_swapped_sys)
self.tgi_batch_current_size.set(stats.num_running_sys)

for ttft in stats.time_to_first_tokens:
for ttft in stats.time_to_first_tokens_iter:
self.tgi_batch_inference_duration.labels({
"method": "prefill"
}).observe(ttft)
for tpot in stats.time_per_output_tokens:
for tpot in stats.time_per_output_tokens_iter:
self.tgi_batch_inference_duration.labels({
"method": "next_token"
}).observe(tpot)

# These metrics depend on open PR: https://github.com/vllm-project/vllm/pull/2764
if hasattr(stats, "num_prompt_tokens_lst"):
for input_len in stats.num_prompt_tokens_lst:
self.tgi_request_input_length.observe(input_len)
for output_len in stats.num_generation_tokens_lst:
self.tgi_request_generated_tokens.observe(output_len)
for input_len in stats.num_prompt_tokens_requests:
self.tgi_request_input_length.observe(input_len)
for output_len in stats.num_generation_tokens_requests:
self.tgi_request_generated_tokens.observe(output_len)

0 comments on commit 06d9876

Please sign in to comment.