Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Doc] Update vllm example with metrics #51156

Open
wants to merge 2 commits into
base: master
Choose a base branch
from

Conversation

ronaldosaheki
Copy link

@ronaldosaheki ronaldosaheki commented Mar 7, 2025

Complete the code from issue
#46360

Why are these changes needed?

Completes the example to expose vllm metrics following mixing the examples from issue #46360 and the test https://github.com/vllm-project/vllm/blob/7193774b1ff8603ad5bf4598e5efba0d9a39b436/tests/metrics/test_metrics.py#L342

The metrics prefix changes from vllm: to ray_vllm:

Dashboard https://docs.vllm.ai/en/latest/getting_started/examples/prometheus_grafana.html
Screenshot 2025-03-07 at 10 00 12

Related issue number

Closes #46360

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.

Complete the code from issue
ray-project#46360

Signed-off-by: Ronaldo Saheki <rsaheki@gmail.com>
@edoakes
Copy link
Collaborator

edoakes commented Mar 7, 2025

@akshay-anyscale PTAL!

@jcotant1 jcotant1 added the serve Ray Serve Related Issue label Mar 7, 2025
@GeneDer
Copy link
Contributor

GeneDer commented Mar 11, 2025

Hi @ronaldosaheki thanks for contributing to this doc. We have recently released new llm serving APIs with Ray 2.43.0. You can check them out https://docs.ray.io/en/latest/serve/llm/overview.html We will soon be removing this doc to keep them more focused

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
serve Ray Serve Related Issue
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Serve] Expose internal VLLM metrics
4 participants