[Serve][1/n] Add autoscaling prometheus metrics #59220
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: abrar <abrar@anyscale.com>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces several new Prometheus metrics to improve the observability of Ray Serve's autoscaling functionality. The new metrics cover autoscaling decisions, policy execution time, and metric reporting delays, which will be very helpful for debugging and monitoring. The implementation looks solid, and it's great to see comprehensive tests added for the new metrics. I have one suggestion to refactor a small piece of duplicated code for better maintainability.
Signed-off-by: abrar <abrar@anyscale.com>
Signed-off-by: abrar <abrar@anyscale.com>
Signed-off-by: abrar <abrar@anyscale.com>
abrarsheikh
changed the title
ray-gardener
bot
added
serve
Signed-off-by: abrar <abrar@anyscale.com>
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
harshit-anyscale
approved these changes
Dec 10, 2025
Signed-off-by: abrar <abrar@anyscale.com>
Signed-off-by: abrar <abrar@anyscale.com>
| │ ┌─────────────────────────────────────────────────────────────────────┐ │ | ||
| │ │ REPLICA │ │ | ||
| │ │ │ │ | ||
| │ │ ○ ray_serve_replica_processing_queries (while processing) │ │ |
Contributor
There was a problem hiding this comment.
would be good to rename this (backwards compatible way) to ray_serve_replica_num_ongoing_requests to align with terminology of autoscaling and other such metrics
| |--------|------|------|-------------| | ||
| | `ray_serve_handle_request_counter_total` **[D]** | Counter | `handle`, `deployment`, `route`, `application` | Total number of requests processed by this `DeploymentHandle`. | | ||
| | `ray_serve_num_router_requests_total` **[H]** | Counter | `deployment`, `route`, `application`, `handle`, `actor_id` | Total number of requests routed to a deployment. | | ||
| | `ray_serve_deployment_queued_queries` **[H]** | Gauge | `deployment`, `application`, `handle`, `actor_id` | Current number of requests waiting to be assigned to a replica. High values indicate backpressure. | |
Contributor
There was a problem hiding this comment.
would be good to rename this too .. ray_serve_router_num_queued_requests
| | `ray_serve_handle_request_counter_total` **[D]** | Counter | `handle`, `deployment`, `route`, `application` | Total number of requests processed by this `DeploymentHandle`. | | ||
| | `ray_serve_num_router_requests_total` **[H]** | Counter | `deployment`, `route`, `application`, `handle`, `actor_id` | Total number of requests routed to a deployment. | | ||
| | `ray_serve_deployment_queued_queries` **[H]** | Gauge | `deployment`, `application`, `handle`, `actor_id` | Current number of requests waiting to be assigned to a replica. High values indicate backpressure. | | ||
| | `ray_serve_num_ongoing_requests_at_replicas` **[H]** | Gauge | `deployment`, `application`, `handle`, `actor_id` | Current number of requests assigned and sent to replicas but not yet completed. | |
Contributor
There was a problem hiding this comment.
I think this metric name should explicitly say handle or router, or it'll be confusing.
Contributor
Author
|
@akshay-anyscale filed #59376 |
akshay-anyscale
approved these changes
Dec 12, 2025
Yicheng-Lu-llll
pushed a commit
to Yicheng-Lu-llll/ray
that referenced
this pull request
Dec 22, 2025
https://anyscale-ray--59220.com.readthedocs.build/en/59220/serve/monitoring.html#built-in-ray-serve-metrics fixes ray-project#59218 docs changes - [x] refactored the table with all metrics, IMO markdown is easier to read in code - [x] split the table of metrics in ordered categories. categories are ordered by typical request path - [x] included a stick diagram of important metrics, show where in the request lifecycle the metric is recorded - [x] order metrics in table by order in request path Adding the following new metrics ``` - ray_serve_deployment_target_replicas: Target number of replicas Tags: deployment, application - ray_serve_autoscaling_decision_replicas: Raw decision before bounds Tags: deployment, application - ray_serve_autoscaling_total_requests: Total requests seen by autoscaler Tags: deployment, application - ray_serve_autoscaling_policy_execution_time_ms: Policy execution time Tags: deployment, application, policy_scope - ray_serve_autoscaling_replica_metrics_delay_ms: Replica metrics delay Tags: deployment, application, replica - ray_serve_autoscaling_handle_metrics_delay_ms: Handle metrics delay Tags: deployment, application, handle ``` --------- Signed-off-by: abrar <abrar@anyscale.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://anyscale-ray--59220.com.readthedocs.build/en/59220/serve/monitoring.html#built-in-ray-serve-metrics
fixes #59218
docs changes
Adding the following new metrics