use correct route value in proxy metrics tags instead of route_prefix #58180

abrarsheikh · 2025-10-26T22:50:15Z

Expose Route Patterns in Proxy Metrics

Problem

Proxy metrics (ray_serve_num_http_requests_total, ray_serve_http_request_latency_ms) only expose route_prefix (e.g., /api) instead of actual route patterns (e.g., /api/users/{user_id}). This prevents granular monitoring of individual endpoints without causing high cardinality from unique request paths.

Design

Route Pattern Extraction & Propagation:

Replicas extract route patterns from ASGI apps (FastAPI/Starlette) at initialization using extract_route_patterns()
Patterns propagate: Replica → ReplicaMetadata → DeploymentState → EndpointInfo → Proxy
Works with both normal patterns (routes in class) and factory patterns (callable returns app)

Proxy Route Matching:

ProxyRouter.match_route_pattern() matches incoming requests to specific patterns using cached mock Starlette apps
Metrics tag requests with parameterized routes (e.g., /api/users/{user_id}) instead of prefixes
Fallback to route_prefix if patterns unavailable or matching fails

Performance:

Metric	Before	After
Requests per second (RPS)	403.39	397.82
Mean latency (ms)	247.9	251.37
p50 (ms)	224	223
p90 (ms)	415	428
p99 (ms)	526	544

Testing

Unit tests for extract_route_patterns()
Integration test verifying metrics use patterns and avoid high cardinality
Parametrized for both normal and factory patterns

Signed-off-by: abrar <abrar@anyscale.com>

…proxy_metrics

akyang-anyscale

potential optimization is to use the matched pattern in the replica metrics as well to not duplicate work

python/ray/serve/_private/thirdparty/get_asgi_route_name.py

abrarsheikh · 2025-10-28T18:45:51Z

potential optimization is to use the matched pattern in the replica metrics as well to not duplicate work

Excellent idea. Doing it now

abrarsheikh · 2025-10-28T19:12:45Z

potential optimization is to use the matched pattern in the replica metrics as well to not duplicate work

Excellent idea. Doing it now

discussed offline, idea is to pass the matched route from proxy to replica through request metatdata. Taking this up in a follow up PR

ok-scale

using starlette matcher was much better! LGTM

…ray-project#58180) ## Expose Route Patterns in Proxy Metrics fixes ray-project#52212 ### Problem Proxy metrics (`ray_serve_num_http_requests_total`, `ray_serve_http_request_latency_ms`) only expose `route_prefix` (e.g., `/api`) instead of actual route patterns (e.g., `/api/users/{user_id}`). This prevents granular monitoring of individual endpoints without causing high cardinality from unique request paths. ### Design **Route Pattern Extraction & Propagation:** - Replicas extract route patterns from ASGI apps (FastAPI/Starlette) at initialization using `extract_route_patterns()` - Patterns propagate: Replica → `ReplicaMetadata` → `DeploymentState` → `EndpointInfo` → Proxy - Works with both normal patterns (routes in class) and factory patterns (callable returns app) **Proxy Route Matching:** - `ProxyRouter.match_route_pattern()` matches incoming requests to specific patterns using cached mock Starlette apps - Metrics tag requests with parameterized routes (e.g., `/api/users/{user_id}`) instead of prefixes - Fallback to `route_prefix` if patterns unavailable or matching fails **Performance:** Metric | Before | After -- | -- | -- Requests per second (RPS) | 403.39 | 397.82 Mean latency (ms) | 247.9 | 251.37 p50 (ms) | 224 | 223 p90 (ms) | 415 | 428 p99 (ms) | 526 | 544 ### Testing - Unit tests for `extract_route_patterns()` - Integration test verifying metrics use patterns and avoid high cardinality - Parametrized for both normal and factory patterns --------- Signed-off-by: abrar <abrar@anyscale.com>

…#58180) ## Expose Route Patterns in Proxy Metrics fixes #52212 ### Problem Proxy metrics (`ray_serve_num_http_requests_total`, `ray_serve_http_request_latency_ms`) only expose `route_prefix` (e.g., `/api`) instead of actual route patterns (e.g., `/api/users/{user_id}`). This prevents granular monitoring of individual endpoints without causing high cardinality from unique request paths. ### Design **Route Pattern Extraction & Propagation:** - Replicas extract route patterns from ASGI apps (FastAPI/Starlette) at initialization using `extract_route_patterns()` - Patterns propagate: Replica → `ReplicaMetadata` → `DeploymentState` → `EndpointInfo` → Proxy - Works with both normal patterns (routes in class) and factory patterns (callable returns app) **Proxy Route Matching:** - `ProxyRouter.match_route_pattern()` matches incoming requests to specific patterns using cached mock Starlette apps - Metrics tag requests with parameterized routes (e.g., `/api/users/{user_id}`) instead of prefixes - Fallback to `route_prefix` if patterns unavailable or matching fails **Performance:** Metric | Before | After -- | -- | -- Requests per second (RPS) | 403.39 | 397.82 Mean latency (ms) | 247.9 | 251.37 p50 (ms) | 224 | 223 p90 (ms) | 415 | 428 p99 (ms) | 526 | 544 ### Testing - Unit tests for `extract_route_patterns()` - Integration test verifying metrics use patterns and avoid high cardinality - Parametrized for both normal and factory patterns --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: elliot-barn <elliot.barnwell@anyscale.com>

…ray-project#58180) ## Expose Route Patterns in Proxy Metrics fixes ray-project#52212 ### Problem Proxy metrics (`ray_serve_num_http_requests_total`, `ray_serve_http_request_latency_ms`) only expose `route_prefix` (e.g., `/api`) instead of actual route patterns (e.g., `/api/users/{user_id}`). This prevents granular monitoring of individual endpoints without causing high cardinality from unique request paths. ### Design **Route Pattern Extraction & Propagation:** - Replicas extract route patterns from ASGI apps (FastAPI/Starlette) at initialization using `extract_route_patterns()` - Patterns propagate: Replica → `ReplicaMetadata` → `DeploymentState` → `EndpointInfo` → Proxy - Works with both normal patterns (routes in class) and factory patterns (callable returns app) **Proxy Route Matching:** - `ProxyRouter.match_route_pattern()` matches incoming requests to specific patterns using cached mock Starlette apps - Metrics tag requests with parameterized routes (e.g., `/api/users/{user_id}`) instead of prefixes - Fallback to `route_prefix` if patterns unavailable or matching fails **Performance:** Metric | Before | After -- | -- | -- Requests per second (RPS) | 403.39 | 397.82 Mean latency (ms) | 247.9 | 251.37 p50 (ms) | 224 | 223 p90 (ms) | 415 | 428 p99 (ms) | 526 | 544 ### Testing - Unit tests for `extract_route_patterns()` - Integration test verifying metrics use patterns and avoid high cardinality - Parametrized for both normal and factory patterns --------- Signed-off-by: abrar <abrar@anyscale.com>

…ray-project#58180) ## Expose Route Patterns in Proxy Metrics fixes ray-project#52212 ### Problem Proxy metrics (`ray_serve_num_http_requests_total`, `ray_serve_http_request_latency_ms`) only expose `route_prefix` (e.g., `/api`) instead of actual route patterns (e.g., `/api/users/{user_id}`). This prevents granular monitoring of individual endpoints without causing high cardinality from unique request paths. ### Design **Route Pattern Extraction & Propagation:** - Replicas extract route patterns from ASGI apps (FastAPI/Starlette) at initialization using `extract_route_patterns()` - Patterns propagate: Replica → `ReplicaMetadata` → `DeploymentState` → `EndpointInfo` → Proxy - Works with both normal patterns (routes in class) and factory patterns (callable returns app) **Proxy Route Matching:** - `ProxyRouter.match_route_pattern()` matches incoming requests to specific patterns using cached mock Starlette apps - Metrics tag requests with parameterized routes (e.g., `/api/users/{user_id}`) instead of prefixes - Fallback to `route_prefix` if patterns unavailable or matching fails **Performance:** Metric | Before | After -- | -- | -- Requests per second (RPS) | 403.39 | 397.82 Mean latency (ms) | 247.9 | 251.37 p50 (ms) | 224 | 223 p90 (ms) | 415 | 428 p99 (ms) | 526 | 544 ### Testing - Unit tests for `extract_route_patterns()` - Integration test verifying metrics use patterns and avoid high cardinality - Parametrized for both normal and factory patterns --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: Aydin Abiar <aydin@anyscale.com>

…te matching (#58927) The correct route value is already part of RequestMetadata after #58180, no need to recompute it again. no observed perf diff in microbenchmark After ``` Type Name # Requests # Fails Median (ms) 95%ile (ms) 99%ile (ms) Average (ms) Min (ms) Max (ms) Average size (bytes) Current RPS Current Failures/s GET /echo?message=hello 28068 0 200 410 470 228.27 80 592 26 430.3 0 Aggregated 28068 0 200 410 470 228.27 80 592 26 430.3 0 ``` Before ``` Type Name # Requests # Fails Median (ms) 95%ile (ms) 99%ile (ms) Average (ms) Min (ms) Max (ms) Average size (bytes) Current RPS Current Failures/s GET /echo?message=hello 27427 0 210 410 470 232.12 76 604 26 429.7 0 Aggregated 27427 0 210 410 470 232.12 76 604 26 429.7 0 ``` Additionally, old implementation wrongly assumed that there will only be one method (GET,PUT) corresponding to a route. This PR fixes that assumption and tests for it. --------- Signed-off-by: abrar <abrar@anyscale.com>

…te matching (ray-project#58927) The correct route value is already part of RequestMetadata after ray-project#58180, no need to recompute it again. no observed perf diff in microbenchmark After ``` Type Name # Requests # Fails Median (ms) 95%ile (ms) 99%ile (ms) Average (ms) Min (ms) Max (ms) Average size (bytes) Current RPS Current Failures/s GET /echo?message=hello 28068 0 200 410 470 228.27 80 592 26 430.3 0 Aggregated 28068 0 200 410 470 228.27 80 592 26 430.3 0 ``` Before ``` Type Name # Requests # Fails Median (ms) 95%ile (ms) 99%ile (ms) Average (ms) Min (ms) Max (ms) Average size (bytes) Current RPS Current Failures/s GET /echo?message=hello 27427 0 210 410 470 232.12 76 604 26 429.7 0 Aggregated 27427 0 210 410 470 232.12 76 604 26 429.7 0 ``` Additionally, old implementation wrongly assumed that there will only be one method (GET,PUT) corresponding to a route. This PR fixes that assumption and tests for it. --------- Signed-off-by: abrar <abrar@anyscale.com> Signed-off-by: YK <1811651+ykdojo@users.noreply.github.com>

…te matching (ray-project#58927) The correct route value is already part of RequestMetadata after ray-project#58180, no need to recompute it again. no observed perf diff in microbenchmark After ``` Type Name # Requests # Fails Median (ms) 95%ile (ms) 99%ile (ms) Average (ms) Min (ms) Max (ms) Average size (bytes) Current RPS Current Failures/s GET /echo?message=hello 28068 0 200 410 470 228.27 80 592 26 430.3 0 Aggregated 28068 0 200 410 470 228.27 80 592 26 430.3 0 ``` Before ``` Type Name # Requests # Fails Median (ms) 95%ile (ms) 99%ile (ms) Average (ms) Min (ms) Max (ms) Average size (bytes) Current RPS Current Failures/s GET /echo?message=hello 27427 0 210 410 470 232.12 76 604 26 429.7 0 Aggregated 27427 0 210 410 470 232.12 76 604 26 429.7 0 ``` Additionally, old implementation wrongly assumed that there will only be one method (GET,PUT) corresponding to a route. This PR fixes that assumption and tests for it. --------- Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh added 2 commits October 26, 2025 22:42

send asgi app routes from replica to proxy

f2e2a85

Signed-off-by: abrar <abrar@anyscale.com>

fix tests

179fbc8

Signed-off-by: abrar <abrar@anyscale.com>

abrarsheikh added the go add ONLY when ready to merge, run all tests label Oct 26, 2025

abrarsheikh added 3 commits October 26, 2025 23:45

move test file

4355f06

Signed-off-by: abrar <abrar@anyscale.com>

fix tests

6bd2420

Signed-off-by: abrar <abrar@anyscale.com>

Merge branch 'master' of github.com:ray-project/ray into 52212-abrar-…

9702b6e

…proxy_metrics

abrarsheikh marked this pull request as ready for review October 27, 2025 16:52

abrarsheikh requested a review from a team as a code owner October 27, 2025 16:52

ray-gardener bot added serve Ray Serve Related Issue observability Issues related to the Ray Dashboard, Logging, Metrics, Tracing, and/or Profiling labels Oct 27, 2025

Merge branch 'master' of github.com:ray-project/ray into 52212-abrar-…

0d3ad19

…proxy_metrics

abrarsheikh requested review from akyang-anyscale and ok-scale October 27, 2025 20:09

akyang-anyscale approved these changes Oct 28, 2025

View reviewed changes

python/ray/serve/_private/thirdparty/get_asgi_route_name.py Show resolved Hide resolved

ok-scale approved these changes Oct 29, 2025

View reviewed changes

abrarsheikh merged commit e466c6f into master Oct 29, 2025
6 checks passed

abrarsheikh deleted the 52212-abrar-proxy_metrics branch October 29, 2025 18:39

abrarsheikh mentioned this pull request Nov 24, 2025

[Serve] remove route matching logic from replica and bug fix with route matching #58927

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

use correct route value in proxy metrics tags instead of route_prefix #58180

use correct route value in proxy metrics tags instead of route_prefix #58180

abrarsheikh commented Oct 26, 2025 •

edited

Loading

Uh oh!

akyang-anyscale left a comment

Uh oh!

Uh oh!

abrarsheikh commented Oct 28, 2025

Uh oh!

abrarsheikh commented Oct 28, 2025

Uh oh!

ok-scale left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

use correct route value in proxy metrics tags instead of route_prefix #58180

use correct route value in proxy metrics tags instead of route_prefix #58180

Conversation

abrarsheikh commented Oct 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Expose Route Patterns in Proxy Metrics

Problem

Design

Testing

Uh oh!

akyang-anyscale left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

abrarsheikh commented Oct 28, 2025

Uh oh!

abrarsheikh commented Oct 28, 2025

Uh oh!

ok-scale left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

abrarsheikh commented Oct 26, 2025 •

edited

Loading