Update the dynamo_component_ prefix, and add deprecation message to component/metrics

Author: keivenchang

components/metrics/README.md

@@ -4,22 +4,17 @@
 
 **This `metrics` component is being deprecated and will be removed in a future release.**
 
-The `metrics` component is being replaced by the **`MetricsRegistry`** built-in functionality that is now available directly in the `DistributedRuntime` framework. The `MetricsRegistry` provides:
-
-- **Automatic metric registration** when creating metrics via endpoint factory methods
-- **Built-in Prometheus HTTP endpoint** accessible via `DYN_SYSTEM_ENABLED=true` and `DYN_SYSTEM_PORT=<port>`
-- **Automatic labeling** with namespace, component, and endpoint information
-- **Simplified API** that eliminates the need for manual Prometheus setup
+The deprecated `metrics` component is being replaced by the **`MetricsRegistry`** built-in functionality that is now available directly in the `DistributedRuntime` framework. The `MetricsRegistry` provides:
 
 **For new projects and existing deployments, please migrate to using `MetricsRegistry` instead of this component.**
 
 See the [Dynamo MetricsRegistry Guide](../../docs/guides/metrics.md) for detailed information on using the new metrics system.
 
 ---
 
-The `metrics` component is a utility for collecting, aggregating, and publishing metrics from a Dynamo deployment, but it is being deprecated and replaced by `MetricsRegistry`.
+The deprecated `metrics` component is a utility for collecting, aggregating, and publishing metrics from a Dynamo deployment, but it is being deprecated and replaced by `MetricsRegistry`.
 
-**Note**: This is a demo implementation. The metrics component is currently under active development and this documentation will change as the implementation evolves.
+**Note**: This is a demo implementation. The deprecated `metrics` component is currently under active development and this documentation will change as the implementation evolves.
 - In this demo the metrics names use the prefix "llm", but in production they will be prefixed with "dynamo" (e.g., the HTTP `/metrics` endpoint will serve metrics with "dynamo" prefixes)
 - This demo will only work when using examples/llm/configs/agg.yml-- other configurations will not work
 
@@ -29,7 +24,7 @@ The `metrics` component is a utility for collecting, aggregating, and publishing
 
 ## Quickstart
 
-To start the `metrics` component, simply point it at the `namespace/component/endpoint`
+To start the deprecated `metrics` component, simply point it at the `namespace/component/endpoint`
 trio for the Dynamo workers that you're interested in monitoring metrics on.
 
 This will:
@@ -58,14 +53,14 @@ will get automatically discovered and the warnings will stop.
 
 ## Workers
 
-The `metrics` component needs running workers to gather metrics from,
+The deprecated `metrics` component needs running workers to gather metrics from,
 so below are some examples of workers and how they can be monitored.
 
 ### Mock Worker
 
-To try out how `metrics` works, there is a demo Rust-based
+To try out how the deprecated `metrics` component works, there is a demo Rust-based
 [mock worker](src/bin/mock_worker.rs) that provides sample data through two mechanisms:
-1. Exposes a stats handler at `dynamo/MyComponent/my_endpoint` that responds to polling requests (from `metrics`) with randomly generated `ForwardPassMetrics` data
+1. Exposes a stats handler at `dynamo/MyComponent/my_endpoint` that responds to polling requests (from the deprecated `metrics` component) with randomly generated `ForwardPassMetrics` data
 2. Publishes mock `KVHitRateEvent` data every second to demonstrate event-based metrics
 
 Step 1: Launch a mock workers via the following command (if already built):
@@ -112,11 +107,11 @@ docker compose -f deploy/docker-compose.yml --profile metrics up -d
 
 ## Metrics Collection Modes
 
-The metrics component supports two modes for exposing metrics in a Prometheus format:
+The deprecated `metrics` component supports two modes for exposing metrics in a Prometheus format:
 
 ### Pull Mode (Default)
 
-When running in pull mode (the default), the metrics component will expose a
+When running in pull mode (the default), the deprecated `metrics` component will expose a
 Prometheus metrics endpoint on the specified host and port that a
 Prometheus server or curl client can pull from:
 
@@ -149,7 +144,7 @@ curl localhost:9091/metrics
 ### Push Mode
 
 For ephemeral or batch jobs, or when metrics need to be pushed through a firewall,
-you can use Push mode. In this mode, the metrics component will periodically push
+you can use Push mode. In this mode, the deprecated `metrics` component will periodically push
 metrics to an externally hosted
 [Prometheus PushGateway](https://prometheus.io/docs/instrumenting/pushing/):
 
@@ -158,7 +153,7 @@ Start a prometheus push gateway service via docker:
 docker run --rm -d -p 9091:9091 --name pushgateway prom/pushgateway
 ```
 
-Start the metrics component in `--push` mode, specifying the host and port of your PushGateway:
+Start the deprecated `metrics` component in `--push` mode, specifying the host and port of your PushGateway:
 ```bash
 # Push metrics to a Prometheus PushGateway every --push-interval seconds
 metrics \
@@ -186,7 +181,7 @@ curl 127.0.0.1:9091/metrics
 ```
 ## Building/Running from Source
 
-For easy iteration while making edits to the metrics component, you can use `cargo run`
+For easy iteration while making edits to the deprecated `metrics` component, you can use `cargo run`
 to build and run with your local changes:
 
 ```bash

deploy/metrics/README.md

@@ -8,7 +8,7 @@ The metrics aggregation service is being replaced by the **`MetricsRegistry`** b
 
 - **Built-in Prometheus HTTP endpoint** accessible via `DYN_SYSTEM_ENABLED=true` and `DYN_SYSTEM_PORT=<port>` (default: 8081)
 - **Automatic metric registration** when creating metrics via endpoint factory methods
-- **Automatic labeling** with namespace, component, and endpoint information
+- **Automatic prefix and labeling** with `dynamo_component_` name prefix, as well as auto labels: dynamo_namespace, dynamo_component, and dynamo_endpoint information. These labels are prefixed to avoid Kubernetes label collisions.
 - **Simplified deployment** - no separate metrics component required
 
 **For new projects and existing deployments, please migrate to using `MetricsRegistry` instead of the metrics aggregation service.**

lib/runtime/examples/system_metrics/README.md

@@ -29,7 +29,7 @@ let ingress = Ingress::for_engine(my_handler)?;
 ingress.add_metrics(&endpoint)?;
 ```
 
-The endpoint automatically provides proper labeling (namespace, component, endpoint) for all metrics.
+The endpoint automatically provides prefix of `dynamo_component_*` in the name, and automatic labeling (dynamo_namespace, dynamo_component, dynamo_endpoint) for all metrics. These labels are prefixed with "dynamo_" to avoid collisions with Kubernetes and other monitoring system labels.
 
 ## Available Methods
 
@@ -44,13 +44,13 @@ The `Ingress` struct provides methods for metrics:
 The following Prometheus metrics are automatically created for all work handlers:
 
 ### Counters
-- `requests_total` - Total requests processed
-- `request_bytes_total` - Total bytes received in requests
-- `response_bytes_total` - Total bytes sent in responses
-- `errors_total` - Total errors encountered (with error_type labels)
+- `dynamo_component_requests_total` - Total requests processed
+- `dynamo_component_request_bytes_total` - Total bytes received in requests
+- `dynamo_component_response_bytes_total` - Total bytes sent in responses
+- `dynamo_component_errors_total` - Total errors encountered (with error_type labels)
 
 ### Error Types
-The `errors_total` metric includes the following error types:
+The `dynamo_component_errors_total` metric includes the following error types:
 - `deserialization` - Errors parsing request messages
 - `invalid_message` - Unexpected message format
 - `response_stream` - Errors creating response streams
@@ -59,65 +59,65 @@ The `errors_total` metric includes the following error types:
 - `publish_final` - Errors publishing final response
 
 ### Histograms
-- `request_duration_seconds` - Request processing time
+- `dynamo_component_request_duration_seconds` - Request processing time
 
 ### Gauges
-- `concurrent_requests` - Number of requests currently being processed
+- `dynamo_component_concurrent_requests` - Number of requests currently being processed
 
 ### Custom Metrics (Optional)
-- `my_custom_bytes_processed_total` - Total data bytes processed by system handler (example)
+- `dynamo_component_my_custom_bytes_processed_total` - Total data bytes processed by system handler (example)
 
 ### Labels
 All metrics automatically include these labels from the endpoint:
-- `namespace` - The namespace name
-- `component` - The component name
-- `endpoint` - The endpoint name
+- `dynamo_namespace` - The namespace name
+- `dynamo_component` - The component name
+- `dynamo_endpoint` - The endpoint name
 
 ## Example Metrics Output
 
 When the system is running, you'll see metrics from the /metrics HTTP path like this:
 
 ```prometheus
-# HELP concurrent_requests Number of requests currently being processed by work handler
-# TYPE concurrent_requests gauge
-concurrent_requests{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace"} 0
-
-# HELP my_custom_bytes_processed_total Example of a custom metric. Total number of data bytes processed by system handler
-# TYPE my_custom_bytes_processed_total counter
-my_custom_bytes_processed_total{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace"} 42
-
-# HELP request_bytes_total Total number of bytes received in requests by work handler
-# TYPE request_bytes_total counter
-request_bytes_total{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace"} 1098
-
-# HELP request_duration_seconds Time spent processing requests by work handler
-# TYPE request_duration_seconds histogram
-request_duration_seconds_bucket{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace",le="0.005"} 3
-request_duration_seconds_bucket{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace",le="0.01"} 3
-request_duration_seconds_bucket{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace",le="0.025"} 3
-request_duration_seconds_bucket{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace",le="0.05"} 3
-request_duration_seconds_bucket{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace",le="0.1"} 3
-request_duration_seconds_bucket{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace",le="0.25"} 3
-request_duration_seconds_bucket{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace",le="0.5"} 3
-request_duration_seconds_bucket{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace",le="1"} 3
-request_duration_seconds_bucket{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace",le="2.5"} 3
-request_duration_seconds_bucket{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace",le="5"} 3
-request_duration_seconds_bucket{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace",le="10"} 3
-request_duration_seconds_bucket{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace",le="+Inf"} 3
-request_duration_seconds_sum{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace"} 0.00048793700000000003
-request_duration_seconds_count{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace"} 3
-
-# HELP requests_total Total number of requests processed by work handler
-# TYPE requests_total counter
-requests_total{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace"} 3
-
-# HELP response_bytes_total Total number of bytes sent in responses by work handler
-# TYPE response_bytes_total counter
-response_bytes_total{component="dyn_example_component",endpoint="dyn_example_endpoint9881",namespace="dyn_example_namespace"} 1917
-
-# HELP uptime_seconds Total uptime of the DistributedRuntime in seconds
-# TYPE uptime_seconds gauge
-uptime_seconds{namespace="http_server"} 1.8226759879999999
+# HELP dynamo_component_concurrent_requests Number of requests currently being processed by work handler
+# TYPE dynamo_component_concurrent_requests gauge
+dynamo_component_concurrent_requests{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace"} 0
+
+# HELP dynamo_component_my_custom_bytes_processed_total Example of a custom metric. Total number of data bytes processed by system handler
+# TYPE dynamo_component_my_custom_bytes_processed_total counter
+dynamo_component_my_custom_bytes_processed_total{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace"} 42
+
+# HELP dynamo_component_request_bytes_total Total number of bytes received in requests by work handler
+# TYPE dynamo_component_request_bytes_total counter
+dynamo_component_request_bytes_total{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace"} 1098
+
+# HELP dynamo_component_request_duration_seconds Time spent processing requests by work handler
+# TYPE dynamo_component_request_duration_seconds histogram
+dynamo_component_request_duration_seconds_bucket{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace",le="0.005"} 3
+dynamo_component_request_duration_seconds_bucket{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace",le="0.01"} 3
+dynamo_component_request_duration_seconds_bucket{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace",le="0.025"} 3
+dynamo_component_request_duration_seconds_bucket{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace",le="0.05"} 3
+dynamo_component_request_duration_seconds_bucket{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace",le="0.1"} 3
+dynamo_component_request_duration_seconds_bucket{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace",le="0.25"} 3
+dynamo_component_request_duration_seconds_bucket{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace",le="0.5"} 3
+dynamo_component_request_duration_seconds_bucket{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace",le="1"} 3
+dynamo_component_request_duration_seconds_bucket{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace",le="2.5"} 3
+dynamo_component_request_duration_seconds_bucket{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace",le="5"} 3
+dynamo_component_request_duration_seconds_bucket{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace",le="10"} 3
+dynamo_component_request_duration_seconds_bucket{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace",le="+Inf"} 3
+dynamo_component_request_duration_seconds_sum{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace"} 0.00048793700000000003
+dynamo_component_request_duration_seconds_count{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace"} 3
+
+# HELP dynamo_component_requests_total Total number of requests processed by work handler
+# TYPE dynamo_component_requests_total counter
+dynamo_component_requests_total{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace"} 3
+
+# HELP dynamo_component_response_bytes_total Total number of bytes sent in responses by work handler
+# TYPE dynamo_component_response_bytes_total counter
+dynamo_component_response_bytes_total{dynamo_component="example_component",dynamo_endpoint="example_endpoint9881",dynamo_namespace="example_namespace"} 1917
+
+# HELP dynamo_component_uptime_seconds Total uptime of the DistributedRuntime in seconds
+# TYPE dynamo_component_uptime_seconds gauge
+dynamo_component_uptime_seconds{dynamo_namespace="http_server"} 1.8226759879999999
 ```
 
 ## Examples
@@ -211,7 +211,7 @@ Once running, you can query the metrics:
 curl http://localhost:8081/metrics | grep -E "(requests_total|request_bytes_total|response_bytes_total|errors_total|request_duration_seconds|concurrent_requests)"
 
 # Get request count for specific endpoint
-curl http://localhost:8081/metrics | grep 'requests_total{endpoint="dyn_example_endpoint"}'
+curl http://localhost:8081/metrics | grep 'requests_total{endpoint="example_endpoint"}'
 
 # Get request duration histogram
 curl http://localhost:8081/metrics | grep 'request_duration_seconds'