From 3114930b61c47a2fbbe2691337bfbfd2f204d3c0 Mon Sep 17 00:00:00 2001 From: Christine Chen Date: Wed, 8 Dec 2021 16:42:28 -0500 Subject: [PATCH] Support Openmetrics metrics collection (#10752) * Add logic for Envoy Openmetricsv2 * Add label remapper * Add new metrics * Finish adding other metrics * reorganize metrics that should be transformed * Introduce openmetrics_endpoint config option * Add watchdog metrics transformers * Add some more label extraction metrics * refactor tests to move to legacy * Add legacy and non legacy fixtures to test files * Update readme * Bump base req * Fix style * Mark legacy metrics * Fix watchdog counter name * document prometheus metrics in metadata csv * Fix metadata csv * Add e2e test * FIx style * Fix metadata format for validation * Flaky metrics * Only support openmetrics in latest api v3 * Fix test imports * Enable Openmetrics option by default * Fix import * Fix style * Update readme * Update config stats_url wording * Fix envoy import * Remove py27 for openmetrics version * Openmetrics endpoint should be optional * Account for flaky metrics * Document service checks * Use unique name * Update envoy/tests/legacy/test_bench.py Co-authored-by: Ofek Lev * Move metrics map to metrics.py * Update with feedback * Use lambda * simplify match * Refactor metadata utils * Support metadata collection in V2 * Use urlunparse * Reintroduce legacy config options as hidden Co-authored-by: Ofek Lev --- envoy/README.md | 48 +- envoy/__init__.py | 0 envoy/assets/configuration/spec.yaml | 42 +- envoy/assets/service_checks.json | 14 + envoy/datadog_checks/envoy/check.py | 156 ++ .../envoy/config_models/defaults.py | 104 ++ .../envoy/config_models/instance.py | 57 +- .../envoy/data/conf.yaml.example | 311 +++- envoy/datadog_checks/envoy/envoy.py | 67 +- envoy/datadog_checks/envoy/metrics.py | 319 ++++ envoy/datadog_checks/envoy/utils.py | 48 + envoy/metadata.csv | 1515 ++++++++++------- envoy/setup.py | 2 +- envoy/tests/common.py | 327 +++- envoy/tests/conftest.py | 16 +- envoy/tests/legacy/__init__.py | 0 envoy/tests/legacy/common.py | 36 + envoy/tests/{ => legacy}/test_bench.py | 4 +- envoy/tests/legacy/test_e2e.py | 297 ++++ envoy/tests/{ => legacy}/test_envoy.py | 72 +- envoy/tests/{ => legacy}/test_metrics.py | 4 + envoy/tests/{ => legacy}/test_parser.py | 4 + envoy/tests/{ => legacy}/test_utils.py | 4 + envoy/tests/test_check.py | 60 + envoy/tests/test_e2e.py | 299 +--- envoy/tox.ini | 5 +- 26 files changed, 2636 insertions(+), 1175 deletions(-) create mode 100644 envoy/__init__.py create mode 100644 envoy/datadog_checks/envoy/check.py create mode 100644 envoy/tests/legacy/__init__.py create mode 100644 envoy/tests/legacy/common.py rename envoy/tests/{ => legacy}/test_bench.py (86%) create mode 100644 envoy/tests/legacy/test_e2e.py rename envoy/tests/{ => legacy}/test_envoy.py (85%) rename envoy/tests/{ => legacy}/test_metrics.py (85%) rename envoy/tests/{ => legacy}/test_parser.py (99%) rename envoy/tests/{ => legacy}/test_utils.py (93%) create mode 100644 envoy/tests/test_check.py diff --git a/envoy/README.md b/envoy/README.md index 39cc772b944fa..944e8726b8f1b 100644 --- a/envoy/README.md +++ b/envoy/README.md @@ -12,7 +12,12 @@ The Envoy check is included in the [Datadog Agent][2] package, so you don't need #### Istio -If you are using Envoy as part of [Istio][3], be sure to use the appropriate [Envoy admin endpoint][4] for the `stats_url`. +If you are using Envoy as part of [Istio][3], configure the Envoy integration to collect metrics from the Istio proxy metrics endpoint. + + ```yaml + instances: + - openmetrics_endpoint: localhost:15090/stats/prometheus + ``` #### Standard @@ -100,45 +105,16 @@ To configure this check for an Agent running on a host: init_config: instances: - ## @param stats_url - string - required - ## The admin endpoint to connect to. It must be accessible: - ## https://www.envoyproxy.io/docs/envoy/latest/operations/admin - ## Add a `?usedonly` on the end if you wish to ignore - ## unused metrics instead of reporting them as `0`. - # - - stats_url: http://localhost:80/stats + ## @param openmetrics_endpoint - string - required + ## The URL exposing metrics in the OpenMetrics format. + # + - openmetrics_endpoint: http://localhost:8001/stats/prometheus + ``` 2. Check if the Datadog Agent can access Envoy's [admin endpoint][5]. 3. [Restart the Agent][9]. -###### Metric filtering - -Metrics can be filtered with the parameters`included_metrics` or `excluded_metrics` using regular expressions. If both parameters are used, `included_metrics` is applied first, then `excluded_metrics` is applied on the resulting set. - -The filtering occurs before tag extraction, so you have the option to have certain tags decide whether or not to keep or ignore metrics. An exhaustive list of all metrics and tags can be found in [metrics.py][10]. Let's walk through an example of Envoy metric tagging! - -```python -... -'cluster.grpc.success': { - 'tags': ( - ('', ), - ('', '', ), - (), - ), - ... -}, -... -``` - -Here there are `3` tag sequences: `('')`, `('', '')`, and empty `()`. The number of sequences corresponds exactly to how many metric parts there are. For this metric, there are `3` parts: `cluster`, `grpc`, and `success`. Envoy separates everything with a `.`, hence the final metric name would be: - -`cluster..grpc...success` - -If you care only about the cluster name and grpc service, you would add this to your `included_metrics`: - -`^cluster\.\.grpc\.\.` - ##### Log collection