-
Notifications
You must be signed in to change notification settings - Fork 1.5k
/
scraping-receivers.md
150 lines (106 loc) · 8.86 KB
/
scraping-receivers.md
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
# Scraping Metrics Receivers
Scraping metrics receivers are receivers that pull data from external sources at regular intervals and translate it
into [pdata](../pdata/README.md) which is sent further in the pipeline. The external source of metrics usually is a
monitored system providing data about itself in some arbitrary format. There are two types of scraping metrics
receivers:
- **Generic scraping metrics receivers:** The set of metrics emitted by this type of receiver fully depends on the
state of the external source and/or the user settings. Examples:
- [Prometheus Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/prometheusreceiver)
- [SQL Query Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/sqlqueryreceiver)
- **Built-in scraping metrics receivers:** Receivers of this type emit a predefined set of metrics. However, the
metrics themselves are configurable via user settings. Examples of scrapings metrics receivers:
- [Redis Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/redisreceiver)
- [Zookeeper Receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/zookeeperreceiver)
This document covers built-in scraping metrics receivers. It defines which metrics these receivers can emit,
defines stability guarantees and provides guidelines for metric updates.
## Defining emitted metrics
Each built-in scraping metrics receiver has a `metadata.yaml` file that MUST define all the metrics emitted by the
receiver. The file is being used to generate an API for metrics recording, user settings to customize the emitted
metrics and user documentation. The file schema is defined in
https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/cmd/mdatagen/metadata-schema.yaml
Defining a metric in `metadata.yaml` DOES NOT guarantee that the metric will always be produced by the receiver. In
some cases it may be impossible to fetch particular metrics from a system in a particular state.
There are two categories of the metrics emitted by scraping receivers:
- **Default metrics**: emitted by default, but can be disabled in user settings.
- **Optional metrics**: not emitted by default, but can be enabled in user settings.
### How to identify if new metric should be default or optional?
There is no strict rule to differentiate default metrics from optional. As a rule of thumb, default metrics SHOULD be
treated as metrics essential to determine healthiness of a particular monitored system. Optional metrics on the
other hand SHOULD be treated as a source of additional information about the monitored system.
Additionally, if any of the following conditions can be applied to a metric, it MUST be marked as optional:
- **It is redundant with another metric.** For example system CPU usage can be emitted as two different metrics:
`system.cpu.time` (CPU time reported as cumulative sum in seconds) or `system.cpu.utilization` (fraction of CPU
time spent in different states reported as gauge), one of them has to be marked as option.
- **It creates a disproportionately high cardinality of resources and/or data points.**
- **There is a notable expected performance impact.**
- **The source must be configured in an unusual way.**
- **It requires dedicated configuration in the receiver.**
## Stability levels of scraping receivers
All the requirements defined for components in [the Collector's README](../README.md#stability-levels) are
applicable to the scraping receivers as well. In addition, the following rules applied specifically to scraping
metrics receivers:
### Development
The receiver is not ready for use. All the metrics emitted by the receiver are not finalized and can change in any way.
### Alpha
The receiver is ready for limited non-critical workloads. The list of emitted default metrics SHOULD be
considered as complete, but any changes to the `metadata.yaml` still MAY be applied.
### Beta
The receiver is ready for non-critical production workloads. The list of emitted default metrics MUST be
considered as complete. Breaking changes to the emitted metrics SHOULD be applied following [the deprecation
process](#changing-the-emitted-metrics).
### Stable
The receiver is ready for production workloads. Breaking changes to the emitted metrics SHOULD be avoided.
Nevertheless, metrics that are emitted by default MUST be always kept up-to-date with the latest stable version of the
monitored system. Given that, occasional breaking changes in the emitted metrics are expected even in the stable
receivers. Any breaking change MUST be applied following [the deprecation process](#changing-the-emitted-metrics).
## Changing the emitted metrics
Some changes are not considered breaking and can be applied to metrics emitted by scraping receivers of any
stability level:
- Adding a new optional metric.
Most of other changes to the emitted metrics are considered breaking and MUST be handled according to the stability
level of the receiver. Each type of breaking change defines a set of steps that MUST (or SHOULD) be applied across
several releases for a Stable (or Beta) components. At least 3 versions SHOULD be kept between the steps to give
users time to prepare, e.g. if the first step is released in v0.62.0, the second step SHOULD be released not earlier
than 0.65.0. Any warnings SHOULD include the version starting from which the next step will take effect. If a
breaking change is more complicated and many metrics are involved in the change, feature gates SHOULD be used instead.
### Removing an optional metric
Steps to remove an optional metric:
1. Mark the metric as deprecated in `metadata.yaml` by adding "[DEPRECATED]" in its description. Show a warning that
the metric will be removed if the `enabled` option is set explicitly to `true` in user settings.
2. Remove the metric.
### Removing a default metric
Steps to remove a default metric:
1. Mark the metric as deprecated in `metadata.yaml` by adding "[DEPRECATED]" in its description. Show a warning that
the metric will be removed if the `enabled` option is not explicitly set to `false` in user settings.
2. Make the metric optional. Show a warning that the metric will be removed if the `enabled` option is set to `true`
in user settings.
3. Remove the metric.
### Making a default metric optional
Steps to turn a metric from default to optional:
1. Add a warning that the metric will be turned into optional if `enabled` field is not set explicitly to any value in
user settings. Warning example: "WARNING: Metric `foo.bar` will be disabled by default in v0.65.0. If you want to
keep it, please enable it explicitly in the receiver settings."
2. Remove the warning and update `metadata.yaml` to make the metric optional.
### Adding a new default metric or turning an existing optional metric into default
Adding a new default metric is a breaking change for a scraping receiver because it introduces an unexpected output
for users and additional load on metric backends. Steps to apply such a change:
1. If the metric doesn't exist yet, add one as an optional metric. Add a warning that the metric will be turned into
default if the `enabled` option is not set explicitly to any value in user settings. A warning example: "WARNING:
Metric `foo.bar` will be enabled by default in v0.65.0. If you don't want the metric to be emitted, please
disable it in the receiver settings."
2. Remove the warning and update `metadata.yaml` to make the metric default.
### Other changes
Other breaking changes SHOULD follow similar strategies inspecting presence of `enabled` field in user settings. For
example, if a metric has to be renamed for any reason, the guidelines for "Removing an optional metric" and "Adding a
new default metric" SHOULD be followed simultaneously.
Breaking changes that cannot be done through enabling/disabling metrics (e.g. removing or adding an extra attribute)
SHOULD be applied using a feature gate with the following steps:
1. Add a feature gate that is disabled by default. Enabling the feature gate changes the metrics behavior in a desired
way. For example, if several metrics emitted by host metrics receiver need to be updated to have an additional
`direction` attribute, the following feature gate can be used:
`receiver.hostmetricsreceiver.emitMetricsWithDirectionAttribute`. Show user a warning if the feature gate is not
enabled explicitly, for example: "[WARNING] Metrics `system.network.packets` and `system.network.errors` will be
changed in v0.65.0 to emit an additional `direction` attribute, enable a feature gate
`receiver.hostmetricsreceiver.emitMetricsWithDirectionAttribute` to apply and test the upcoming changes earlier".
2. Enable the feature gate by default and update the warning thrown if user disables the feature gate explicitly.
3. Remove the feature gate along with the old behavior.