-
Notifications
You must be signed in to change notification settings - Fork 156
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Implement Prometheus remote write receiver compatible with sfx gateway #3064
Conversation
…o dps if none found
internal/receiver/prometheusremotewritereceiver/prometheus_to_otel.go
Outdated
Show resolved
Hide resolved
internal/receiver/prometheusremotewritereceiver/prometheus_to_otel.go
Outdated
Show resolved
Hide resolved
Co-authored-by: Antoine Toulme <antoine@lunar-ocean.com>
…rovider not MetricsProvider)
…ot just make lint
…gateway impl behavior (besides obsreport)
.../receiver/signalfxgatewayprometheusremotewritereceiver/internal/metadata/generated_status.go
Show resolved
Hide resolved
...rnal/receiver/signalfxgatewayprometheusremotewritereceiver/internal/prometheus_spec_utils.go
Outdated
Show resolved
Hide resolved
…/internal/prometheus_spec_utils.go Co-authored-by: Antoine Toulme <antoine@lunar-ocean.com>
@@ -0,0 +1,24 @@ | |||
# SignalFx Gateway Prometheus Remote write Receiver Example |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A bit long(ish). Can we shorten to, say, Prometheus remote write example
?
This example provides a `docker-compose` environment that continually sends some fake prometheus remote writes to an otel receiver replacement for the deprecated SignalFx Gateway for Prometheus Remote Writes. | ||
To run this, ensure you have `docker-compose` installed. | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This example provides a `docker-compose` environment that continually sends some fake prometheus remote writes to an otel receiver replacement for the deprecated SignalFx Gateway for Prometheus Remote Writes. | |
To run this, ensure you have `docker-compose` installed. | |
This example provides a `docker-compose` environment that continually sends fake prometheus remote writes to an OTel receiver replacement for the deprecated SignalFx Gateway for Prometheus Remote Writes. To run the example, make sure you have `docker-compose` installed. |
To run this, ensure you have `docker-compose` installed. | ||
|
||
## Configuration | ||
You can change the exporters to your liking by modifying `otel-collector-config.yaml`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can change the exporters to your liking by modifying `otel-collector-config.yaml`. | |
To change the exporters, edit the `otel-collector-config.yaml` configuration file. |
Ensure the following environment variables are properly set, should you wish to send data to splunk observability cloud: | ||
1. `SPLUNK_ACCESS_TOKEN` | ||
2. `SPLUNK_REALM` | ||
|
||
Alternatively, you can remove the `signalfx` array item from the `exporters` configuration map in `otel-collector-config.yaml` | ||
|
||
Feel free to modify the sample client to your liking, or even disable it and write your own! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ensure the following environment variables are properly set, should you wish to send data to splunk observability cloud: | |
1. `SPLUNK_ACCESS_TOKEN` | |
2. `SPLUNK_REALM` | |
Alternatively, you can remove the `signalfx` array item from the `exporters` configuration map in `otel-collector-config.yaml` | |
Feel free to modify the sample client to your liking, or even disable it and write your own! | |
If you want to send data to Splunk Observability Cloud, set the following environment variables: | |
-`SPLUNK_ACCESS_TOKEN` | |
-`SPLUNK_REALM` | |
You can also remove the `signalfx` array item from the `exporters` configuration map in the `otel-collector-config.yaml` configuration file. | |
> **Tip:** Experiment and modify the sample client, or even disable it and write your own. |
## Running | ||
Once you've verified your environment, you can run the example by | ||
|
||
```bash | ||
$> docker-compose up | ||
``` | ||
|
||
If everything is configured properly, logs with sample writes should start appearing in stdout shortly. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
## Running | |
Once you've verified your environment, you can run the example by | |
```bash | |
$> docker-compose up | |
``` | |
If everything is configured properly, logs with sample writes should start appearing in stdout shortly. | |
## Run the example | |
After you've verified your environment, enter the following command to run the example: | |
```bash | |
$> docker-compose up |
If everything is configured properly, logs with sample writes start appearing in stdout.
@@ -0,0 +1,32 @@ | |||
# SignalFx Gateway Prometheus remote write receiver | |||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@hughesjj Could you add here what the receiver is for?
## Limitations and Nuances in translation | ||
This receiver specifically obsoletes the near-exact behavior of the [SignalFx prometheus remote write gateway](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go). | ||
The behavior of the prometheus remote write gateway predates the formalization of the PRW v1 specification, and thus differs in the following ways. | ||
|
||
- We do not [remove suffixes](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/6658646e7705b74f13031c777fcd8dd1cd64c850/receiver/prometheusreceiver/internal/metricfamily.go#L316) as is done in the otel-contrib `prometheusreceiver` | ||
- It will transform histograms [into counters](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#L98). | ||
- It will transform quantiles (summaries) into gauges. | ||
- If the representation of a float could be expressed as an integer without loss, we will set it as an integer | ||
- If the representation of a sample is NAN, we will report an additional counter with the metric name [`"prometheus.total_NAN_samples"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL190C24-L190C53) | ||
- If the representation of a sample is missing a metric name, we will report an additional counter with the metric name [`"prometheus.total_bad_datapoints"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL191C24-L191C24) | ||
- Any errors in parsing the request will report an additional counter [`"prometheus.invalid_requests"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL189C80-L189C91) | ||
- Metadata from the `prompb.WriteRequest` is **ignored** | ||
|
||
The following behavior from sfx gateway is not supported | ||
- `"request_time.ns"` is no longer reported. `obsreport` handles similar functionality. | ||
- `"drain_size"` is no longer reported. `obsreport` handles similar functionality. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
## Limitations and Nuances in translation | |
This receiver specifically obsoletes the near-exact behavior of the [SignalFx prometheus remote write gateway](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go). | |
The behavior of the prometheus remote write gateway predates the formalization of the PRW v1 specification, and thus differs in the following ways. | |
- We do not [remove suffixes](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/6658646e7705b74f13031c777fcd8dd1cd64c850/receiver/prometheusreceiver/internal/metricfamily.go#L316) as is done in the otel-contrib `prometheusreceiver` | |
- It will transform histograms [into counters](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#L98). | |
- It will transform quantiles (summaries) into gauges. | |
- If the representation of a float could be expressed as an integer without loss, we will set it as an integer | |
- If the representation of a sample is NAN, we will report an additional counter with the metric name [`"prometheus.total_NAN_samples"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL190C24-L190C53) | |
- If the representation of a sample is missing a metric name, we will report an additional counter with the metric name [`"prometheus.total_bad_datapoints"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL191C24-L191C24) | |
- Any errors in parsing the request will report an additional counter [`"prometheus.invalid_requests"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL189C80-L189C91) | |
- Metadata from the `prompb.WriteRequest` is **ignored** | |
The following behavior from sfx gateway is not supported | |
- `"request_time.ns"` is no longer reported. `obsreport` handles similar functionality. | |
- `"drain_size"` is no longer reported. `obsreport` handles similar functionality. | |
## Known limitations | |
This receiver obsoletes the near-exact behavior of the [SignalFx Prometheus Remote-Writegateway](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go). The behavior of the Prometheus Remote-Write gateway predates the formalization of the Prometheus Remote-Write specification version 1, and differs in the following ways: | |
- The receiver doesn't [remove suffixes](https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/6658646e7705b74f13031c777fcd8dd1cd64c850/receiver/prometheusreceiver/internal/metricfamily.go#L316) as this is done in the otel-contrib `prometheusreceiver`. | |
- The receiver transforms histograms [into counters](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#L98). | |
- The receiver transforms quantiles (summaries) into gauges. | |
- If the representation of a float can be expressed as an integer without loss, the receiver sets the representation of a float as an integer. | |
- If the representation of a sample is NaN, the receiver reports an additional counter with the metric name [`"prometheus.total_NAN_samples"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL190C24-L190C53). | |
- If the representation of a sample is missing a metric name, the receiver reports an additional counter with the metric name [`"prometheus.total_bad_datapoints"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL191C24-L191C24). | |
- Any errors in parsing the request report an additional counter, [`"prometheus.invalid_requests"`](https://github.com/signalfx/gateway/blob/main/protocol/prometheus/prometheuslistener.go#LL189C80-L189C91). | |
- Metadata from the `prompb.WriteRequest` is ignored. | |
The following behavior from sfx gateway is not supported: | |
- `"request_time.ns"` is no longer reported. `obsreport` handles similar functionality. | |
- `"drain_size"` is no longer reported. `obsreport` handles similar functionality. |
## Receiver Configuration | ||
This receiver is configured via standard OpenTelemetry mechanisms. See [`config.go`](./config.go) for specific details. | ||
|
||
* `path` is the path in which the receiver should respond to prometheus remote write requests. | ||
* Defaults to `/metrics` | ||
* `buffer_size` is the degree to which metric translations may be buffered without blocking further write requests. | ||
* Defaults to `100` | ||
|
||
This receiver uses `opentelemetry-collector`'s [`confighttp`](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/confighttp.go#L206) options if you would like to set up tls or similar. (See linked documentation for the most up-to-date details). | ||
However, we make the following changes to their default options: | ||
* `endpoint` is the default interface + port to listen on | ||
* Defaults to `localhost:19291` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
## Receiver Configuration | |
This receiver is configured via standard OpenTelemetry mechanisms. See [`config.go`](./config.go) for specific details. | |
* `path` is the path in which the receiver should respond to prometheus remote write requests. | |
* Defaults to `/metrics` | |
* `buffer_size` is the degree to which metric translations may be buffered without blocking further write requests. | |
* Defaults to `100` | |
This receiver uses `opentelemetry-collector`'s [`confighttp`](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/confighttp.go#L206) options if you would like to set up tls or similar. (See linked documentation for the most up-to-date details). | |
However, we make the following changes to their default options: | |
* `endpoint` is the default interface + port to listen on | |
* Defaults to `localhost:19291` | |
## Receiver configuration | |
This receiver is configured through standard OpenTelemetry mechanisms. See [`config.go`](./config.go) for details. | |
* `path` is the path in which the receiver responds to prometheus remote-write requests. The default values is `/metrics`. | |
* `buffer_size` is the degree to which metric translations can be buffered without blocking further write requests. The default value is `100`. | |
This receiver uses `opentelemetry-collector`'s [`confighttp`](https://github.com/open-telemetry/opentelemetry-collector/blob/main/config/confighttp/confighttp.go#L206) options if you want to set up TLS and other features. However, the receiver makes the following changes to upstream default options: | |
* `endpoint` is the default interface and port to listen on. The default value is `localhost:19291`. |
README.md
for nuances in configuration