Add Cortex and Prometheus Remote Write exporter design #1464

huyan0 · 2020-07-31T01:29:04Z

Design Proposal: Collector Cortex/Prometheus remote write Exporter

This document outlines our design proposal of the Prometheus remote write Exporter requested in #1150.

In the document, we cover the following areas:

Provide context on Cortex and Prometheus remote write API
Outline assumptions we made for this project (we are working with OTLP metrics defined in proto PR #145
Exporting cumulative values only for monotonic types, histogram, and summary
How metrics will be converted and send to the backend by the exporter
Discuss the factory component and configuration of the exporter, supplemented by a sample configuration
Cover thread-safety, shutdown behavior, and testing plan

We have referenced design principles outlined in CONTRIBUTING.md and the design of the OTLP and Prometheus exporters.

cc: @huyan0 @danielbang907 @alolita

Request review: @open-telemetry/collector-approvers @jmacd @bogdandrutu

codecov · 2020-07-31T01:36:08Z

Codecov Report

Merging #1464 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master    #1464   +/-   ##
=======================================
  Coverage   90.90%   90.90%           
=======================================
  Files         239      239           
  Lines       16692    16692           
=======================================
  Hits        15174    15174           
  Misses       1086     1086           
  Partials      432      432

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f954863...7f8529a. Read the comment docs.

annanay25

+1

exporter/cortexexporter/DESIGN.md

annanay25 · 2020-07-31T11:23:40Z

exporter/cortexexporter/DESIGN.md

+
+When adding a quantile of the summary data point to the map, the string signature should also contain a `quantile ` label that indicates the quantile value. This label should also be exported. Any summary metric that is not cumulative should be dropped.
+
+### **2.3 Exporting Metrics**


Would it be easier to vendor in Prometheus remote write package here?

Thanks for the suggestion, using the client from remote write makes sense. We are also trying to allow users to specify any header or pass in a http.Client to send requests. I will take a look at whether this is possible with remote write package.

It doesn't sadly, but we should see if we can make the current client accept a custom http.Client.

https://github.com/prometheus/prometheus/blob/master/storage/remote/client.go#L125-L133

We can try adding a SetHTTPClient() method to the Prometheus client object.

Yes, I think that change would be very helpful for our usage.

gouthamve

As a Prometheus and Cortex maintainer, I love this proposal! But we would be open to many more integrations and services if we rename this from Cortex to Prometheus Remote Write :)

See: https://prometheus.io/docs/operating/integrations/#remote-endpoints-and-storage

exporter/cortexexporter/DESIGN.md

gouthamve · 2020-07-31T14:46:08Z

exporter/cortexexporter/DESIGN.md

+**Assumptions:**
+Because of the gaps mentioned above, this project will convert from the current OTLP metrics and work under the assumption one of the above solutions will be implemented, and all incoming monotonic scalars/histogram/summary metrics should be cumulative or otherwise dropped. More details on the behavior of the exporter is in section 2.2.
+
+## **2. Cortex Exporter**


Can we rename this from Cortex exporter to "Prometheus Remote Write exporter"? Cortex just uses the Prom RW APIs and by renaming we'll be clearer in intent that we support Prom RW and not just Cortex.

Yes, we do want to emphasize the support for Cortex, but we definitely can indicate its compatibility with other remote write backends in our document. Thanks for the suggestion. Also, a question on remote write: does it send out sorted label set and how do we treat duplicate labels?

We send a list of labels. And I think its upto the upstream implementation to deal with the discrepancy. In general, in Prometheus, you can never have a label name that has 2 values, so the upstream will likely reject it with a 4xx. I am fairly certain that the labels need not be sorted.

Thanks for the reply. Another related question: for Cortex to work, should our exporter meet the requirement that metric of the same type and name must have the same labels? If so, how could we maintain it in the case of multiple exporters?

OTel gives some specification about duplicate labels. They should be eliminated before the exporter sees them. I'm not sure if the SDK specification will say they're sorted, but in the OTel Go SDK they are sorted as an efficient means of deduplication.

gouthamve · 2020-07-31T14:50:26Z

exporter/cortexexporter/DESIGN.md

+
+When adding a quantile of the summary data point to the map, the string signature should also contain a `quantile ` label that indicates the quantile value. This label should also be exported. Any summary metric that is not cumulative should be dropped.
+
+### **2.3 Exporting Metrics**


It doesn't sadly, but we should see if we can make the current client accept a custom http.Client.

https://github.com/prometheus/prometheus/blob/master/storage/remote/client.go#L125-L133

We can try adding a SetHTTPClient() method to the Prometheus client object.

exporter/cortexexporter/DESIGN.md

jmacd · 2020-08-04T07:15:42Z

exporter/cortexexporter/DESIGN.md

+**Assumptions:**
+Because of the gaps mentioned above, this project will convert from the current OTLP metrics and work under the assumption one of the above solutions will be implemented, and all incoming monotonic scalars/histogram/summary metrics should be cumulative or otherwise dropped. More details on the behavior of the exporter is in section 2.2.
+
+## **2. Cortex Exporter**


OTel gives some specification about duplicate labels. They should be eliminated before the exporter sees them. I'm not sure if the SDK specification will say they're sorted, but in the OTel Go SDK they are sorted as an efficient means of deduplication.

jmacd · 2020-08-04T07:18:50Z

@huyan0 Thanks! I appreciate the detailed explanation of how you can carry this out with the current state of the specification and OTLP support. 😀

bogdandrutu

Overall design looks promising, some suggestions to make things a bit more generic. Having the first version only supporting cumulative is fine.

exporter/cortexexporter/DESIGN.md

bogdandrutu · 2020-08-10T20:14:37Z

exporter/remotewriteexporter/DESIGN.md

+
+The following diagram shows an example of Prometheus remote write API usage, with Cortex,n open source, horizontally scalable, highly available, multi-tenant, long term storage, as a remote storage backend.
+
+![Cortex Archietecture](https://raw.githubusercontent.com/open-o11y/opentelemetry-collector/design-doc/exporter/cortexexporter/cortex.png)


Use relative path (./cortex.png) same for the other png. May consider to move the pngs in a sub-directory.

bogdandrutu · 2020-08-10T20:15:31Z

@huyan0 there are some dead links:
[✖] https://raw.githubusercontent.com/open-o11y/opentelemetry-collector/design-doc/exporter/cortexexporter/cortex.png → Status: 404
[✖] https://raw.githubusercontent.com/open-o11y/opentelemetry-collector/design-doc/exporter/cortexexporter/timeseries.png → Status: 404

Which may be because you add them now :)

bogdandrutu · 2020-08-11T14:38:51Z

Ping :)

huyan0 · 2020-08-11T14:51:07Z

Ping :)

Thanks! I have renamed the folder and used relative links for images :-)

* Prepare for releasing v0.16.0 * Prepare CHANGELOG for v0.16.0 release Signed-off-by: Anthony J Mirabella <a9@aneurysm9.com>

* Upgrade to golang 1.18.1 * Update changelog

Add Cortex exporter design

ac39113

huyan0 requested review from bogdandrutu, dmitryax, james-bebbington, nilebox, owais, pjanotti and tigrannajaryan as code owners July 31, 2020 01:29

annanay25 reviewed Jul 31, 2020

View reviewed changes

fix typo in image

2abf604

gouthamve reviewed Jul 31, 2020

View reviewed changes

change name to remote write exporter

407c536

huyan0 changed the title ~~Add Cortex exporter design~~ Add Prometheus Remote Write exporter design Jul 31, 2020

huyan0 changed the title ~~Add Prometheus Remote Write exporter design~~ Add Cortex and Prometheus Remote Write exporter design Jul 31, 2020

jmacd approved these changes Aug 4, 2020

View reviewed changes

bogdandrutu reviewed Aug 4, 2020

View reviewed changes

This was referenced Aug 5, 2020

add helper.go and tests open-o11y/opentelemetry-collector#5

Closed

and cortex.go and tests open-o11y/opentelemetry-collector#6

Closed

add cortex.go and helper.go and tests open-o11y/opentelemetry-collector#7

Closed

change name to Prometheus Remote Write/Cortex Exporter

7b2ad3f

huyan0 force-pushed the design-doc branch from 4648d77 to 7b2ad3f Compare August 6, 2020 01:10

morigs mentioned this pull request Aug 6, 2020

Add Prometheus remote write exporter to collector default package #1150

Closed

bogdandrutu approved these changes Aug 10, 2020

View reviewed changes

dabangarang mentioned this pull request Aug 11, 2020

Add Prometheus Remote Write Exporter supporting Cortex - Factory and Config #1524

Closed

huyan0 mentioned this pull request Aug 11, 2020

Add Prometheus Remote Write Exporter supporting Cortex - Metrics exporter #1525

Closed

huyan0 added 3 commits August 11, 2020 09:46

fix dead links

039bbbd

Update DESIGN.md

dd390ea

Update DESIGN.md

7f8529a

bogdandrutu merged commit b5b185a into open-telemetry:master Aug 11, 2020

huyan0 mentioned this pull request Aug 14, 2020

Add Prometheus Remote Write Exporter supporting Cortex - helper #1547

Closed

gouthamve mentioned this pull request Aug 19, 2020

Add the ability to inject a custom round tripper prometheus/common#252

Closed

This was referenced Aug 27, 2020

Add Prometheus Remote Write Exporter supporting Cortex - conversion and export for Histogram OTLP metrics #1643

Merged

Add Prometheus Remote Write Exporter supporting Cortex - conversion and export for Summary OTLP metrics #1649

Closed

hughesjj pushed a commit to hughesjj/opentelemetry-collector that referenced this pull request Apr 27, 2023

Upgrade to golang 1.18.1 (open-telemetry#1464)

92564d6

* Upgrade to golang 1.18.1 * Update changelog

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Cortex and Prometheus Remote Write exporter design #1464

Add Cortex and Prometheus Remote Write exporter design #1464

huyan0 commented Jul 31, 2020 •

edited

Loading

codecov bot commented Jul 31, 2020 •

edited

Loading

annanay25 left a comment

annanay25 Jul 31, 2020

huyan0 Jul 31, 2020

gouthamve Jul 31, 2020

huyan0 Aug 6, 2020

gouthamve left a comment

gouthamve Jul 31, 2020

huyan0 Jul 31, 2020 •

edited

Loading

gouthamve Jul 31, 2020

huyan0 Aug 3, 2020

jmacd Aug 4, 2020

gouthamve Jul 31, 2020

jmacd Aug 4, 2020

jmacd commented Aug 4, 2020

bogdandrutu left a comment

bogdandrutu Aug 10, 2020

bogdandrutu commented Aug 10, 2020

bogdandrutu commented Aug 11, 2020

huyan0 commented Aug 11, 2020


		When adding a quantile of the summary data point to the map, the string signature should also contain a `quantile ` label that indicates the quantile value. This label should also be exported. Any summary metric that is not cumulative should be dropped.

		### 2.3 Exporting Metrics


		The following diagram shows an example of Prometheus remote write API usage, with Cortex,n open source, horizontally scalable, highly available, multi-tenant, long term storage, as a remote storage backend.

		![Cortex Archietecture](https://raw.githubusercontent.com/open-o11y/opentelemetry-collector/design-doc/exporter/cortexexporter/cortex.png)

Add Cortex and Prometheus Remote Write exporter design #1464

Add Cortex and Prometheus Remote Write exporter design #1464

Conversation

huyan0 commented Jul 31, 2020 • edited Loading

Design Proposal: Collector Cortex/Prometheus remote write Exporter

codecov bot commented Jul 31, 2020 • edited Loading

Codecov Report

annanay25 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

gouthamve left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

huyan0 Jul 31, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jmacd commented Aug 4, 2020

bogdandrutu left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bogdandrutu commented Aug 10, 2020

bogdandrutu commented Aug 11, 2020

huyan0 commented Aug 11, 2020

huyan0 commented Jul 31, 2020 •

edited

Loading

codecov bot commented Jul 31, 2020 •

edited

Loading

huyan0 Jul 31, 2020 •

edited

Loading