Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus Remote-Write receiver Alpha #37277

Open
5 of 9 tasks
dashpole opened this issue Jan 16, 2025 · 26 comments · Fixed by #38795
Open
5 of 9 tasks

Prometheus Remote-Write receiver Alpha #37277

dashpole opened this issue Jan 16, 2025 · 26 comments · Fixed by #38795
Assignees

Comments

@dashpole
Copy link
Contributor

dashpole commented Jan 16, 2025

Component(s)

receiver/prometheusremotewrite

Describe the issue you're reporting

  • Handle empty otel_scope_name and otel_scope_version
  • Handle Timeseries samples
  • Handle Counters
  • Handle Histograms
  • Handle Summaries (maybe too hard and could be left out for Alpha)
  • Handle Created Timestamp
  • Return number of samples and histograms ingested via response headers
  • Handle multiple timeseries with same metric name+type+unit
  • Handle target_info
@dashpole dashpole added needs triage New item requiring triage receiver/prometheusremotewrite enhancement New feature or request labels Jan 16, 2025
Copy link
Contributor

Pinging code owners for receiver/prometheusremotewrite: @dashpole @ArthurSens. See Adding Labels via Comments if you do not have permissions to add labels yourself. For example, comment '/label priority:p2 -needs-triaged' to set the priority and remove the needs-triaged label.

@adwait-godbole
Copy link

adwait-godbole commented Jan 24, 2025

Hi @ArthurSens! Hi @dashpole! I am eager to be a part of this project and looking forward to work under your mentorship and guidance. Are there any pre-requisites / pre-tests for the selection process of mentees ?

@ArthurSens
Copy link
Member

Hi Adwait, glad to see your interest. There are no pre-requisites or tests, just make sure to apply following the LFX process. Also be aware that our proposal is still just a PR, we still need both CNCF and LFX to approve the mentorship

@PiyushRajDev
Copy link

Hello @ArthurSens ! I am looking forward to contribute to this project. Can you please guide me on how to get started?

@ArthurSens
Copy link
Member

Hello @ArthurSens ! I am looking forward to contribute to this project. Can you please guide me on how to get started?

A good starting point is understanding Remote-Write and the OTLP->Prometheus compatibility spec. If you few comfortable tackling some of the checkboxes above, that's also nice :)

@hanshal101
Copy link

Hey @ArthurSens, Hanshal here. I am an open-source developer contributed to various projects like Buildsafe, Glasskube, Cyclops and a couple of Apache's too. I am currently interning at CloudRaft as a DevOps Intern. I recently published a blog on OTel Auto Instrumentation, and my curiosity towards observability brought me here. I'm looking forward to working on this issue if it gets accepted.
I am pretty much aware of the OTLP -> Prometheus spec; what else should I focus on more to understand the issue in a better way?

Thank you.

@ArthurSens
Copy link
Member

ArthurSens commented Feb 2, 2025

Hi Hanshal, this project is quite simple so we're not requiring that much experience, to be honest. There's nothing else to be familiar with besides the two protocols that were already mentioned: Prometheus Remote Write and OTLP.

We'd love to mentor someone who hasn't had a chance with OSS yet and is excited to participate in the community, that's all.

Anyone interested, just make sure you follow the LFX process correctly.

@PiyushRajDev
Copy link

Hello @ArthurSens ! I have read through the suggested documentation and familiarized myself with the protocols. As I dive deeper into the project, I may encounter some challenges along the way. Would it be possible to join a Slack channel or another communication platform where I can ask quick clarifying questions when needed? I understand you’re busy, so I really appreciate any help or pointers you can offer.

@pratikgitss
Copy link

pratikgitss commented Feb 5, 2025

I agree with @PiyushRajDev, it would be great to have a communication channel as I have also gone through the protocols and I am figuring out how to tackle the requirements mentioned above and will be really helpful if there is a channel to get the questions clarified.

@ArthurSens
Copy link
Member

Hello hello 👋 -- This issue is a great communication channel! Feel free to ask questions here :)

But yes, there is also the CNCF slack. We have a channel called #otel-prometheus that can also be used to ask questions. Asking it here would be a lot better though, since the information would be accessible to all.

@sonalgaud12
Copy link
Contributor

Hey everyone, I'm Sonal! Recently, I've been contributing to the OTEL Collector and OTEL Collector Contrib, specifically under the Receiver package. I'm really excited about this project and believe there's so much to learn from it. By the way, are there any upcoming meetings or briefings about the project? I'd love to contribute more!

@PiyushRajDev
Copy link

Hello @sonalgaud, there are not any meeting or briefings as such but we need to just apply through the LFX process for this issue.

@Brant-lzh
Copy link

Hello, I just need this feature and I am very eager to participate in this project, so what do I need to do now?

@PiyushRajDev
Copy link

Hey @Brant-lzh, for getting started you need to refer to one of the messages sent by @ArthurSens up above on this thread where he sent links of the two protcols for getting started.

@Brant-lzh
Copy link

Hey @PiyushRajDev ,I understand these contents. I have a certain understanding of prometheus and opentelemetry. How should I start now? Is there a working group?

@sonalgaud12
Copy link
Contributor

Hey @Brant-lzh , this project will be developed under the LFX mentorship program. You’ll need to apply for it, and the selected candidate will be working on it. For communication, you can join the CNCF Slack channel named otel-prometheus.

@Brant-lzh
Copy link

Brant-lzh commented Feb 8, 2025

Hey @sonalgaud12 , I joined the Slack Channel, and how to apply for this LFX mentorship program? Is there any relevant documentation?

@sonalgaud12
Copy link
Contributor

sonalgaud12 commented Feb 8, 2025

@Brant-lzh
Copy link

@sonalgaud12 thx!

@PiyushRajDev
Copy link

Hey @ArthurSens , Could you please point me to any existing implementations or helper functions in the code that follow a similar structure to addCounterDatapoints? It would be helpful to review how similar cases are handled elsewhere in the code for consistency.

@ArthurSens
Copy link
Member

It should be very very similar to addGaugeDatapoints, but I wouldn't implement this first though, let's first handle the scope attributes or the samples.

@sonalgaud12
Copy link
Contributor

sonalgaud12 commented Feb 11, 2025

Hey @ArthurSens , for the first to do I could think of two possible solution.

  1. Use Collector Build Tags
  2. Use a Default Fallback Value (If Build Tags Are Not Available)
    As the comment suggest the collector build approach do let me know which one will be better I'll raise an PR.

sujalshah-bit added a commit to sujalshah-bit/opentelemetry-collector-contrib that referenced this issue Feb 18, 2025
Convert sample timestamps from milliseconds to nanoseconds. Set
datapoint values using SetDoubleValue based on incoming sample values.
Update tests in translateV2 to verify that samples are correctly
ingested and their timestamps properly converted.

Fixes part of open-telemetry#37277.

Signed-off-by: sujal shah <sujalshah28092004@gmail.com>
sujalshah-bit added a commit to sujalshah-bit/opentelemetry-collector-contrib that referenced this issue Feb 18, 2025
Convert sample timestamps from milliseconds to nanoseconds. Set
datapoint values using SetDoubleValue based on incoming sample values.
Update tests in translateV2 to verify that samples are correctly
ingested and their timestamps properly converted.

Fixes part of open-telemetry#37277.

Signed-off-by: sujal shah <sujalshah28092004@gmail.com>
sujalshah-bit added a commit to sujalshah-bit/opentelemetry-collector-contrib that referenced this issue Feb 19, 2025
Convert sample timestamps from milliseconds to nanoseconds. Set
datapoint values using SetDoubleValue based on incoming sample values.
Update tests in translateV2 to verify that samples are correctly
ingested and their timestamps properly converted.

Fixes part of open-telemetry#37277.

Signed-off-by: sujal shah <sujalshah28092004@gmail.com>
sujalshah-bit added a commit to sujalshah-bit/opentelemetry-collector-contrib that referenced this issue Feb 19, 2025
Convert sample timestamps from milliseconds to nanoseconds. Set
datapoint values using SetDoubleValue based on incoming sample values.
Update tests in translateV2 to verify that samples are correctly
ingested and their timestamps properly converted.

Fixes part of open-telemetry#37277.

Signed-off-by: sujal shah <sujalshah28092004@gmail.com>
sujalshah-bit added a commit to sujalshah-bit/opentelemetry-collector-contrib that referenced this issue Feb 19, 2025
Convert sample timestamps from milliseconds to nanoseconds. Set
datapoint values using SetDoubleValue based on incoming sample values.
Update tests in translateV2 to verify that samples are correctly
ingested and their timestamps properly converted.

Fixes part of open-telemetry#37277.

Signed-off-by: sujal shah <sujalshah28092004@gmail.com>
songy23 pushed a commit that referenced this issue Feb 20, 2025
…37895)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Convert sample timestamps from milliseconds to nanoseconds. Set
datapoint values using SetDoubleValue based on incoming sample values.
Update tests in translateV2 to verify that samples are correctly
ingested and their timestamps properly converted.

<!-- Issue number (e.g. #1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes part of #37277.

<!--Please delete paragraphs that you did not use before submitting.-->

Signed-off-by: sujal shah <sujalshah28092004@gmail.com>
sujalshah-bit added a commit to sujalshah-bit/opentelemetry-collector-contrib that referenced this issue Feb 21, 2025
This commit implements the addCounterDatapoints function to properly
convert Prometheus counter metrics to OTLP format.

Fixes part of open-telemetry#37277

Signed-off-by: sujal shah <sujalshah28092004@gmail.com>
sujalshah-bit added a commit to sujalshah-bit/opentelemetry-collector-contrib that referenced this issue Feb 21, 2025
This commit implements the addCounterDatapoints function to properly
convert Prometheus counter metrics to OTLP format.

Fixes part of open-telemetry#37277

Signed-off-by: sujal shah <sujalshah28092004@gmail.com>
sujalshah-bit added a commit to sujalshah-bit/opentelemetry-collector-contrib that referenced this issue Feb 22, 2025
This commit implements the addCounterDatapoints function to properly
convert Prometheus counter metrics to OTLP format.

Fixes part of open-telemetry#37277

Signed-off-by: sujal shah <sujalshah28092004@gmail.com>
songy23 pushed a commit that referenced this issue Mar 6, 2025
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

This PR ensures that when Prometheus samples lack the otel_scope_name or
otel_scope_version labels, the receiver fills in default values.

It take care of part of
#37277
atoulme pushed a commit that referenced this issue Mar 18, 2025
…ith same metric name+type+unit (#38453)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

This PR belongs to part of the Linux foundation mentee program. Here we
are dealing with data points slices that must be added to the same
metric when the incoming time series have some attributes in common.
Like same resource, metricName, scopeName, scopeVersion, unitRef and
timeseries type. The reference to this implementation can be found
https://opentelemetry.io/docs/specs/otel/metrics/data-model/#opentelemetry-protocol-data-model

Besides that. We are creating 2 new test cases. The first one, to
validate the described behavior above.

The second is to validate an error case. When the unitRef passed doesn't
match with the symbols slice, causing a panic. This test case is
important to guarantee that the error is being handled correctly.

<!-- Issue number (e.g. #1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes part of #37277. Bullet point `Handle multiple timeseries with same
metric name+type+unit`

---------

Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>
atoulme pushed a commit that referenced this issue Mar 18, 2025
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

This PR belongs to part of the Linux foundation mentee program.
Here we are adding the help ref attribute to metric following the rules
described
[here](https://opentelemetry.io/docs/specs/otel/metrics/data-model/#opentelemetry-protocol-data-model-producer-recommendations)

<!-- Issue number (e.g. #1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes part of
[#37277](#37277)
atoulme pushed a commit that referenced this issue Mar 18, 2025
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description

This PR belongs to part of the Linux foundation mentee program.
Here we are setting the start timestamp for data point attributes

<!-- Issue number (e.g. #1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
Fixes part of
[#37277](#37277)

---------

Co-authored-by: Arthur Silva Sens <arthursens2005@gmail.com>
sujalshah-bit added a commit to sujalshah-bit/opentelemetry-collector-contrib that referenced this issue Mar 19, 2025
…h-based Hash method.

Creates MetricIdentity struct for uniquely identifying metrics and adds
a deterministic Hash method using xxhash with proper field separation.

partly fixes open-telemetry#37277.
sujalshah-bit added a commit to sujalshah-bit/opentelemetry-collector-contrib that referenced this issue Mar 19, 2025
…h-based Hash method.

Creates MetricIdentity struct for uniquely identifying metrics and adds
a deterministic Hash method using xxhash with proper field separation.

partly fixes open-telemetry#37277.
sujalshah-bit added a commit to sujalshah-bit/opentelemetry-collector-contrib that referenced this issue Mar 19, 2025
…h-based Hash method.

Creates MetricIdentity struct for uniquely identifying metrics and adds
a deterministic Hash method using xxhash with proper field separation.

partly fixes open-telemetry#37277.
sujalshah-bit added a commit to sujalshah-bit/opentelemetry-collector-contrib that referenced this issue Mar 21, 2025
…h-based Hash method.

Creates MetricIdentity struct for uniquely identifying metrics and adds
a deterministic Hash method using xxhash with proper field separation.

partly fixes open-telemetry#37277.
sujalshah-bit added a commit to sujalshah-bit/opentelemetry-collector-contrib that referenced this issue Mar 21, 2025
…h-based Hash method.

Creates MetricIdentity struct for uniquely identifying metrics and adds
a deterministic Hash method using xxhash with proper field separation.

partly fixes open-telemetry#37277.
atoulme pushed a commit that referenced this issue Mar 21, 2025
…th xxhash-based hash method (#38795)

<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->
#### Description
Creates MetricIdentity struct for uniquely identifying metrics and adds
a deterministic Hash method using xxhash with proper field separation.
<!-- Issue number (e.g. #1234) or full URL to issue, if applicable. -->
#### Link to tracking issue
partly fixes #37277.
<!--Describe what testing was performed and which tests were added.-->
#### Testing

<!--Describe the documentation added.-->
#### Documentation

<!--Please delete paragraphs that you did not use before submitting.-->
@ArthurSens
Copy link
Member

Whoops, this was closed by accident

@ArthurSens ArthurSens reopened this Mar 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
8 participants