Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Initial cut at migrating jmacd's datamodel document into the spec #1512
Initial cut at migrating jmacd's datamodel document into the spec #1512
Changes from 14 commits
8bf6969
d38c7b9
12f9734
d73341b
c0e1326
f358772
7bff96b
f2a07cd
8d0db7a
f157ed9
7fd0ad2
8895928
517b654
b844bc1
aef7a65
d562c32
d63dddd
93a16b3
611282a
File filter
Filter by extension
Conversations
Jump to
There are no files selected for viewing
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think I fail a bit to understand this:
It is in scope to have
Prometheus -> OTLP -> PRW
and it is in scope to haveStatsD -> OTLP
then why is not in scope to haveStatsD -> OTLP -> PRW
? mathematically if the translation functionOTLP -> PRW
is defined andStatsD -> OTLP
is defined then I can compose them to getStatsD -> OTLP -> PRW
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think
StatsD -> OTLP -> PRW
is in-scope. We haven't added support for raw sampled metric events, which is how they appear to the StatsD receiver, so somewhere in that picture is a stateful conversion to OTLP(cumulative) which is not a difficult task, just requires memory.I believe there is an additional concept that belongs in the specification, and I'm sorry to introduce it here, that I think of as a counterpart to the Single-Writer definition we have discussed. It can be called a Single-Reader property, and it is required to perform a correct delta-to-cumulative translation.
Single-Reader: Having access to the whole stream of data for a metric. This happens when an OTel collector is configured as a per-node agent and every process on the node reports through that agent. This happens when an OTel collector pool runs the Kafka exporter and Kafka receiver with appropriate configuration.
StatsD -> OTLP -> PRW
should be in-scope for a single-node deployment, and it ought to behave just as https://github.com/prometheus/statsd_exporter would.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@bogdandrutu There's a lot of nuance to Statsd => PRW. While TECHNICALLY you can shove bits into PRW, when I say "compatibility" i mean the way we defined it for Prometheus: "A user of prometheus shouldn't be able to tell if OTLP was used to transport prometheues style metric to their backend". Similarly, for users of Statsd today, we should HONOR the statsd conventions and formats from statsd => PRW. The reason Statsd => PRW is out of scope is because of all the non-data-related nuance (e.g. naming conventions, namespace options etc. that a statsd user would expect).
To be frank: I don't think it's OTLP's job to define how Statsd maps into PRW, and we shouldn't sacrifice the design of OTLP for that use case. We can attempt to provide the best translation possible, but it should not "block" or otherwise inhibit progress. There's a difference between "out of scope" and "can't do". What I'm suggesting is we won't prioritize or make a ton of effort to solve statsd => PRW bugs that are inherent to the impedance mismatch of those two models.
@jmacd I'm actually tying your "Single-Reader" into the "SingleWriter rule. Specifically the way I'm phrasing Single-WRiter in the not-yet-public PR is that any given timeseries should originate from a --SingleWriter. This includes aggregation time series, meaning if the collector is manipulating a metrics, it becomes the new single-writer of that aggregation. It is also required for delta-to-cumulative manipulation (which in my opinion, that's a new series of data). To some extent I think this nuance is just terminology and we need to make sure we COVER the instance of when it's OK to do "Delta-Cumulative" and what that implies on your telemetry-pipeline architecture.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
💯
Yes, that's how I've been thinking about it too. It's permitted to output the same metric name, in this case, because the incoming data is "terminal".