Initial cut at migrating jmacd's datamodel document into the spec

[jsuereth]

From Task: https://github.com/open-telemetry/opentelemetry-specification/projects/3#card-56227942

• Imports an initial version of @jmacd's Metric DataModel document, which elucidates a LOT of important points/goals in the metric data model over the existing specification.
• Adds TODO sections for discussion beyond the initial "Events" - "Data Model" - "Time Series" fragmentation.
• Updates verbiage in root specification around the metrics data model.

Note: This PR is 100% focused on framing the problem with the 3 models and separation of concerns, as well as defining some key use cases for evaluating the model.

Replaces #1510 with permission to do so.

[aabmass]

Original doc: https://docs.google.com/document/d/1lwLZlQ2RttzZ60N9xhJuJlivs3m5kFjUEZgrfpaF09c/edit#

[jmacd]

(Is "3P" defined somewhere?)

[bogdandrutu]

I would replace with the explicit text :)

[bogdandrutu]

Overall LGTM, I think we can iterate over some details later.

[bogdandrutu]

I would replace with the explicit text :)

[bogdandrutu]

I think I fail a bit to understand this:

It is in scope to have Prometheus -> OTLP -> PRW and it is in scope to have StatsD -> OTLP then why is not in scope to have StatsD -> OTLP -> PRW? mathematically if the translation function OTLP -> PRW is defined and StatsD -> OTLP is defined then I can compose them to get StatsD -> OTLP -> PRW

[jmacd]

I think StatsD -> OTLP -> PRW is in-scope. We haven't added support for raw sampled metric events, which is how they appear to the StatsD receiver, so somewhere in that picture is a stateful conversion to OTLP(cumulative) which is not a difficult task, just requires memory.

I believe there is an additional concept that belongs in the specification, and I'm sorry to introduce it here, that I think of as a counterpart to the Single-Writer definition we have discussed. It can be called a Single-Reader property, and it is required to perform a correct delta-to-cumulative translation.

Single-Reader: Having access to the whole stream of data for a metric. This happens when an OTel collector is configured as a per-node agent and every process on the node reports through that agent. This happens when an OTel collector pool runs the Kafka exporter and Kafka receiver with appropriate configuration.

StatsD -> OTLP -> PRW should be in-scope for a single-node deployment, and it ought to behave just as https://github.com/prometheus/statsd_exporter would.

[jsuereth]

@bogdandrutu There's a lot of nuance to Statsd => PRW. While TECHNICALLY you can shove bits into PRW, when I say "compatibility" i mean the way we defined it for Prometheus: "A user of prometheus shouldn't be able to tell if OTLP was used to transport prometheues style metric to their backend". Similarly, for users of Statsd today, we should HONOR the statsd conventions and formats from statsd => PRW. The reason Statsd => PRW is out of scope is because of all the non-data-related nuance (e.g. naming conventions, namespace options etc. that a statsd user would expect).

To be frank: I don't think it's OTLP's job to define how Statsd maps into PRW, and we shouldn't sacrifice the design of OTLP for that use case. We can attempt to provide the best translation possible, but it should not "block" or otherwise inhibit progress. There's a difference between "out of scope" and "can't do". What I'm suggesting is we won't prioritize or make a ton of effort to solve statsd => PRW bugs that are inherent to the impedance mismatch of those two models.

@jmacd I'm actually tying your "Single-Reader" into the "SingleWriter rule. Specifically the way I'm phrasing Single-WRiter in the not-yet-public PR is that any given timeseries should originate from a --SingleWriter. This includes aggregation time series, meaning if the collector is manipulating a metrics, it becomes the new single-writer of that aggregation. It is also required for delta-to-cumulative manipulation (which in my opinion, that's a new series of data). To some extent I think this nuance is just terminology and we need to make sure we COVER the instance of when it's OK to do "Delta-Cumulative" and what that implies on your telemetry-pipeline architecture.

[jmacd]

💯

It is also required for delta-to-cumulative manipulation (which in my opinion, that's a new series of data).

Yes, that's how I've been thinking about it too. It's permitted to output the same metric name, in this case, because the incoming data is "terminal".

[jmacd]

I think StatsD -> OTLP -> PRW is in-scope. We haven't added support for raw sampled metric events, which is how they appear to the StatsD receiver, so somewhere in that picture is a stateful conversion to OTLP(cumulative) which is not a difficult task, just requires memory.

I believe there is an additional concept that belongs in the specification, and I'm sorry to introduce it here, that I think of as a counterpart to the Single-Writer definition we have discussed. It can be called a Single-Reader property, and it is required to perform a correct delta-to-cumulative translation.

Single-Reader: Having access to the whole stream of data for a metric. This happens when an OTel collector is configured as a per-node agent and every process on the node reports through that agent. This happens when an OTel collector pool runs the Kafka exporter and Kafka receiver with appropriate configuration.

StatsD -> OTLP -> PRW should be in-scope for a single-node deployment, and it ought to behave just as https://github.com/prometheus/statsd_exporter would.