Automatically determine interval for new metrics

[DanCech]

When new metrics are received that don't have an interval specified, MT should attempt to automatically derive the interval by buffering a few points and looking at the interval between them.

[Dieterbe]

What's the use case ? Which input plugin ?

[DanCech]

Specifically carbon

[Dieterbe]

What's the use case ? On prem for people who neglected their storage-schema.conf ?

[DanCech]

People who want to send carbon direct to hosted metrics is the primary use-case since MT won't have a way to determine the interval. It may also be useful for people using MT on-prem who want it to figure out the interval without having to manually keep the schema updated.

[woodsaj]

this is needed for all input plugins.

The driving purpose is to support multi-tenant hosted-metrics with direct carbon input. Right now, we support direct carbon input for single tenant HM instances as we just have the customer provide us with their storage-schemas.conf. But that is just a stop gap until we can provide a more complete solution.

For hosted-metrics that data will be coming into MT via the Kafka-mdm input plugin, but the tsdb-gw will receive the data via carbon, then send it to kafka with the MetricData.Interval field set to 0 (unknown).

The reasoning for doing the interval detection in MT instead of in tsdb-gw is that the metrics are consistently routed to the same the MT (metricData.Name is used as the routing key), whereas with tsdb-gw, the metrics could be ingested by any one of a number of tsdb-gw instances. So to support the interval detection in tsdb-gw we would need an external index (redis, memcache, etc...) which would introduce a significant overhead.

[woodsaj]

This is also probably needed to support Prometheus remote write. Prometheus metrics don't have an interval property.

We could use the storage-schemas.conf for the prometheus metrics, but as with direct carbon input it requires users to provide us with their storage-schemas.conf. To support changes we would need to build out a way for users to configure their storage-schemas.conf via grafana.com.

Just detecting the interval seems much more user-friendly.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.