Exporter version 0.32.0 and higher is not compatible with opentelemetry-collector-contrib

[krupyansky]

Description

The library opentelemetry-collector-contrib along the path pkg/translator/prometheusremotewrite/metrics_to_prw.go has a method FromMetrics():

// FromMetrics converts pmetric.Metrics to prometheus remote write format.
func FromMetrics(md pmetric.Metrics, settings Settings) (tsMap map[string]*prompb.TimeSeries, errs error) {
    ...
    // check for valid type and temporality combination and for matching data field and type
    if ok := validateMetrics(metric); !ok {
	    errs = multierr.Append(errs, errors.New("invalid temporality and type combination"))
	    continue
    }
    ...
}

Due to the fact that an invalid metric comes to the opentelemetry-collector-contrib (in my case, SUM with len(DataPoints) == 0), I get error "invalid temporality and type combination" when exporting data into prometheusremotewrite from opentelemetry-collector-contrib.

// validateMetrics returns a bool representing whether the metric has a valid type and temporality combination and a
// matching metric type and field
func validateMetrics(metric pmetric.Metric) bool {
	switch metric.Type() {
	case pmetric.MetricTypeGauge:
		return metric.Gauge().DataPoints().Len() != 0
	case pmetric.MetricTypeSum:
		return metric.Sum().DataPoints().Len() != 0 && metric.Sum().AggregationTemporality() == pmetric.AggregationTemporalityCumulative
	case pmetric.MetricTypeHistogram:
		return metric.Histogram().DataPoints().Len() != 0 && metric.Histogram().AggregationTemporality() == pmetric.AggregationTemporalityCumulative
	case pmetric.MetricTypeSummary:
		return metric.Summary().DataPoints().Len() != 0
	}
	return false
}

Before version 0.32.0 we had at least one DataPoint.

func sumPoint(record export.Record, num number.Number, start, end time.Time, temporality aggregation.Temporality, monotonic bool) (*metricpb.Metric, error) {}

After 0.32.0 we have zero or more DataPoint into grpc request to opentelemetry-collector.

Environment

• OS: Linux
• Architecture: x86
• Go Version: 1.18
• opentelemetry-go version: latest

Steps To Reproduce

1. Create counter metric with name, example, accept_request
2. Don't increase the counter
3. Export the counter to opentelemetry-collector-contib. It will happen automatically.
4. Export the counter by prometheusremotewrite method (example to victoria metrics) from opentelemetry-collector-contib. It will happen automatically.
5. We get error "invalid temporality and type combination".

Expected behavior

On the one hand, I expect this error to not occur.

On the other hand, I want the metric with the current value to be sent to opentelemetry-collector-contrib
and then to victoria metrics by prometheusremotewrite method.

[MrAlias]

Can you clarify your "steps to reproduce" with code?

I'm very confused by this issue. Are you wondering why a counter that you created with this repository, https://github.com/open-telemetry/opentelemetry-go, doesn't export any data if you don't record any values?

[krupyansky]

https://github.com/krupyansky/otel-bug

1. make init
2. make curl-metrics
3. wait one-two minutes
4. make logs-otel
5. and you can see "Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: Permanent error: Post "victoriametrics:8428": unsupported protocol scheme "victoriametrics"", "dropped_items": 2}"

[MrAlias]

"Permanent error: Permanent error: Post "victoriametrics:8428": unsupported protocol scheme "victoriametrics""

It looks like you have an "unsupported protocol scheme" in your collector configuration. I would recommend correcting your configured endpoint.

[krupyansky]

Yes, thanks for the note @MrAlias

Made changes in https://github.com/krupyansky/otel-bug.

Steps:

1. make init
2. make curl-metrics
3. wait one-two minutes
4. make logs-otel
5. and you can see "Exporting failed. The error is not retryable. Dropping data. {"kind": "exporter", "data_type": "metrics", "name": "prometheusremotewrite", "error": "Permanent error: invalid temporality and type combination", "dropped_items": 2}"

[MrAlias]

If we returned nil here:

opentelemetry-go/sdk/metric/internal/sum.go

Line 149 in 49b62ae

return out

Instead of a value, then this:

opentelemetry-go/sdk/metric/pipeline.go

Lines 134 to 135 in 49b62ae

	data := inst.aggregator.Aggregation()
	if data != nil {

Would drop the empty aggregation and the data would not be sent.

I think this is appropriate. I was considering if this current behavior is desirable because it signals the instrument exists and no measurements have been made. But that is not important. It is only important to signal that existing measurements already made have had no increment.