Prometheus receiver: log error message when process_start_time_seconds gauge is missing

[nilebox]

Prometheus receiver supports the flag use_start_time_metric: true.
When this flag is enabled, every Prometheus endpoint must have the process_start_time_seconds gauge, e.g.

# TYPE process_start_time_seconds gauge
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
process_start_time_seconds 1508230997

When this metric is present, it will set the startTime in

opentelemetry-collector/receiver/prometheusreceiver/internal/metricsbuilder.go

Lines 106 to 108 in 66d7acd

	} else if b.useStartTimeMetric && metricName == startTimeMetricName {
	b.startTime = v
	}

and will be used for adjusting time in metrics - this is expected behavior.

But if the process_start_time_seconds gauge is missing, startTime will have the default zero value.

The problem is that during transaction commit phase, we perform the check

opentelemetry-collector/receiver/prometheusreceiver/internal/transaction.go

Lines 168 to 171 in 66d7acd

	// AdjustStartTime - startTime has to be non-zero in this case.
	if tr.metricBuilder.startTime == 0.0 {
	metrics = []*metricspb.Metric{}
	droppedTimeseries = numTimeseries

which silently drops all metrics and updates the local variable droppedTimeseries that is never used.

As a result, debugging this corner case is a nightmare, and the only way to discover all of this is to use Go debugger (that's how I found this issue).

At the very least, we should log a message when metrics get dropped.
I would suggest using WARN level for this specific issue, but if it's considered normal for some situations, we should print INFO or DEBUG message.

---

Is there also some other existing way of reporting counters like droppedTimeseries?

[nilebox]

@dinooliva as the author of PR #394 which introduced this flag, you may have more context on this, and share your opinion?

[rf232]

I also got hit by this and second @nilebox' desire for at least some kind of logging in this situation and maybe adding this to some exposed metric