Gauge metrics stop getting exported after 10-12 hours of running

[tgrieger-sf]

I'm setting up an observable gauge:

MeterContainer.Meter.CreateObservableGauge(AVERAGE_LOG_CONSUMPTION_METRIC_NAME, () =>
{
	List<Measurement<double>> measurements = new();
	AddMeasurements(measurements, _instancesContainer.Inserts, LogConsumptionType.Insert);
	AddMeasurements(measurements, _instancesContainer.Deletes, LogConsumptionType.Delete);
	AddMeasurements(measurements, _instancesContainer.Updates, LogConsumptionType.Update);

	_logger.LogDebug("All measurements added: {count}", measurements.Count);
	return measurements;
});

Because of the added LogDebug, I'm seeing how many measurements I'm getting back and it continues to be between 500 and 3000 at any given time. In my exporter, I'm also logging what metrics are going through my exporter and see that it's not reaching the exporter at all. Other metrics continue to work after this gauge stops working.

I've tried enabling the environment variables for extra OpenTelemetry logging outlined here but no extra logs come through.

Anyone know of why this might be happening? I'm using OpenTelemetry 1.3.2 and OpenTelemetry.Extensions.Hosting 1.0.0-rc9.6 and on .NET 6.0 running on a dotnet/aspnet:6.0-alpine container running in Kubernetes.

[alanwest]

I've tried enabling the environment variables for extra OpenTelemetry logging outlined here but no extra logs come through.

Are you using the opentelemetry-dotnet-instrumentation project? Or are you just using the SDK and programmatically configuring metrics? If the latter, then the environment variables documented here do not apply.

If using the SDK, you can try enabling self-diagnostics to see if that surfaces up any issues.

[Kielek]

@alanwest, issue transferred from AutoInstrumentation project open-telemetry/opentelemetry-dotnet-instrumentation#1903.
As I know, not related to auto-instrumentation.

[tgrieger-sf]

This can be closed, discussed in #2360 (comment) and now know the real issue.