GMP exporter claims unsupported metric data type (data_type empty)

[matthewcyy]

Component(s)

exporter/googlemanagedprometheus

Describe the issue you're reporting

Hi, I have a cloud run and a kubernetes deployment for my OTel collector. But I notice both will have logs with this error at about every second:

error   collector@v0.45.0/metrics.go:780        Unsupported metric data type    {"kind": "exporter", "data_type": "metrics", "name": "googlemanagedprometheus", "data_type": "Empty"}
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(*metricMapper).metricToTimeSeries
        github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.45.0/metrics.go:780
github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector.(*MetricsExporter).PushMetrics
        github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/collector@v0.45.0/metrics.go:338
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsRequest).Export
        go.opentelemetry.io/collector/exporter@v0.89.0/exporterhelper/metrics.go:58
go.opentelemetry.io/collector/exporter/exporterhelper.(*timeoutSender).send
        go.opentelemetry.io/collector/exporter@v0.89.0/exporterhelper/timeout_sender.go:38
go.opentelemetry.io/collector/exporter/exporterhelper.(*baseRequestSender).send
        go.opentelemetry.io/collector/exporter@v0.89.0/exporterhelper/common.go:33
go.opentelemetry.io/collector/exporter/exporterhelper.(*metricsSenderWithObservability).send
        go.opentelemetry.io/collector/exporter@v0.89.0/exporterhelper/metrics.go:173
go.opentelemetry.io/collector/exporter/exporterhelper.(*queueSender).Start.func1
        go.opentelemetry.io/collector/exporter@v0.89.0/exporterhelper/queue_sender.go:141
go.opentelemetry.io/collector/exporter/exporterhelper/internal.(*boundedMemoryQueue).Start.func1
        go.opentelemetry.io/collector/exporter@v0.89.0/exporterhelper/internal/bounded_memory_queue.go:46

I'm using the logging exporter as well, yet right before this error I get the info message

info    MetricsExporter {"kind": "exporter", "data_type": "metrics", "name": "logging", "resource metrics": 4, "metrics": 47, "data points": 25}

Does anyone know how to resolve this? It seems it will sometimes prevent my data from being exported properly to Google Cloud Monitoring.

[github-actions]

Pinging code owners:

• exporter/googlemanagedprometheus: @aabmass @dashpole @jsuereth @punya @damemi @psx95

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[dashpole]

can you configure the logging exporter with the logLevel: debug, so we can see what is printed?

It it hitting this error: https://github.com/GoogleCloudPlatform/opentelemetry-operations-go/blob/a227a2319903eb2170c2cdf7c9c7aedf73df5503/exporter/collector/metrics.go#L780, which happens when it is sent a metric that is not one of the valid OTel metric types.

[matthewcyy]

Thanks, I added the loglevel:debug but it didn't output anything differently so I added the debug exporter to view the metrics. It seems that some of my histogram metrics have an empty DataType. Other metrics like the counter are fine.

Metric #3
Descriptor:
     -> Name: cpu
     -> Description: Percentage of CPU utilized in the process.
     -> Unit: percent
     -> DataType: Empty
Metric #4
Descriptor:
     -> Name: memory
     -> Description: Amount of memory in bytes utilized in the process.
     -> Unit: B
     -> DataType: Empty
...

It will sometimes have the DataType be histogram but it's almost entirely Empty. I'm using OTLP in Python to record my metrics like this:

from opentelemetry import metrics
from opentelemetry.exporter.otlp.proto.grpc.metric_exporter import OTLPMetricExporter
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader

app_metrics = {}


def init_monitoring():
    """Initializes OpenTelemetry MetricProvider and defines the metrics captured in the app"""
    # Init OTEL metric provider
    otlp_reader = PeriodicExportingMetricReader(OTLPMetricExporter())
    provider = MeterProvider(metric_readers=[otlp_reader])
    metrics.set_meter_provider(provider)
    meter = metrics.get_meter("sequalizer/app")

    # Define Metrics
    app_metrics["cpu"] = meter.create_histogram(
        "cpu",
        unit="percent",
        description="Percentage of CPU utilized in the process.",
    )

    app_metrics["memory"] = meter.create_histogram(
        "memory",
        unit="B",
        description="Amount of memory in bytes utilized in the process.",
    )

and my OTel config is using the googlamanagedprometheus exporter.

[dashpole]

Interesting. That sounds like an issue with OTel python. I would expect the DataType to still have a Histogram data point type in that case... @aabmass

[matthewcyy]

@dashpole @aabmass Actually it seems my issue is the same as this error open-telemetry/opentelemetry-python#3277, where Histograms are being exported as an empty DataType. I'm using these OTel packages:

opentelemetry-api==1.20.0
opentelemetry-exporter-otlp-proto-common==1.20.0
opentelemetry-exporter-otlp-proto-grpc==1.20.0
opentelemetry-instrumentation==0.41b0
opentelemetry-proto==1.20.0
opentelemetry-sdk==1.20.0
opentelemetry-semantic-conventions==0.41b0

Do we know if there's a version I can upgrade to that has fixed this error? It looks like there's been some PR's to fix this issue open-telemetry/opentelemetry-python#3277 (comment)

[dashpole]

Based on open-telemetry/opentelemetry-python@3a651c7, it is fixed in v1.21.0 of opentelemetry-python. I'd bump all the deps that are at 1.20 to 1.21