opentelemetry/exporter-metrics-otlp-http Does not send metrics to collector (with last collector version 0.40.0) #2675

Nighttraveler · 2021-12-17T16:33:29Z

Hello guys, We wanted to use https://github.com/GoogleChrome/web-vitals#browser-support to get some metrics from our browser instrumentation and send them through opentelemetry-js. In order to accomplish that we started to make some tests with the examples provided here.

What version of OpenTelemetry are you using?

I'm using 0.27.0 opentelemetry-js version (this repo in main branch) and otelcontribcol 0.40.0

What version of Node are you using?

v12.16.3

Please provide the code you used to setup the OpenTelemetry SDK

At the moment the exporter does not work with the examples provided.
Just changed this in the example (tracer-web/metrics/index.js)
const labels = { pid: 1, environment: 'staging' }; because 'process' is undefined

What did you do?

We just simply start up the 'tracer-web' examples.
All the traces examples work as expected and arrive to our collector, but metrics doesn't.

If possible, provide a recipe for reproducing the error.

Have your collector running
cd opentelemetry-js/examples/tracer-web
npm install
in metrics/index.js change:
const labels = { pid: process.pid, environment: 'staging' }; to const labels = { pid: 1, environment: 'staging' };
npm run start
Go to http://localhost:8090/metrics/ and click 'Start metrics'

What did you expect to see?

I expected to see the metrics in our collector/backed, like all the other traces examples that works correctly.

What did you see instead?

In Network tab from the browser I have a 400 Bad request response from the collector.

{"code":3,"message":"unknown field \"doubleSum\" in v1.Metric"}

Additional context

traces are exported to localhost:55681/v1/traces (default legacy endpoint)
metrics are exported to localhost:55681/v1/metrics (default legacy endpoint)
Docker compose collector file:

version: "2"
services:

  # Collector
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.40.0
    command: ["--config=/etc/otel-collector-local-config.yaml"]
    volumes:
      - ./otel-collector-local-config.yaml:/etc/otel-collector-local-config.yaml
      - /home/fernando/workspace/khoros/agent/collector-volume:/etc/collector-logging/
    environment:
      LOG_EXPORTER_LOG_LEVEL: "DEBUG"
      NEW_RELIC_API_KEY: "key123"
    ports:
      - "1888:1888"   # pprof extension
      - "13133:13133" # health_check extension
      - "4317:4317"   # OTLP gRPC receiver
      - "4318:4318"   # OTLP HTTP receiver
      - "55681:55681" # default legacy OTLP HTTP receiver
      - "55670:55679" # zpages extension
      - "8888:8888"   # Prometheus metrics exposed by the collector
      - "8889:8889"   # Prometheus exporter metrics

Collector yml config file:

receivers:
  otlp:
    protocols:
      grpc:
      http:
        cors_allowed_origins:
        - http://*
        - https://*        
exporters:
  logging:
    logLevel: $LOG_EXPORTER_LOG_LEVEL
  otlp:
    endpoint: https://otlp.nr-data.net:4317
    headers:
      "api-key": $NEW_RELIC_API_KEY
processors:
  batch:
extensions:
  health_check: {}
service:
  extensions: [health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp, logging]
    metrics:
      receivers: [otlp]
      exporters: [otlp, logging]

Hope the information is enough to reproduce the issue, if not, please reach me and I provide more details.

The text was updated successfully, but these errors were encountered:

morigs · 2021-12-17T20:47:14Z

Same issue here. Also tested with otelcol 0.41.0, otelcontribcol 0.40.0 and otelcontribcol 0.41.0

morigs · 2021-12-17T20:56:39Z

I believe this is due to inconsistencies between payload of the Metrics and actual proto type.

This is caused by this change. This changed in proto 0.9.0 which delivered to the collector in version 0.33.0.

Perhaps fix is quite easy: just rename fields to the new scheme and remove unused (int) fields.

dyladan · 2021-12-17T20:57:37Z

Yeah our proto is a little out of date. i'll update it

morigs · 2021-12-17T21:02:09Z

Looking forward! If any help is needed, just let me know

dyladan · 2021-12-17T21:03:15Z

I won't have time to tackle this until monday so if you want to do it before then I will try to keep an eye out for notifications and merge/release the fix over the weekend. If you can wait until monday i'll just handle it then.

morigs · 2021-12-21T06:40:29Z

@dyladan sorry for pinging you, is there any success?

RomeroGaliza · 2021-12-23T19:34:44Z

Same issue here!

morigs · 2022-01-10T15:55:31Z

@dyladan sorry for disturbing you again. Could you pls add help-needed badge or something. Just to draw attention of possible contributors to the issue.
Or maybe you can provide a high-level list of steps required for proto upgrade, so I can do it myself? That would be awesome

dyladan · 2022-01-10T16:02:29Z

In order to get this working, the proto transformations need to be updated. The transformations are in experimental/packages/opentelemetry-exporter-metrics-otlp-http/src/transformMetrics.ts and depend on the types in packages/exporter-trace-otlp-http/src/types.ts. Because of this unfortunate coupling, the proto files, their types (hand rolled), and the transformations (also hand rolled) need to be updated for all 6 otlp exporters at the same time. If you want to get the fix out ASAP, I would recommend you do that.

Alternatively, I am working on a separate package which will handle transformations into the intermediate OTLP JSON format and serialization as protobuf in #2691. Once that package is completed, the metrics exporters will be refactored to use it (and it already has updated proto). Depending on how time-sensitive this is for you, it may be prudent to just wait for that (i would estimate a couple weeks).

mbecca · 2022-03-09T15:44:25Z

Hello, is there any progress?

sgracias1 · 2022-04-11T05:31:08Z

I ran a test with the OTLP trace exporter and noticed it fails to connect to the collector for version > 0.42.0. Does this issue track the trace exporter as well, or should that be a separate issue?

hejkerooo · 2022-04-13T15:34:50Z

"message":"Bad Request","name":"OTLPExporterError","data":"{\"code\":3,\"message\":\"unknown field \\\"doubleSum\\\" in v1.Metric\"}","code":"400"}

This is the error I'm receiving when using metrics exporter

  otlp:
    protocols:
      grpc:
      http:
  zipkin:

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  logging:
  zipkin:
    endpoint: "http://zipkin:9411/api/v2/spans"

processors:
  batch:

extensions:
  health_check:
  pprof:
  zpages:

service:
  extensions: [pprof, zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp, zipkin]
      exporters: [zipkin, jaeger, logging]
      processors: [batch]
    metrics:
      receivers: [ otlp ]
      exporters: [ logging ]

const metricExporter = new OTLPMetricExporter({
  url: 'http://localhost:4318/v1/metrics',
});

radoslavirha · 2022-05-18T13:26:53Z

It's still an issue with "@opentelemetry/exporter-metrics-otlp-http": "0.28.0" and otel/opentelemetry-collector-contrib:0.51.0. Any ideas how to fix it? Thanks

gvelez17 · 2022-05-25T14:40:41Z

Using same versions here (0.28.0 and 0.51.0) and also not seeing metrics appearing in the collector

jeevanearform · 2022-05-25T15:14:14Z

Same issue here. I am on @opentelemetry/exporter-metrics-otlp-http:0.28.0 and using docker image otel/opentelemetry-collector:latest

mplachter · 2022-05-26T15:29:55Z

Same issue here. I thought the latests 0.28.0 was going to have a newer otlp metric proto but it does not and still will not work with anything passed 0.42.0

mplachter · 2022-05-26T15:35:35Z

It's still an issue with "@opentelemetry/exporter-metrics-otlp-http": "0.28.0" and otel/opentelemetry-collector-contrib:0.51.0. Any ideas how to fix it? Thanks

gotta use 0.42.0 or older it seems.

radoslavirha · 2022-05-26T15:42:22Z

I had to use 0.29.0 docker image .... I had some trouble with <=0.42.0 too

mplachter · 2022-05-31T16:06:05Z

@dyladan any progress on this?

dyladan · 2022-05-31T16:08:05Z

I'm sorry i forgot to create a release but 1.3.0/0.29.0 was released at the end of last week #2995 the most recent version should work

mplachter · 2022-06-02T16:21:46Z

@dyladan trying both 0.29.0 and 0.29.1 and i'm getting these grpc errors flooding the collector with no traces or metrics.

2022-06-02T16:19:46.988Z        warn    zapgrpc/zapgrpc.go:191  [core] grpc: Server.Serve failed to create ServerTransport:  connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"\\x16\\x03\\x01\\x01o\\x01\\x00\\x01k\\x03\\x03\\xca\\xd0\\v\\xfd\\xc8\\u007f)\\x8av\\xd3\\x11\\xcci\""        {"grpc_log": true}
2022-06-02T16:19:50.642Z        warn    zapgrpc/zapgrpc.go:191  [core] grpc: Server.Serve failed to create ServerTransport:  connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"\\x16\\x03\\x01\\x01o\\x01\\x00\\x01k\\x03\\x03\\x06[u\\x00\\b8/\\x16aM;\\xec\\x92\""  {"grpc_log": true}
2022-06-02T16:19:50.642Z        warn    zapgrpc/zapgrpc.go:191  [core] grpc: Server.Serve failed to create ServerTransport:  connection error: desc = "transport: http2Server.HandleStreams received bogus greeting from client: \"\\x16\\x03\\x01\\x01o\\x01\\x00\\x01k\\x03\\x03\\xf5\\x15\\xb6S\\xef\\x1aZ\\x15|\\t\\x05\\xd6{\""

Tired with 0.42.0, 0.44.0 and 0.52.0 otel collector contribs

dyladan · 2022-06-02T16:29:50Z

That looks like a collector issue not an issue with OTel js

dyladan · 2022-06-02T16:30:30Z

@pichlermarc if you have time can you look into this?

mplachter · 2022-06-02T16:31:29Z

I'm running this locally in a docker compose.. Not in a deployed env.

I'll double check again to make sure nothing is trying to communicate with otel over the grpc port.

mplachter · 2022-06-02T16:43:25Z

Yeah I did double check and also i get this from the otel-js side.

code":14,"details":"No connection established","metadata":{}},"exception":true,"hostname":"machine_name","level":"error","message":"uncaughtException: 14 UNAVAILABLE: No connection established\nError: 14 UNAVAILABLE: No connection established\n    at Object.callErrorFromStatus (/..../@grpc/grpc-js/src/call.ts:81:24)

mplachter · 2022-06-02T16:45:54Z

The Go modules work no problem with the collectors only getting this behavior using the js packages.

pichlermarc · 2022-06-03T11:50:17Z

@mplachter you're using @opentelemetry/exporter-metrics-otlp-grpc, right? 🤔

I looked into your issue and found that the @opentelemetry/exporter-metrics-otlp-http works, but @opentelemetry/exporter-metrics-otlp-grpc is currently broken. It seems like a PR after #2886 introduced another bug. This causes the exporter to try and open a secure connection on an insecure server.

I'm currently working on a fix.

Edit: the fix should be ready over at #3019 🙂

dyladan · 2022-06-28T19:27:03Z

@mplachter @pichlermarc can this be closed?

pichlermarc · 2022-06-29T14:28:53Z

@dyladan for me it is working again. But we have not released that fix yet.

lamroger · 2023-01-30T22:30:51Z

Running into this again on 0.35.1. I'm using RemixInstrumentation. Maybe that is sending the wrong shape?

pichlermarc · 2023-01-31T07:35:13Z

Hi @lamroger, thanks for reaching out. Could you please open a new bug for the problem you're seeing?

It has gotten a bit hard to follow the thread here, and a lot has changed since this issue here was first opened.

Pratap22 · 2023-08-09T09:26:57Z

I keep getting this error.

{"stack":"OTLPExporterError: Not Found\n    at IncomingMessage.<anonymous> (/Users/admin/personal/otel/otel-collector-sender/node_modules/@opentelemetry/otlp-exporter-base/build/src/platform/node/util.js:103:39)\n    at IncomingMessage.emit (node:events:402:35)\n    at endReadableNT (node:internal/streams/readable:1343:12)\n    at processTicksAndRejections (node:internal/process/task_queues:83:21)","message":"Not Found","name":"OTLPExporterError","data":"404 page not found\n","code":"404"}

package.json

{
"dependencies": {
    "@opentelemetry/api": "^1.4.1",
    "@opentelemetry/sdk-node": "^0.41.2",
    "opentelemetry-instrumentation-express": "^0.39.1"
  }

}

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:8000
      http:
        endpoint: 0.0.0.0:4318

exporters:
  prometheus:
    endpoint: collector:7777
    namespace: mysearcher


extensions:
  pprof:
    endpoint: :1888
  zpages:
    endpoint: :55679

service:
  pipelines:
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
  telemetry:
    logs:
      level: info

version: '3.5'
services:
  prometheus:
    image: prom/prometheus
    container_name: prometheus
    ports:
      - 9090:9090
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - ./prometheus_data:/prometheus
  collector:
    image: otel/opentelemetry-collector-contrib
    container_name: collector
    command: [--config=/etc/otel-collector-config.yaml]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - 8000:8000
      - "4318:4318"
  grafana:
    image: grafana/grafana
    container_name: grafana
    ports:
      - 3000:3000

Can you help me why am I getting the error?

pj-pi · 2024-02-04T07:54:07Z

Hi @Pratap22 , I got the same error as yours. Did you manage to solve the issue?

Nighttraveler added the bug Something isn't working label Dec 17, 2021

morigs mentioned this issue Feb 21, 2022

feat(proto): add @opentelemetry/otlp-transformer package with hand-rolled transformation #2746

Merged

2 tasks

This was referenced Apr 5, 2022

Update OTLP Metrics Exporter with latest Metrics SDK #2774

Closed

Update proto versions of OTLP exporters #2886

Closed

pichlermarc mentioned this issue Apr 26, 2022

feat(exporters): update proto version and use otlp-transformer #2929

Merged

9 tasks

pichlermarc mentioned this issue Jun 3, 2022

fix(grpc-exporter): use non-normalized URL to determine channel security #3019

Merged

9 tasks

dyladan linked a pull request Jun 30, 2022 that will close this issue

release proposal 1.4.0/0.30.0 #3065

Merged

dyladan closed this as completed in #3065 Jul 6, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

opentelemetry/exporter-metrics-otlp-http Does not send metrics to collector (with last collector version 0.40.0) #2675

opentelemetry/exporter-metrics-otlp-http Does not send metrics to collector (with last collector version 0.40.0) #2675

Nighttraveler commented Dec 17, 2021 •

edited

Loading

morigs commented Dec 17, 2021

morigs commented Dec 17, 2021

dyladan commented Dec 17, 2021

morigs commented Dec 17, 2021

dyladan commented Dec 17, 2021

morigs commented Dec 21, 2021

RomeroGaliza commented Dec 23, 2021

morigs commented Jan 10, 2022

dyladan commented Jan 10, 2022

mbecca commented Mar 9, 2022

sgracias1 commented Apr 11, 2022 •

edited

Loading

hejkerooo commented Apr 13, 2022

radoslavirha commented May 18, 2022

gvelez17 commented May 25, 2022

jeevanearform commented May 25, 2022

mplachter commented May 26, 2022

mplachter commented May 26, 2022 •

edited

Loading

radoslavirha commented May 26, 2022

mplachter commented May 31, 2022

dyladan commented May 31, 2022

mplachter commented Jun 2, 2022 •

edited

Loading

dyladan commented Jun 2, 2022

dyladan commented Jun 2, 2022

mplachter commented Jun 2, 2022

mplachter commented Jun 2, 2022 •

edited

Loading

mplachter commented Jun 2, 2022 •

edited

Loading

pichlermarc commented Jun 3, 2022 •

edited

Loading

dyladan commented Jun 28, 2022

pichlermarc commented Jun 29, 2022

lamroger commented Jan 30, 2023 •

edited

Loading

pichlermarc commented Jan 31, 2023

Pratap22 commented Aug 9, 2023 •

edited

Loading

pj-pi commented Feb 4, 2024

opentelemetry/exporter-metrics-otlp-http Does not send metrics to collector (with last collector version 0.40.0) #2675

opentelemetry/exporter-metrics-otlp-http Does not send metrics to collector (with last collector version 0.40.0) #2675

Comments

Nighttraveler commented Dec 17, 2021 • edited Loading

What version of OpenTelemetry are you using?

What version of Node are you using?

Please provide the code you used to setup the OpenTelemetry SDK

What did you do?

What did you expect to see?

What did you see instead?

Additional context

morigs commented Dec 17, 2021

morigs commented Dec 17, 2021

dyladan commented Dec 17, 2021

morigs commented Dec 17, 2021

dyladan commented Dec 17, 2021

morigs commented Dec 21, 2021

RomeroGaliza commented Dec 23, 2021

morigs commented Jan 10, 2022

dyladan commented Jan 10, 2022

mbecca commented Mar 9, 2022

sgracias1 commented Apr 11, 2022 • edited Loading

hejkerooo commented Apr 13, 2022

radoslavirha commented May 18, 2022

gvelez17 commented May 25, 2022

jeevanearform commented May 25, 2022

mplachter commented May 26, 2022

mplachter commented May 26, 2022 • edited Loading

radoslavirha commented May 26, 2022

mplachter commented May 31, 2022

dyladan commented May 31, 2022

mplachter commented Jun 2, 2022 • edited Loading

dyladan commented Jun 2, 2022

dyladan commented Jun 2, 2022

mplachter commented Jun 2, 2022

mplachter commented Jun 2, 2022 • edited Loading

mplachter commented Jun 2, 2022 • edited Loading

pichlermarc commented Jun 3, 2022 • edited Loading

dyladan commented Jun 28, 2022

pichlermarc commented Jun 29, 2022

lamroger commented Jan 30, 2023 • edited Loading

pichlermarc commented Jan 31, 2023

Pratap22 commented Aug 9, 2023 • edited Loading

pj-pi commented Feb 4, 2024

Nighttraveler commented Dec 17, 2021 •

edited

Loading

sgracias1 commented Apr 11, 2022 •

edited

Loading

mplachter commented May 26, 2022 •

edited

Loading

mplachter commented Jun 2, 2022 •

edited

Loading

mplachter commented Jun 2, 2022 •

edited

Loading

mplachter commented Jun 2, 2022 •

edited

Loading

pichlermarc commented Jun 3, 2022 •

edited

Loading

lamroger commented Jan 30, 2023 •

edited

Loading

Pratap22 commented Aug 9, 2023 •

edited

Loading