Issue with gRPC while using auto instrumentation

[makeavish]

What happened?

Steps to Reproduce

While auto instrumenting a node.js app with node auto instrumentation library
Using below command to install OTLP:

npm install --save @opentelemetry/api@^1.4.1                                                                       
npm install --save @opentelemetry/sdk-node@^0.39.1
npm install --save @opentelemetry/auto-instrumentations-node@^0.37.0
npm install --save @opentelemetry/exporter-trace-otlp-http@^0.39.1

Here's tracing.js:

// tracing.js
'use strict'
const process = require('process');
const opentelemetry = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

const exporterOptions = {
url: 'http://localhost:4318/v1/traces'
}

const traceExporter = new OTLPTraceExporter(exporterOptions);
const sdk = new opentelemetry.NodeSDK({
traceExporter,
instrumentations: [getNodeAutoInstrumentations()],
resource: new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: 'node_app'
})
});

// initialize the SDK and register with the OpenTelemetry API
// this enables the API to record telemetry
sdk.start()

// gracefully shut down the SDK on process exit
process.on('SIGTERM', () => {
  sdk.shutdown()
  .then(() => console.log('Tracing terminated'))
  .catch((error) => console.log('Error terminating tracing', error))
  .finally(() => process.exit(0));
  });

Then running the App:

 node -r ./tracing.js app.js

Expected Result

No errors in logs.

Actual Result

I am seeing below warning/error related to gRPC even though my app doesn't has any gRPC imports:

@opentelemetry/instrumentation-grpc Module @grpc/grpc-js has been loaded before @opentelemetry/instrumentation-grpc so it might not work, please initialize it before requiring @grpc/grpc-js

Additional Details

OpenTelemetry Setup Code

No response

package.json

{
  "name": "node-app",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "author": {
    "name": "SigNoz",
    "email": "dev@signoz.io",
    "url": "https://github.com/SigNoz"
  },
  "scripts": {
    "start": "node -r ./tracing.js index.js",
    "start:local": "OTEL_EXPORTER_OTLP_ENDPOINT=\"http://localhost:4317\" OTEL_RESOURCE_ATTRIBUTES=service.name=node-js-sample-app node -r ./tracing.js index.js"
  },
  "dependencies": {
    "@opentelemetry/api": "^1.4.1",
    "@opentelemetry/auto-instrumentations-node": "^0.37.1",
    "@opentelemetry/exporter-trace-otlp-http": "^0.39.1",
    "@opentelemetry/sdk-node": "^0.39.1",
    "cors": "^2.8.5",
    "dd-trace": "^4.3.0",
    "express": "^4.17.3",
    "newrelic": "^10.3.2",
    "nodemon": "^2.0.15"
  }
}

Relevant log output

@opentelemetry/instrumentation-grpc Module @grpc/grpc-js has been loaded before @opentelemetry/instrumentation-grpc so it might not work, please initialize it before requiring @grpc/grpc-js

[Flarna]

Is this still reproducible with latest versions of Otel modules?

[Flarna]

Yes it does. I think the problem is here - the SDK loads the OTLP GRPC exporter which in turn loads grpc-js.

Workaround is to call getNodeAutoInstrumentations() before const opentelemetry = require('@opentelemetry/sdk-node');

[makeavish]

It is reproducible on latest versions of Otel modules as well.

Workaround is to call getNodeAutoInstrumentations() before const opentelemetry = require('@opentelemetry/sdk-node');

This worked!

Here's updated tracing.js:

// tracing.js
'use strict'
const process = require('process');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const instrumentations = getNodeAutoInstrumentations();
const opentelemetry = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

const exporterOptions = {
url: 'http://localhost:4318/v1/traces'
}

const traceExporter = new OTLPTraceExporter(exporterOptions);
const sdk = new opentelemetry.NodeSDK({
traceExporter,
instrumentations: [instrumentations],
resource: new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: 'node_app'
})
});

// initialize the SDK and register with the OpenTelemetry API
// this enables the API to record telemetry
sdk.start()

// gracefully shut down the SDK on process exit
process.on('SIGTERM', () => {
  sdk.shutdown()
  .then(() => console.log('Tracing terminated'))
  .catch((error) => console.log('Error terminating tracing', error))
  .finally(() => process.exit(0));
  });

[makeavish]

Closing this issue as I guess calling getNodeAutoInstrumentations() before const opentelemetry = require('@opentelemetry/sdk-node'); is the only solution.

[beggers]

I'm having this same problem. I tried the solution in this thread but it isn't working. Here's my code:

const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const autoInstrumentations = getNodeAutoInstrumentations({
  '@opentelemetry/instrumentation-fs': {
    enabled: false,
  },
});
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { diag } = require('@opentelemetry/api');
const { PrismaInstrumentation } = require('@prisma/instrumentation');
const { TELEMETRY_SERVICE_NAME } = require("./config");
const {
  trace,
} = require('@opentelemetry/api');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');

const exporter = new OTLPTraceExporter();

const sdk = new NodeSDK({
  traceExporter: exporter,
  instrumentations: [autoInstrumentations, new PrismaInstrumentation()],
});

if (process.env.OTEL_EXPORTER_OTLP_HEADERS && process.env.OTEL_EXPORTER_OTLP_ENDPOINT) {
  try {
    sdk.start();
    diag.info('OpenTelemetry automatic instrumentation started successfully');
  } catch (error) {
    diag.error(
      'Error initializing OpenTelemetry SDK. Your application is not instrumented and will not produce telemetry',
      error
    );
  }
} else {
  diag.info('OpenTelemetry automatic instrumentation disabled');
}

export const tracer = trace.getTracer(TELEMETRY_SERVICE_NAME);

Any recommendations for how to debug this? I'm trying to clean up our log pollution and this one is particularly annoying :D

[Flarna]

Any recommendations for how to debug this?

Set a breakpoint in the file node_modules/@grpc/grpc-js/build/src/index.js and take a look into the callstack where this file gets required.

[beggers]

I'll look into doing so when I have a minute -- I don't have much experience with JS so figuring out how to hold the debugger right is probably beyond my available time today. @Flarna can we re-open this issue as it appears to not have been fixed?

[Flarna]

According to #4145 (comment) it is solved.

Seems you have a new problem which looks similar but this doesn't imply it's the same problem.

What about creating a new issue once you verified that call sequence is correct in your setup?

[ilia-beliaev-miro]

Should the README be updated then, as currently it still states that const opentelemetry = require('@opentelemetry/sdk-node'); goes before const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');