Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with gRPC while using auto instrumentation #4145

Closed
makeavish opened this issue Sep 14, 2023 · 9 comments
Closed

Issue with gRPC while using auto instrumentation #4145

makeavish opened this issue Sep 14, 2023 · 9 comments
Labels
bug Something isn't working triage

Comments

@makeavish
Copy link

What happened?

Steps to Reproduce

While auto instrumenting a node.js app with node auto instrumentation library
Using below command to install OTLP:

npm install --save @opentelemetry/api@^1.4.1                                                                       
npm install --save @opentelemetry/sdk-node@^0.39.1
npm install --save @opentelemetry/auto-instrumentations-node@^0.37.0
npm install --save @opentelemetry/exporter-trace-otlp-http@^0.39.1

Here's tracing.js:

// tracing.js
'use strict'
const process = require('process');
const opentelemetry = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

const exporterOptions = {
url: 'http://localhost:4318/v1/traces'
}

const traceExporter = new OTLPTraceExporter(exporterOptions);
const sdk = new opentelemetry.NodeSDK({
traceExporter,
instrumentations: [getNodeAutoInstrumentations()],
resource: new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: 'node_app'
})
});

// initialize the SDK and register with the OpenTelemetry API
// this enables the API to record telemetry
sdk.start()

// gracefully shut down the SDK on process exit
process.on('SIGTERM', () => {
  sdk.shutdown()
  .then(() => console.log('Tracing terminated'))
  .catch((error) => console.log('Error terminating tracing', error))
  .finally(() => process.exit(0));
  });

Then running the App:

 node -r ./tracing.js app.js

Expected Result

No errors in logs.

Actual Result

I am seeing below warning/error related to gRPC even though my app doesn't has any gRPC imports:

@opentelemetry/instrumentation-grpc Module @grpc/grpc-js has been loaded before @opentelemetry/instrumentation-grpc so it might not work, please initialize it before requiring @grpc/grpc-js

Additional Details

OpenTelemetry Setup Code

No response

package.json

{
  "name": "node-app",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "author": {
    "name": "SigNoz",
    "email": "dev@signoz.io",
    "url": "https://github.com/SigNoz"
  },
  "scripts": {
    "start": "node -r ./tracing.js index.js",
    "start:local": "OTEL_EXPORTER_OTLP_ENDPOINT=\"http://localhost:4317\" OTEL_RESOURCE_ATTRIBUTES=service.name=node-js-sample-app node -r ./tracing.js index.js"
  },
  "dependencies": {
    "@opentelemetry/api": "^1.4.1",
    "@opentelemetry/auto-instrumentations-node": "^0.37.1",
    "@opentelemetry/exporter-trace-otlp-http": "^0.39.1",
    "@opentelemetry/sdk-node": "^0.39.1",
    "cors": "^2.8.5",
    "dd-trace": "^4.3.0",
    "express": "^4.17.3",
    "newrelic": "^10.3.2",
    "nodemon": "^2.0.15"
  }
}

Relevant log output

@opentelemetry/instrumentation-grpc Module @grpc/grpc-js has been loaded before @opentelemetry/instrumentation-grpc so it might not work, please initialize it before requiring @grpc/grpc-js
@makeavish makeavish added bug Something isn't working triage labels Sep 14, 2023
@Flarna
Copy link
Member

Flarna commented Sep 14, 2023

Is this still reproducible with latest versions of Otel modules?

@Flarna
Copy link
Member

Flarna commented Sep 14, 2023

Yes it does. I think the problem is here - the SDK loads the OTLP GRPC exporter which in turn loads grpc-js.

Workaround is to call getNodeAutoInstrumentations() before const opentelemetry = require('@opentelemetry/sdk-node');

@makeavish
Copy link
Author

makeavish commented Sep 14, 2023

It is reproducible on latest versions of Otel modules as well.

Workaround is to call getNodeAutoInstrumentations() before const opentelemetry = require('@opentelemetry/sdk-node');

This worked!

Here's updated tracing.js:

// tracing.js
'use strict'
const process = require('process');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const instrumentations = getNodeAutoInstrumentations();
const opentelemetry = require('@opentelemetry/sdk-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');
const { Resource } = require('@opentelemetry/resources');
const { SemanticResourceAttributes } = require('@opentelemetry/semantic-conventions');

const exporterOptions = {
url: 'http://localhost:4318/v1/traces'
}

const traceExporter = new OTLPTraceExporter(exporterOptions);
const sdk = new opentelemetry.NodeSDK({
traceExporter,
instrumentations: [instrumentations],
resource: new Resource({
  [SemanticResourceAttributes.SERVICE_NAME]: 'node_app'
})
});

// initialize the SDK and register with the OpenTelemetry API
// this enables the API to record telemetry
sdk.start()

// gracefully shut down the SDK on process exit
process.on('SIGTERM', () => {
  sdk.shutdown()
  .then(() => console.log('Tracing terminated'))
  .catch((error) => console.log('Error terminating tracing', error))
  .finally(() => process.exit(0));
  });

@makeavish
Copy link
Author

Closing this issue as I guess calling getNodeAutoInstrumentations() before const opentelemetry = require('@opentelemetry/sdk-node'); is the only solution.

@beggers
Copy link

beggers commented Nov 29, 2023

I'm having this same problem. I tried the solution in this thread but it isn't working. Here's my code:

const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const autoInstrumentations = getNodeAutoInstrumentations({
  '@opentelemetry/instrumentation-fs': {
    enabled: false,
  },
});
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { diag } = require('@opentelemetry/api');
const { PrismaInstrumentation } = require('@prisma/instrumentation');
const { TELEMETRY_SERVICE_NAME } = require("./config");
const {
  trace,
} = require('@opentelemetry/api');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc');

const exporter = new OTLPTraceExporter();

const sdk = new NodeSDK({
  traceExporter: exporter,
  instrumentations: [autoInstrumentations, new PrismaInstrumentation()],
});

if (process.env.OTEL_EXPORTER_OTLP_HEADERS && process.env.OTEL_EXPORTER_OTLP_ENDPOINT) {
  try {
    sdk.start();
    diag.info('OpenTelemetry automatic instrumentation started successfully');
  } catch (error) {
    diag.error(
      'Error initializing OpenTelemetry SDK. Your application is not instrumented and will not produce telemetry',
      error
    );
  }
} else {
  diag.info('OpenTelemetry automatic instrumentation disabled');
}

export const tracer = trace.getTracer(TELEMETRY_SERVICE_NAME);

Any recommendations for how to debug this? I'm trying to clean up our log pollution and this one is particularly annoying :D

@Flarna
Copy link
Member

Flarna commented Nov 29, 2023

Any recommendations for how to debug this?

Set a breakpoint in the file node_modules/@grpc/grpc-js/build/src/index.js and take a look into the callstack where this file gets required.

@beggers
Copy link

beggers commented Nov 29, 2023

I'll look into doing so when I have a minute -- I don't have much experience with JS so figuring out how to hold the debugger right is probably beyond my available time today. @Flarna can we re-open this issue as it appears to not have been fixed?

@Flarna
Copy link
Member

Flarna commented Nov 29, 2023

According to #4145 (comment) it is solved.

Seems you have a new problem which looks similar but this doesn't imply it's the same problem.

What about creating a new issue once you verified that call sequence is correct in your setup?

@ilia-beliaev-miro
Copy link

Should the README be updated then, as currently it still states that const opentelemetry = require('@opentelemetry/sdk-node'); goes before const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working triage
Projects
None yet
Development

No branches or pull requests

4 participants