Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Gracefull shutdown doesn't work with GRPC Exporter #4611

Closed
krim opened this issue Apr 8, 2024 · 3 comments · Fixed by #4612
Closed

Gracefull shutdown doesn't work with GRPC Exporter #4611

krim opened this issue Apr 8, 2024 · 3 comments · Fixed by #4612
Assignees
Labels
bug Something isn't working pkg:exporter-logs-otlp-grpc pkg:exporter-metrics-otlp-grpc pkg:exporter-trace-otlp-grpc priority:p1 Bugs which cause problems in end-user applications such as crashes, data inconsistencies, etc

Comments

@krim
Copy link

krim commented Apr 8, 2024

What happened?

Steps to Reproduce

app.js

const tracingSDK = require('./tracing')
const express = require('express')
const http = require('http')
const server = http.createServer(app)
const port = 8888
server.listen(port)
console.log('Start listening on port ' + port)

function shutdown (signal) {
  const second = 1000
  const shutdownDelay = 15 * second
  console.log(`Receive signal ${signal} wait ${shutdownDelay}ms`)
  setTimeout(gracefulShutdown, shutdownDelay)
}

process.on('SIGTERM', shutdown)
process.on('SIGINT', shutdown)

function gracefulShutdown () {
  console.log('HTTP server close start')
  server.close((err) => {
    if (err) {
      console.log(err)
    } else {
      console.log('HTTP server close success')
    }

    tracingSDK.shutdown()
      .then(() => console.log('Tracing terminated'))
      .catch((error) => console.log(error))
      .finally(() => process.exit(0))
    })
  })
}

Start an app:

npm run start

Expected Result

When you stop an app you expect to get Tracing terminated

Actual Result

TypeError: _a.shutdown is not a function
  File "TypeError: _a.shutdown is not a function"
  File " at GrpcExporterTransport.shutdown (/usr/src/app/node_modules/@opentelemetry/otlp-grpc-exporter-base/build/src/grpc-exporter-transport.js", line 55, in 69)
  File " at OTLPTraceExporter.onShutdown (/usr/src/app/node_modules/@opentelemetry/otlp-grpc-exporter-base/build/src/OTLPGRPCExporterNodeBase.js", line 75, in 25)
  File " at OTLPTraceExporter._shutdown (/usr/src/app/node_modules/@opentelemetry/otlp-exporter-base/build/src/OTLPExporterBase.js", line 103, in 14)
  File " at BindOnceFuture.call (/usr/src/app/node_modules/@opentelemetry/core/build/src/utils/callback.js", line 40, in 48)
  File " at OTLPTraceExporter.shutdown (/usr/src/app/node_modules/@opentelemetry/otlp-exporter-base/build/src/OTLPExporterBase.js", line 88, in 35)
  File " at SimpleSpanProcessor._shutdown (/usr/src/app/node_modules/@opentelemetry/sdk-trace-base/build/src/export/SimpleSpanProcessor.js", line 81, in 31)
  File " at BindOnceFuture.call (/usr/src/app/node_modules/@opentelemetry/core/build/src/utils/callback.js", line 40, in 48)
  File " at SimpleSpanProcessor.shutdown (/usr/src/app/node_modules/@opentelemetry/sdk-trace-base/build/src/export/SimpleSpanProcessor.js", line 78, in 35)
  File " at MultiSpanProcessor.shutdown (/usr/src/app/node_modules/@opentelemetry/sdk-trace-base/build/src/MultiSpanProcessor.js", line 57, in 41)
  File " at NodeTracerProvider.shutdown (/usr/src/app/node_modules/@opentelemetry/sdk-trace-base/build/src/BasicTracerProvider.js", line 143, in 41)

OpenTelemetry Setup Code

const sdkNode = require('@opentelemetry/sdk-node')
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node')
const { SimpleSpanProcessor, ConsoleSpanExporter, AlwaysOnSampler } = require('@opentelemetry/sdk-trace-base')
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-grpc')

const traceExporter = new OTLPTraceExporter({
  url: process.env.OTEL_COLLECTOR_URL,
})
let spanProcessors = [
  new SimpleSpanProcessor(traceExporter),
]

if (process.env.DEBUG_TELEMETRY) {
  spanProcessors = [...spanProcessors, new SimpleSpanProcessor(new ConsoleSpanExporter())]

  const { diag, DiagConsoleLogger, DiagLogLevel } = require('@opentelemetry/api')
  diag.setLogger(new DiagConsoleLogger(), DiagLogLevel.DEBUG)
}

// configure the SDK to export telemetry data to the console
// enable all auto-instrumentations from the meta package
const sdk = new sdkNode.NodeSDK({
  traceExporter,
  spanProcessors,
  instrumentations: [
    getNodeAutoInstrumentations({
      '@opentelemetry/instrumentation-fs': {
        enabled: false,
      },
      '@opentelemetry/instrumentation-mongodb': {
        enabled: false,
      },
      '@opentelemetry/instrumentation-http': {
        ignoreIncomingRequestHook (req) {
          // Ignore OPTIONS requests
          return req.method === 'OPTIONS'
        },
      },
    }),
  ],
  sampler: new AlwaysOnSampler(),
})

// initialize the SDK and register with the OpenTelemetry API
// this enables the API to record telemetry
sdk.start()

module.exports = sdk

package.json

{
  "name": "test-app",
  "version": "1.0.0",
  "private": true,
  "scripts": {
    "start": "node app"
  },
  "dependencies": {
    "@opentelemetry/api": "1.8.0",
    "@opentelemetry/auto-instrumentations-node": "0.44.0",
    "@opentelemetry/exporter-trace-otlp-grpc": "0.50.0",
    "@opentelemetry/sdk-node": "0.50.0",
    "@opentelemetry/sdk-trace-base": "1.23.0",
    "express": "^4.15.2",
  }
}

Relevant log output

No response

@krim krim added bug Something isn't working triage labels Apr 8, 2024
@pichlermarc pichlermarc added priority:p1 Bugs which cause problems in end-user applications such as crashes, data inconsistencies, etc pkg:exporter-trace-otlp-grpc pkg:exporter-metrics-otlp-grpc pkg:exporter-logs-otlp-grpc and removed triage labels Apr 8, 2024
@pichlermarc pichlermarc self-assigned this Apr 8, 2024
@pichlermarc
Copy link
Member

I'm looking into this.

@pichlermarc
Copy link
Member

That's my fault. The transport should call close() instead of shutdown() on the _client. The test I wrote for this case was entirely ineffective as I also managed to mock the wrong thing. 😞

Sorry about that. I'll open a PR momentarily.

@krim
Copy link
Author

krim commented Apr 8, 2024

Thanks a lot!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working pkg:exporter-logs-otlp-grpc pkg:exporter-metrics-otlp-grpc pkg:exporter-trace-otlp-grpc priority:p1 Bugs which cause problems in end-user applications such as crashes, data inconsistencies, etc
Projects
None yet
2 participants