Insights: auto-Instrument BullMQ for Queue Insights

[bcoe]

Problem Statement

Similar to Celery for Python, BullMQ seems like it would be a good choice to auto-instrument for queue insights, see: #10148, along with conversations on Twitter:

Solution Brainstorm

We should model an implementation after the implementation for Celery.

[AbhiPrasad]

There is OTEL instrumentation we can leverage here: https://github.com/appsignal/opentelemetry-instrumentation-bullmq

[AbhiPrasad]

In the time being, we can probably write some docs that uses the otel instrumentation manually with bullmq. So users would have to install opentelemetry-instrumentation-bullmq themselves and then call span.setAttribute themselves.

@bcoe for an intermediate solution what do you think?

[iacobus]

Hi! Our team is also interested in instrumenting our BullMQ workloads running inside of NestJS. I know close to nothing about OTEL or about how Sentry works, but I've been playing with the OTEL instrumentation package that you referenced, and got something, but not much.

What I've done is basically trying to integrate the OTEL instrumentation with @sentry/node by looking at how other integrations work. The following snippet defines a BullMQ integration based off the PG tracing code:

import { BullMQInstrumentation } from '@appsignal/opentelemetry-instrumentation-bullmq'
import { defineIntegration } from '@sentry/core'
import { generateInstrumentOnce } from '@sentry/node'
import type { IntegrationFn } from '@sentry/types'

const INTEGRATION_NAME = 'BullMQ'

export const instrumentBullMQ = generateInstrumentOnce(
  INTEGRATION_NAME,
  () =>
    new BullMQInstrumentation({}),
)

const _bullmqIntegration = (() => {
  return {
    name: INTEGRATION_NAME,
    setupOnce() {
      instrumentBullMQ()
    },
  }
}) satisfies IntegrationFn

/**
 * BullMQ integration
 *
 * Capture tracing data for BullMQ.
 */
export const bullmqIntegration = defineIntegration(_bullmqIntegration)

With this snippet I can add the bullmqIntegration to my integrations array in Sentry.init (which also contains nodeProfilingIntegration()), so that when I trigger a Bull job, I see a transaction of type message under the Performance section of Sentry, with these relevant attributes:

messaging.operation: process,
messaging.system: bullmq,
otel.kind: CONSUMER,
sentry.op: message,
sentry.origin: manual,

However, this transaction is empty. Its duration is just a few milliseconds (shorter than the job duration), and it contains no spans at all, even though there are PG queries inside (PG queries are correctly reported in our regular application tracing). We also see no reference to the producing side of the job, nor we see anything under the Queues section of Sentry.

Not sure if you were able to get started on those docs you mentioned, but if you have some tips that could unblock us, we'd appreciate it 🙏

[andreiborza]

Hello, thanks for writing in. We are on company-wide hackweek and thus on limited support this week. We'll take a look at this next week.

[iacobus]

Oh cool, have fun and thanks!