Sentry upgrade from 7.118.0 to 8.26.0 leak memory

[flav-code]

Is there an existing issue for this?

• I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
• I have reviewed the documentation https://docs.sentry.io/
• I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Self-hosted/on-premise

Which SDK are you using?

@sentry/node

SDK Version

8.26.0

Framework Version

No response

Link to Sentry event

No response

Reproduction Example/SDK Setup

my sentry init in 7.118.0

    init({
        dsn: process.env.SENTRY_DSN,
        environment: process.env.NODE_ENV || "none",
        release: require("../../package.json").version,
        serverName: `${cluster.bot}-c=${cluster.id}-first=${cluster.first}-last=${cluster.last}` || "none",
        integrations: [
            new Integrations.Postgres({ module: require("pg") }),
            new Integrations.Modules(),
            new Integrations.FunctionToString(),
            new Integrations.LinkedErrors(),
            new Integrations.Console(),
            new Integrations.Http({ breadcrumbs: true, tracing: true }),
            rewriteFramesIntegration({ root: path.join(__dirname, "..") }),
        ],
        // Performance Monitoring
        tracesSampleRate: 1.0, //  Capture 100% of the transactions
    });

my sentry init in 8.6.0

    Sentry.init({
        dsn: process.env.SENTRY_DSN,
        environment: process.env.NODE_ENV || "none",
        release: require("../../package.json").version,
        serverName: `${cluster.bot}-c=${cluster.id}-first=${cluster.first}-last=${cluster.last}` || "none",
        integrations: [
            Sentry.modulesIntegration(),
            Sentry.functionToStringIntegration(),
            Sentry.linkedErrorsIntegration(),
            Sentry.consoleIntegration(),
            Sentry.httpIntegration({ breadcrumbs: true }),
            Sentry.rewriteFramesIntegration({ root: path.join(__dirname, "..") }),
            Sentry.onUnhandledRejectionIntegration(),
            Sentry.onUncaughtExceptionIntegration(),
            Sentry.redisIntegration(),
            Sentry.postgresIntegration(),
        ],
        // To avoid sending too much data to Sentry, we can reduce the sample rate of traces and profiles
        tracesSampleRate: 1.0,
        profilesSampleRate: 1.0,
    });

Steps to Reproduce

I'll try removing some of the integrations to see what's causing the problem.

Expected Result

A normal memory usage

Actual Result

anormal memory usage

[Lms24]

Hey @flav-code thanks for writing in!

We'll look into your issue next week as this week is Hackweek at Sentry (see #13421).

[lforst]

Hi, @flav-code would you be able to provide a memory snapshot with the Node/v8 profiler so that we can look at what is holding the references causing the leak? Feel free to also shoot us an email or twitter dm if you don't want to publicly share it. Thanks!

[flav-code]

I can't give you an snapshot because I have a lot of private information.

[flav-code]

I made a snapshot, I'll examine it and show you

[flav-code]

[flav-code]

Set, Span and NonRecordingSpan are from Sentry

[lforst]

Yeah that looks like Sentry. Would you mind digging around a bit ant examine what holds the references to the spans?

[flav-code]

[flav-code]

[amakhrov]

Set, Span and NonRecordingSpan are from Sentry

This indicates that Sentry objects are retained in memory. It doesn't mean they are causing this retention though!
E.g. the expanded Span object has the distance (from root) of 15. The sentryRootSpan - distance of 12. It means something else is holding the reference to it, something that is closer to the root.
You might want to explore the bottom part of the memory profiler - the Retainers section

[flav-code]

[amakhrov]

timerListMap[180000] - my understanding is that the app has started a timer for 180s (3min).
Any chance it's your app code rather than Sentry?

[flav-code]

I don't think so. Before switching to sentry v8, someone warned me about the memory leak.

[lforst]

I have a few questions to narrow this down further. I am not ruling out that our SDK is causing the leak:

• What backend framework are you using - if at all?
• Are you manually creating spans somewhere?
• Can you share any more code that is relevant to Sentry?
• What Node.js version are you using? We have found that there are memory leaks in Node.js with regards to timers?
• Can you double-check whether you have any setTimeouts or setIntervals?
• One way we have found things may "leak" is if you create a span and don't end it. That way children get attached and they will be held indefinitely.

[lforst]

That setup looks good 👌 Can you share more of the debug logs? It would be good to see logs up until and including when you do things like send requests and database queries and similar. Thanks!

[flav-code]

there was a query to my db but it doesn't seem to appear

[lforst]

Would you mind sharing the start of your application up to a certain point in text format? Thanks!

[flav-code]

Do you have discord ?
It would be easier to talk and send logs

[lforst]

Yes! Feel free to join https://discord.com/invite/sentry and ping @lforst

[lforst]

After some back and fourth we have discovered that the memory leak in this issue happens due to the httpIntegration creating a NonRecordingSpan for a very long-lived websocket request. The span is never ended because the request basically never ends for the entire lifetime of the process and stuff (ie. child spans) keep getting attached to that span.

The workaround for now is to do the following:

Sentry.init({
  integrations: [
    Sentry.httpIntegration({
      ignoreOutgoingRequests(url, request) {
        return true;
      },
    }),
  ],
})

Thanks for the collaboration @flav-code!!

---

Action items (varying degrees of possible):

• Stop attaching stuff to spans
• Create an upper bound for stuff being attached to spans

[flav-code]

No problem 👍

[mydea]

Hey, I am closing this because I believe we found the issue and provided a workaround for the time being. Please re-open if you still encounter the problem - thank you!

[flav-code]

yes, it's not as bad as before

[lforst]

Note that this is one of the cases that will be solved once we switch to span streaming. So in like 5 years.

[flav-code]

5 years ???

[lforst]

I should have added a sarcasm indicator 😬 It's coming but it's taking longer than we (I) would like. Hopefully sooner than 5 years.