Sentry upgrade from 7.118.0 to 8.26.0 leak memory #18

hiroshinishio · 2024-10-06T01:24:00Z

Is there an existing issue for this?

I have checked for existing issues https://github.com/getsentry/sentry-javascript/issues
I have reviewed the documentation https://docs.sentry.io/
I am using the latest SDK release https://github.com/getsentry/sentry-javascript/releases

How do you use Sentry?

Self-hosted/on-premise

Which SDK are you using?

@sentry/node

SDK Version

8.26.0

Framework Version

No response

Link to Sentry event

No response

Reproduction Example/SDK Setup

my sentry init in 7.118.0

    init({
        dsn: process.env.SENTRY_DSN,
        environment: process.env.NODE_ENV || "none",
        release: require("../../package.json").version,
        serverName: `${cluster.bot}-c=${cluster.id}-first=${cluster.first}-last=${cluster.last}` || "none",
        integrations: [
            new Integrations.Postgres({ module: require("pg") }),
            new Integrations.Modules(),
            new Integrations.FunctionToString(),
            new Integrations.LinkedErrors(),
            new Integrations.Console(),
            new Integrations.Http({ breadcrumbs: true, tracing: true }),
            rewriteFramesIntegration({ root: path.join(__dirname, "..") }),
        ],
        // Performance Monitoring
        tracesSampleRate: 1.0, //  Capture 100% of the transactions
    });

my sentry init in 8.6.0

    Sentry.init({
        dsn: process.env.SENTRY_DSN,
        environment: process.env.NODE_ENV || "none",
        release: require("../../package.json").version,
        serverName: `${cluster.bot}-c=${cluster.id}-first=${cluster.first}-last=${cluster.last}` || "none",
        integrations: [
            Sentry.modulesIntegration(),
            Sentry.functionToStringIntegration(),
            Sentry.linkedErrorsIntegration(),
            Sentry.consoleIntegration(),
            Sentry.httpIntegration({ breadcrumbs: true }),
            Sentry.rewriteFramesIntegration({ root: path.join(__dirname, "..") }),
            Sentry.onUnhandledRejectionIntegration(),
            Sentry.onUncaughtExceptionIntegration(),
            Sentry.redisIntegration(),
            Sentry.postgresIntegration(),
        ],
        // To avoid sending too much data to Sentry, we can reduce the sample rate of traces and profiles
        tracesSampleRate: 1.0,
        profilesSampleRate: 1.0,
    });

Steps to Reproduce

I'll try removing some of the integrations to see what's causing the problem.

Expected Result

A normal memory usage

Actual Result

anormal memory usage

hiroshinishio · 2024-10-06T01:24:02Z

Hey thanks for writing in!

We'll look into your issue next week as this week is Hackweek at Sentry (see getsentry#13421).

hiroshinishio · 2024-10-06T01:24:03Z

Hi, would you be able to provide a memory snapshot with the Node/v8 profiler so that we can look at what is holding the references causing the leak? Feel free to also shoot us an email or twitter dm if you don't want to publicly share it. Thanks!

gitauto-ai · 2024-10-06T01:24:03Z

Click the checkbox below to generate a PR!

Generate PR

@hiroshinishio, You have 2 requests left in this cycle which refreshes on 2024-11-04 02:56:25+00:00.
If you have any questions or concerns, please contact us at info@gitauto.ai.

gitauto-for-dev · 2024-10-06T01:24:03Z

Click the checkbox below to generate a PR!

Generate PR - gitauto-wes

@hiroshinishio, You have 3 requests left in this cycle which refreshes on 2024-11-05 22:45:17+00:00.
If you have any questions or concerns, please contact us at info@gitauto.ai.

hiroshinishio · 2024-10-06T01:24:05Z

I can't give you an snapshot because I have a lot of private information.

hiroshinishio · 2024-10-06T01:24:06Z

I made a snapshot, I'll examine it and show you

hiroshinishio · 2024-10-06T01:24:08Z

hiroshinishio · 2024-10-06T01:24:09Z

Set, Span and NonRecordingSpan are from Sentry

hiroshinishio · 2024-10-06T01:24:11Z

Yeah that looks like Sentry. Would you mind digging around a bit ant examine what holds the references to the spans?

hiroshinishio · 2024-10-06T01:24:12Z

hiroshinishio · 2024-10-06T01:24:14Z

hiroshinishio · 2024-10-06T01:24:15Z

Set, Span and NonRecordingSpan are from Sentry

This indicates that Sentry objects are retained in memory. It doesn't mean they are causing this retention though!
E.g. the expanded Span object has the distance (from root) of 15. The sentryRootSpan - distance of 12. It means something else is holding the reference to it, something that is closer to the root.
You might want to explore the bottom part of the memory profiler - the Retainers section

hiroshinishio · 2024-10-06T01:24:17Z

hiroshinishio · 2024-10-06T01:24:18Z

timerListMap[180000] - my understanding is that the app has started a timer for 180s (3min).
Any chance it's your app code rather than Sentry?

hiroshinishio · 2024-10-06T01:24:19Z

I don't think so. Before switching to sentry v8, someone warned me about the memory leak.

hiroshinishio · 2024-10-06T01:24:21Z

I have a few questions to narrow this down further. I am not ruling out that our SDK is causing the leak:

What backend framework are you using - if at all?
Are you manually creating spans somewhere?
Can you share any more code that is relevant to Sentry?
What Node.js version are you using? We have found that there are memory leaks in Node.js with regards to timers?
Can you double-check whether you have any setTimeouts or setIntervals?
One way we have found things may "leak" is if you create a span and don't end it. That way children get attached and they will be held indefinitely.

hiroshinishio · 2024-10-06T01:24:22Z

Hello,
I use sentry on a discord bot not a website
Node version: v20.17.0

hiroshinishio · 2024-10-06T01:24:24Z

Would you mind answering the questions I asked. It's important we have answers to them so we can rule out certain things.

hiroshinishio · 2024-10-06T01:24:25Z

I've already double-checked my code
When you talk to me about span, I think of HTML elements, but I don't do web.

hiroshinishio · 2024-10-06T01:24:28Z

I am talking about Sentry spans. Which you would start with Sentry.startSpan(). (https://docs.sentry.io/concepts/key-terms/tracing/distributed-tracing/#traces-transactions-and-spans)

Do you happen to have very long running requests in your process? Like server-sent-events, or streaming?

hiroshinishio · 2024-10-06T01:24:29Z

On my side I never call Sentry.startSpan()

hiroshinishio · 2024-10-06T01:24:31Z

Do you have any very long running requests / request handlers in your code?

hiroshinishio · 2024-10-06T01:24:32Z

I wouldn't say long requests, but I have a lot of requests per second, over 110req/s at times, and I receive a lot of messages via websocket.

hiroshinishio · 2024-10-06T01:24:34Z

here you can see that the memory leak arrived at the same time as my sentry update, and since then I've had the performance tab working.

hiroshinishio · 2024-10-06T01:24:35Z

I believe you saying this is correlated with the update. Can you share a bit more about your program architecture? Is discord opening a websocket request on your server or is your server making a websocket request to discord?

Also, are you initializing Sentry more than once per process?

hiroshinishio · 2024-10-06T01:24:37Z

From what I can tell from your profiler screenshots, something is creating a Span (a root span) that seemingly never finishes, and spans keep getting attached to that span and that ends up leaking memory. It would be nice to be able to figure out, what it is that is creating this root span.

hiroshinishio · 2024-10-06T01:24:38Z

I initialize sentry only once
On each cluster I open 16 websocket connections to discord (clients)

I'll try to remove the profillerRate line, and we'll see if it comes from that.

hiroshinishio · 2024-10-06T01:24:40Z

If you could somehow also try logging console.log(Sentry.getRootSpan()) somewhere in your websocket code, and share what is logged, that would be cool!

hiroshinishio · 2024-10-06T01:24:41Z

Sentry.getRootSpan() requires and argument

hiroshinishio · 2024-10-06T01:24:43Z

Sorry right. Can you try

const s = Sentry.getActiveSpan();
if (s) {
  console.log('root span', Sentry.getRootSpan(s));
}

hiroshinishio · 2024-10-06T01:24:44Z

On the active process i used eval()

NonRecordingSpan {
  _spanContext: {
    traceId: 'ba6488d048422cfba347c2a2b9b1eca5',
    spanId: '5df94fad8fd85299',
    traceFlags: 0,
    traceState: [TraceState]
  }
}

hiroshinishio · 2024-10-06T01:24:45Z

Thanks! We are struggling trying to understand what is creating this non-recording Span. Usually that shouldn't happen unless you set a traces sample rate of <1 or you manually continue a trace.

Can you try setting debug: true in your ˚Sentry.init()`? It will probably generate a lot of logs, but it may tell you what exactly is continued or not sampled and so on.

hiroshinishio · 2024-10-06T01:24:47Z

can I enable logs without restarting my process ?

hiroshinishio · 2024-10-06T01:24:48Z

I don't think so :/

hiroshinishio · 2024-10-06T01:24:50Z

Another thing that we just noticed: Have you properly followed the migration guide for how and when to call Sentry.init()? You may be calling Sentry.init() too late, which may create weird spans. https://docs.sentry.io/platforms/javascript/guides/node/migration/v7-to-v8/#updated-sdk-initialization

hiroshinishio · 2024-10-06T01:24:53Z

in my case I don't require sentry first

hiroshinishio · 2024-10-06T01:24:54Z

can you try doing so?

hiroshinishio · 2024-10-06T01:24:56Z

Is it normal for it to load integrations that I haven't put in?

hiroshinishio · 2024-10-06T01:24:57Z

yes, there is a set of integrations that are enabled by default, but don't worry, as long as you do not use (ie import/require) these packages these integrations are no-ops.

hiroshinishio · 2024-10-06T01:24:59Z

okay,

good now ?

NonRecordingSpan {
  _spanContext: {
    traceId: 'd6ce7aec75e1fbdac62a14ca1a5d3353',
    spanId: 'eea933c24e347f1d',
    traceFlags: 0,
    traceState: [TraceState]
  }
}

hiroshinishio · 2024-10-06T01:25:01Z

I can't make out from the screenshot what you did tbh. If you are still importing other things inside ./extras/sentry, that's wrong.

hiroshinishio · 2024-10-06T01:25:02Z

This corresponds to sentry

hiroshinishio · 2024-10-06T01:25:04Z

That setup looks good 👌 Can you share more of the debug logs? It would be good to see logs up until and including when you do things like send requests and database queries and similar. Thanks!

hiroshinishio · 2024-10-06T01:25:05Z

there was a query to my db but it doesn't seem to appear

hiroshinishio · 2024-10-06T01:25:07Z

Would you mind sharing the start of your application up to a certain point in text format? Thanks!

hiroshinishio · 2024-10-06T01:25:08Z

Do you have discord ?
It would be easier to talk and send logs

hiroshinishio · 2024-10-06T01:25:10Z

Yes! Feel free to join https://discord.com/invite/sentry and ping ``

hiroshinishio · 2024-10-06T01:25:11Z

After some back and fourth we have discovered that the memory leak in this issue happens due to the httpIntegration creating a NonRecordingSpan for a very long-lived websocket request. The span is never ended because the request basically never ends for the entire lifetime of the process and stuff (ie. child spans) keep getting attached to that span.

The workaround for now is to do the following:

Sentry.init({
  integrations: [
    Sentry.httpIntegration({
      ignoreOutgoingRequests(url, request) {
        return true;
      },
    }),
  ],
})

Thanks for the collaboration !!

Action items (varying degrees of possible):

Stop attaching stuff to spans
Create an upper bound for stuff being attached to spans

hiroshinishio · 2024-10-06T01:25:13Z

No problem 👍

gitauto-for-dev · 2024-10-06T01:55:50Z

Sorry, we have an error. Please try again.

Have feedback or need help?
Feel free to email info@gitauto.ai.

gitauto-for-dev · 2024-10-06T01:56:45Z

Sorry, we have an error. Please try again.

Have feedback or need help?
Feel free to email info@gitauto.ai.

gitauto-ai · 2024-10-14T00:03:26Z

Sorry, we have an error. Please try again.

Have feedback or need help?
Feel free to email info@gitauto.ai.

hiroshinishio added the Package: node label Oct 6, 2024

gitauto-ai bot added the gitauto label Oct 14, 2024

Sentry upgrade from 7.118.0 to 8.26.0 leak memory #18

Sentry upgrade from 7.118.0 to 8.26.0 leak memory #18

Comments

hiroshinishio commented Oct 6, 2024

Is there an existing issue for this?

How do you use Sentry?

Which SDK are you using?

SDK Version

Framework Version

Link to Sentry event

Reproduction Example/SDK Setup

Steps to Reproduce

Expected Result

Actual Result

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

gitauto-ai bot commented Oct 6, 2024

gitauto-for-dev bot commented Oct 6, 2024 • edited by hiroshinishio Loading

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

hiroshinishio commented Oct 6, 2024

gitauto-for-dev bot commented Oct 6, 2024 • edited Loading

gitauto-for-dev bot commented Oct 6, 2024 • edited Loading

gitauto-ai bot commented Oct 14, 2024 • edited Loading

gitauto-for-dev bot commented Oct 6, 2024 •

edited by hiroshinishio

Loading

gitauto-for-dev bot commented Oct 6, 2024 •

edited

Loading

gitauto-for-dev bot commented Oct 6, 2024 •

edited

Loading

gitauto-ai bot commented Oct 14, 2024 •

edited

Loading