@grpc/grpc-js has memory leak issue with latest version

[alanzhao]

Problem description

We use spanner node js for our api, and the latest grpc-js 1.8.14 has memory leak issues, same as 1.8.13.

Reproduction steps

We use spanner node library for our api. We recently upgraded to node 18. so in package-lock json, grpc-js gets upgraded to 1.18.14. Before upgrade, we were not having the memory leak issue. After upgrade, we have a slow memory leak, manifested in 4-7 days. And our pods restarted due to low memory.

Our stop gap solution is to downgrade @grpc/grpc-js to 1.6.2 and that seems to fix the issue.
Thanks for looking into this. Much appreciated.

Environment

• x86_64 Linux
• 18.15
• nvm
• @grpc/grpc-js@1.8.14
• @google-cloud/spanner@5.18.0

Additional context

We have about 100 pods in prod, and all of them exhibit the symptom after the library gets updated.

[murgatroid99]

This isn't really enough information to find the problem. Can you provide code that demonstrates this memory leak, preferably without needing to access an external service? Can you provide heap dumps from at least two different points in a process's lifetime that show the memory growth?

Please also check whether you are creating multiple separate client objects. If you are, either change your code to not do that or make sure you are closing them when you are done with them. See #2436, #2402, and #2003 for previous examples of this.

[alanzhao]

async function getValue({ userId }) {
  const sql = `
    SELECT *
    FRROM ${table}
    WHERE userId = @userId
  `;

  const query = {
    sql,
    params: { userId },
    json: true,
  };

  const [[total]] = await clients.spanner.clientA.run(query);
  return total;
}

module.exports = getValue;

Here is a sample query that we use. We do have a wrapper around spanner and have 3 clients: clients.spanner.clientA, clients.spanner.clientB, and clients.spanner.clientC.

function allClose() {
  const closePromises = [];

  if (clients.spanner) {
    if (clients.spanner.clientA) {
      closePromises.push(clients.spanner.clientA.close());
    }

    if (clients.spanner.clientB) {
      closePromises.push(clients.spanner.clientB.close());
    }

    if (clients.spanner.clientC) {
      closePromises.push(clients.spanner.clientC.close());
    }
  }

which is used in the graphql server graceful shutdown here:

return startApp(app, ready, () => allClose());

The reason why we had 3 clients to spanner is because those 3 clients actually talk to 3 separate spanner databases. Right now, we do not use 2 of them, I can try to remove it and upgrade grpc back to latest version to see if that'll fix the issue. Thanks for the tips.

[murgatroid99]

Just three clients shouldn't be an issue. The problem I was talking about comes from creating a large number of clients.

[weyert]

Yes, I am also experiencing memory leaks with a library that uses this one. Newer versions of google-gax seem to cause out of memory dumps in Node.js v18 in GKE. Downgrading to 3.5.2 improves things. They have this library pinned to ~1.7.0. So maybe it's related to change sinc v1.8?

[murgatroid99]

Do you know how quickly you were leaking memory, in terms of amount of memory leaked per request made or per period of time? An order of magnitude estimate would be enough to help narrow this down.

[murgatroid99]

Update: we have been unable to reproduce a memory leak with our own tests. Please provide a complete reproduction that leaks memory so that we can investigate this further.

[kruegernet]

I also have this and will try to provide a reproduction when possible/time allows.

I'm coming from the last few years in Go, so IOC/DI frameworks in node are new to me and I believed originally my problem originated there, with unintentionally multiple clients. I'm still open to the notion it could be there in inversify misuse, but I'm more inclined to believe the issue is with grpc-js as with @alanzhao.

[SophiaH67]

Hi,

I have also found there to be memory leaks originating from this package(also saw some mentions in google-gax at some places in the memory debugger).

From what I can tell, this seems to be an issue with certain timeouts in Detached Node / nghttp2_memory entries:

With the following retainers list:

This snapshot is using comparison mode in devtools with 1 HTTP request being made between the 2 snapshots, which in our app does do multiple grpc calls.

I personally could not create a minimal reproduction app, even with the knowledge of this existing codebase. I still felt like this would be useful to share, for reasons:

• Someone else who might end up at this issue, knowing now it might be an issue with a different package often used together with this one.
• My memory leak dump might be useful somehow, since the retainers list is only files from the @grpc/grpc-js package.
• Letting OP know you're not the only one having issues with it!

Thank you for your time,

Sophia~

[murgatroid99]

@SophiaH67 In order to clearly demonstrate a significant memory leak, your test would need to do the following:

1. Run your test in a loop, to show that memory usage keeps growing.
2. Ensure that garbage collection runs, to make sure this is a real leak, and not a temporary spike in memory usage. You can do this by running Node with the --expose-gc flag and calling global.gc() before taking a snapshot.

If you do this and you observe similar memory retention in a number proportional to the number, then please share your test code and I can investigate.

[SophiaH67]

@murgatroid99

Thanks for your time! I should've prefaced this by saying this is my first time debugging memory leaks, so my bad if I miss anything. From what I could tell, the record snapshot button would automatically do garbage collection, but I cannot find a source for that so don't know where I learnt that.

After rerunning my tests in our production app with manual gc() calls through devtools before snapshotting, it does still happen.

In a small reproduction setup however, I could not find any problem with this library. Which leads me to think it's probably something in either a different library, or our own code!

[murgatroid99]

Have you also confirmed that the memory keeps growing with multiple successive calls? gRPC creates some resources on the first request that it reuses in later requests, so seeing more memory after the first request than before only confirms that, it doesn't show that anything is wrong.

[SophiaH67]

I have found the issue our team was running into finally!

We were using nestjs's official terminus package for health checks, which created a new channel for every health check attempt.

After doing some digging, it seems that these channels are not able to be garbage collected by NodeJS, due to several self-references(in timeouts and other stuff). This by itself could also be considered a bug, but I'm not too familiar with the garbage collector to solve this.

I did make a PR in terminus to cache the channels it opens, which I built and deployed in our app.

PR: nestjs/terminus#2329

Results:

As you can see here, this has totally solved our issue, and explains why I couldn't reproduce it with a minimal setup!

Still thanks to @murgatroid99 for your time and explaining garbage collector stuff to me ❤️