MaxListenersExceededWarning: Possible EventEmitter memory leak detected. error #1027

apatora · 2019-09-11T17:34:24Z

Problem description

I am using the package @google-cloud/bigtable which relies on this one and when I ran npm install today it updated @grpc/grpc-js from 0.5.2 to 0.5.3. I am getting the following error :

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added. Use emitter.setMaxListeners() to increase limit
    at _addListener (events.js:256:17)
    at TLSSocket.addListener (events.js:272:10)
    at TLSSocket.Readable.on (_stream_readable.js:826:35)
    at Http2CallStream.attachHttp2Stream (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/call-stream.js:275:35)
    at Http2SubChannel.startCallStream (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/subchannel.js:110:20)
    at finalMetadata.then.metadataValue (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/channel.js:205:28)
    at process._tickCallback (internal/process/next_tick.js:68:7)

Reproduction steps

Run npm i @google-cloud/bigtable in a project. Then try to execute for example :

const Bigtable = require('@google-cloud/bigtable')

for (let j = 0; j < 100; j++) {
    Bigtable.instance('INSTANCE').table('TABLE').insert([{ key: 'key', data: { data: { some: 'data' } } }]
}

Environment

I am running the code in the following docker image : node:10.16.0

The text was updated successfully, but these errors were encountered:

murgatroid99 · 2019-09-11T18:27:39Z

I think this is the result of #1021. I'll need to figure out a better way to do that.

ddehghan · 2019-09-16T01:38:27Z

also happens on node:12.10.0-alpine

 MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 close listeners added to [TLSSocket]. Use emitter.setMaxListeners() to increase limit
    at _addListener (events.js:267:17)
    at TLSSocket.addListener (events.js:283:10)
    at TLSSocket.Readable.on (_stream_readable.js:863:35)
    at Http2CallStream.attachHttp2Stream (/var/www/app/node_modules/@grpc/grpc-js/build/src/call-stream.js:275:35)
    at Http2SubChannel.startCallStream (/var/www/app/node_modules/@grpc/grpc-js/build/src/subchannel.js:110:20)
    at /var/www/app/node_modules/@grpc/grpc-js/build/src/channel.js:205:28

ddehghan · 2019-09-16T23:59:15Z

for now i rolled back to ver "0.5.2" and it fixed the problem.

yatingoswami · 2019-09-17T05:02:15Z

Also facing the same issue on 12.4.0-stretch
Getting pod restarts every 4 hours.

Cant rollback version as its part of google-gax which is nested inside @google-cloud/pubsub.
Is there a proper solution for this

apatora · 2019-09-18T14:47:56Z

@yatingoswami You can solve the issue for now by putting :

"resolutions": {
  "@grpc/grpc-js": "0.5.2"
}

in your package.json file. Then you execute npx npm-force-resolutions and npm i.

It seems that the issue is affecting a lot of packages. Maybe it would be better to rollback #1021 until it is fully resolved.

nicolasnoble · 2019-09-18T19:52:32Z

How is a warning impeding production usage exactly?

yatingoswami · 2019-09-18T20:14:09Z

"resolutions": { "@grpc/grpc-js": "0.5.2" }
This solved it for me.
Thanks

merlinnot · 2019-09-18T20:26:57Z

How is a warning impeding production usage exactly?

There's a massive memory leak. This warning is accurate. I observed consumption of ~1GB/h/instance across 200+ Google Cloud Functions (1k+ instances), causing so many OOMs that the retry mechanism gave up.

This is a severe issue and I highly recommend downgrading to 0.5.2, which does not have this leak.

murgatroid99 · 2019-09-18T20:50:25Z

In the past we have had various reports of this specific warning in other code paths, and in general it has not been indicative of memory leaks, but simply that there are more than 11 calls open simultaneously in a single channel. Because the warning was the only thing reported here, I assumed that that was the only user-facing symptom. Now that I know that there is a memory leak, I can investigate that and deploy a fix.

merlinnot · 2019-09-18T20:56:49Z

Thanks, I thought it's being worked on and was just patiently waiting :)

It should be very easy to reproduce, but just in case you can't, I can provide a repro (although there are some code samples in this thread already).

murgatroid99 · 2019-09-20T17:51:10Z

grpc-js version 0.5.4 is now out with the fix for this.

jrthib · 2019-09-20T18:06:46Z

@murgatroid99 looks like the build is failing

murgatroid99 · 2019-09-20T18:11:29Z

If you're referring to the version bump build, that's just because I merged it quickly.

merlinnot · 2019-09-20T18:54:25Z

I can confirm that there's no memory leak anymore.

kirillgroshkov · 2019-09-23T13:48:20Z

We still have these errors in production (screenshot from today attached). We're using Datastore node library.

yarn list @grpc/grpc-js                                                            01:37 Dur
yarn list v1.17.3
warning Filtering by arguments is deprecated. Please use the pattern option instead.
└─ @grpc/grpc-js@0.5.4
✨  Done in 0.74s.

murgatroid99 · 2019-09-23T16:36:03Z

What is the call stack for those errors? I removed the line of code that caused the original errors; it doesn't exist in version 0.5.4.

jakeleventhal · 2019-09-29T03:03:42Z

I am on 5.2 as per recommended and am seeing a MASSIVE memory leak from something different that may be related. googleapis/nodejs-firestore#768

apatora · 2019-10-03T22:37:34Z

I tried the last version of this module and got the following error after a while :

Error: No connection established
    at Http2CallStream.call.on (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/call.js:68:41)
    at Http2CallStream.emit (events.js:203:15)
    at Http2CallStream.EventEmitter.emit (domain.js:448:20)
    at process.nextTick (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22)
    at process._tickCallback (internal/process/next_tick.js:61:11)

murgatroid99 · 2019-10-03T23:20:43Z

@apatora that looks like an entirely separate error. I suggest filing a separate issue with more details.

nekdev · 2019-10-06T12:13:30Z

I tried the last version of this module and got the following error after a while :

Error: No connection established
    at Http2CallStream.call.on (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/call.js:68:41)
    at Http2CallStream.emit (events.js:203:15)
    at Http2CallStream.EventEmitter.emit (domain.js:448:20)
    at process.nextTick (/usr/src/node_modules/@google-cloud/bigtable/node_modules/@grpc/grpc-js/build/src/call-stream.js:75:22)
    at process._tickCallback (internal/process/next_tick.js:61:11)

@apatora I'm facing the same error in Google Cloud Functions. Did you find any workaround?

apatora · 2019-10-06T12:44:20Z

@nekdev You should check #1064. For now the only way to circumvent this error is to force (in package.json) :

"resolutions": {
    "@grpc/grpc-js": "0.5.2",
    "google-gax": "1.6.2"
}

I am closing this issue as it seems the MaxListenersExceededWarning is gone in the newest versions of grpc-js.

nekdev · 2019-10-07T21:46:40Z

All problems solved after updating to:

    "firebase-admin": "^8.6.0",
    "firebase-functions": "^3.2.0",

swftvsn mentioned this issue Sep 16, 2019

Exception occurred in retry method that was not classified as transient firebase/firebase-functions#536

Closed

schmidt-sebastian mentioned this issue Sep 16, 2019

GRPC-JS: Possible EventEmitter memory leak detected googleapis/nodejs-firestore#661

Closed

mad-it mentioned this issue Sep 17, 2019

Issue with 1.4.0 release. googleapis/nodejs-kms#207

Closed

murgatroid99 mentioned this issue Sep 19, 2019

Hopefully fix memory leak with socket close listener #1034

Closed

merlinnot mentioned this issue Sep 19, 2019

MaxListenersExceededWarning: Possible EventEmitter memory leak detected googleapis/nodejs-pubsub#752

Closed

murgatroid99 mentioned this issue Sep 19, 2019

Revert "Add specific call error for TCP disconnection" #1036

Merged

callmehiphop mentioned this issue Sep 23, 2019

EventEmitter memory leak googleapis/nodejs-pubsub#757

Closed

jakeleventhal mentioned this issue Sep 29, 2019

HUGE memory leak problem googleapis/nodejs-firestore#768

Closed

apatora closed this as completed Oct 6, 2019

rvillane mentioned this issue Oct 7, 2019

Error: No connection established #1064

Closed

clytras mentioned this issue Oct 16, 2019

Firestore/triggers grpc-js No connection established errors firebase/firebase-functions#568

Closed

lock bot locked as resolved and limited conversation to collaborators Jan 5, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. error #1027

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. error #1027

apatora commented Sep 11, 2019

murgatroid99 commented Sep 11, 2019

ddehghan commented Sep 16, 2019 •

edited

Loading

ddehghan commented Sep 16, 2019

yatingoswami commented Sep 17, 2019

apatora commented Sep 18, 2019

nicolasnoble commented Sep 18, 2019

yatingoswami commented Sep 18, 2019 •

edited

Loading

merlinnot commented Sep 18, 2019

murgatroid99 commented Sep 18, 2019

merlinnot commented Sep 18, 2019

murgatroid99 commented Sep 20, 2019

jrthib commented Sep 20, 2019

murgatroid99 commented Sep 20, 2019

merlinnot commented Sep 20, 2019

kirillgroshkov commented Sep 23, 2019

murgatroid99 commented Sep 23, 2019

jakeleventhal commented Sep 29, 2019

apatora commented Oct 3, 2019

murgatroid99 commented Oct 3, 2019

nekdev commented Oct 6, 2019 •

edited

Loading

apatora commented Oct 6, 2019

nekdev commented Oct 7, 2019

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. error #1027

MaxListenersExceededWarning: Possible EventEmitter memory leak detected. error #1027

Comments

apatora commented Sep 11, 2019

Problem description

Reproduction steps

Environment

murgatroid99 commented Sep 11, 2019

ddehghan commented Sep 16, 2019 • edited Loading

ddehghan commented Sep 16, 2019

yatingoswami commented Sep 17, 2019

apatora commented Sep 18, 2019

nicolasnoble commented Sep 18, 2019

yatingoswami commented Sep 18, 2019 • edited Loading

merlinnot commented Sep 18, 2019

murgatroid99 commented Sep 18, 2019

merlinnot commented Sep 18, 2019

murgatroid99 commented Sep 20, 2019

jrthib commented Sep 20, 2019

murgatroid99 commented Sep 20, 2019

merlinnot commented Sep 20, 2019

kirillgroshkov commented Sep 23, 2019

murgatroid99 commented Sep 23, 2019

jakeleventhal commented Sep 29, 2019

apatora commented Oct 3, 2019

murgatroid99 commented Oct 3, 2019

nekdev commented Oct 6, 2019 • edited Loading

apatora commented Oct 6, 2019

nekdev commented Oct 7, 2019

ddehghan commented Sep 16, 2019 •

edited

Loading

yatingoswami commented Sep 18, 2019 •

edited

Loading

nekdev commented Oct 6, 2019 •

edited

Loading