Http request vs https request - Memory Issue

[s888]

I am using request module in my app, so I am testing the memory usage for multiple requests.

I am making request to a server an interval of around 100 ms(for http) and 500 ms (for https)

As the http request increases the memory remains constant 78 mb.
But in case of https request the memory keeps on growing rapidly and it reaches around 140 mb (see the screenshot)

Here's the code:

---> Request to http

    var request = require('request');
    var req_counter = 0, res_counter = 0;
    function makeRequest() {
             req_counter++
             request('http://www.google.com', function (error, response, body) {
             res_counter++;
             console.log('res_counter',res_counter,req_counter);
      });
    }
    setInterval(makeRequest, 100);

---> Request to https

    var request = require('request');
    var req_counter = 0, res_counter = 0;
    function makeRequest() {
             req_counter++
             request('https://www.google.com', function (error, response, body) {
             res_counter++;
             console.log('res_counter',res_counter,req_counter);
      });
    }
    setInterval(makeRequest, 500);

[it-can]

I have the same issue... is this a bug?

[pajtai]

👍 I'm have a memory leak issue in one of my apps, and I have not been able to pinpoint the source, but it'd great to be able to eliminate this npm.

[pyhedgehog]

It would be interesting to run this test directly on http/https modules. If you've got same issue this can be bug of nodejs itself.

[maxrabin]

👍 we just ran into this. Anybody have any insight?

[flamepygmy]

I'm facing similar issues when trying to download large files (500MB+) from a HTTPS URL. Memory usage very quickly jumps to 40MB+ and keeps hiking up as the download progresses. Even if I manually run gc() (for testing purposes) at every data event, the memory keeps creeping up eventually managing to get the node killed. When I view the heap dumps, there are a ton of 16kB uint8buffers around (they grow in number as the download progresses).

Some potentiall relevant details: OSX El Capitan, Node 5.3.0, URL is not a direct link but causes a 302 redirect to the final URL.

If I download the very same file with https module, the memory usage is around 12MB and stays there.

I have the same behaviour even if I just make the request and don't write the data anywhere (no 'data' event, no pipe, ...). It is most likely something that's inside the request module and how it handles the data.

Workaround for me (for now): implement manually the HTTP redirects and then just use the https module instead as it's not leaking.

[pierre-b]

Same problem for me, and it crashes my server because my app do thousands of POST per minutes...
Node 4.4.0.
Any idea please?

[sentanos]

I have an app that has to make upwards of 15000 requests in a short amount of time but it kept going up to well over 1 gb memory even though I was doing all I could to stop anything from leaking. I've been debugging it for days, inspecting heap dumps and running gc manually but nothing seemed to fix it. Finally changing from https to http keeps the memory stable under 300/400 mb; however, https is preferred and I hope this can be fixed as soon as possible.

[pierre-b]

Try to use an HTTPS agent, with keepAlive and maxSockets infinity, that helps a lot!

[DuBistKomisch]

After trying out various node versions, this seems to be a bug with the https (and/or tls I guess) module in node 0.12 (and 0.10) itself. I can easily reproduce the process's memory skyrocketing into the GBs until the process is killed, simply by making a request every second with a big options object (e.g. passing a ~200 element list of certs as strings with the ca option and agent: false). This happens with both https and request.

Can't reproduce at all with node 4 (or iojs 1-3) though, the memory just caps out as it should, like when using http or request with an 'http://...' url. If you search the node repo issues for "https memory leak" you'll find heaps of issues fixed in 1.x but not backported to 0.12.x.

People above have reported issues with node 4 and 5 though, so maybe there's a second issue here.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.