Sending large amount of data to ServerResponse causes RSS and External memory leak

[andrewnester]

• Node.js Version: 10.17.0, 8.6.12
• OS: Ubuntu 16.04
• Scope (install, code, runtime, meta, other?): Runtime
• Module (and version) (if relevant): Stream. Most probably related to http_outgoing.js stream_writable.js

Hello!

When sending lots of data to response( for example downloading big files) using Buffer there is significant increase in RSS and External memory usage.

As you can see in my example I stop to send data when stream is paused and continue after resume call so it shouldn’t be buffered in response stream.

Even though resume/pause called properly RSS/external memory is growing. Heap seems more or less stable.

Here are results of the calls to process.memoryUsage after starting downloading.

{ rss: 29802496,
  heapTotal: 7684096,
  heapUsed: 5211920,
  external: 8224 }

// Download triggered

{ rss: 33968128,
  heapTotal: 10305536,
  heapUsed: 5312576,
  external: 26222624 }
{ rss: 38477824,
  heapTotal: 10829824,
  heapUsed: 5743944,
  external: 56631328 }
{ rss: 60731392,
  heapTotal: 8208384,
  heapUsed: 5257744,
  external: 92282912 }

Debugging using Chrome inspector made me suspect flow which is below.

1. When we try to write lots of data to response the response connection will be corked and uncorked on next tick here
   https://github.com/nodejs/node/blob/v8.x/lib/_http_outgoing.js#L652

2. write_ method will return false to initiate backpressure and pause will be called:
   https://github.com/nodejs/node/blob/v8.x/lib/internal/streams/legacy.js#L17

3. On the next tick connection will be uncorked and clearBuffer will be called
   https://github.com/nodejs/node/blob/v8.x/lib/_stream_writable.js#L312

4. It results in calling doWrite here
   https://github.com/nodejs/node/blob/v8.x/lib/_stream_writable.js#L500

5. Then stream._writev will be called and result of this call will be ignored
   https://github.com/nodejs/node/blob/v8.x/lib/_stream_writable.js#L394

6. Then as a result afterWrite will be called which causes emitting of drain event and resuming stream even though we don't know if data on Step 5 was buffered or not.
   https://github.com/nodejs/node/blob/v8.x/lib/_stream_writable.js#L461
   https://github.com/nodejs/node/blob/v8.x/lib/_stream_writable.js#L473

My suspect is that memory leaking happens on Step 5 but unfortunately I am not sure as I can't get V8 memory profile to see what is leaking in RSS/External memory.

Below is simple Node JS code which emulates downloading of large file with chunk of 1MBs sent to stream using Buffer as data.

const http = require("http");
const PORT = 8080;
const Stream = require("stream");

const requestHandler = (req, res) => {
    let mystream = new Stream();
    let paused = false;

    mystream.resume = () => {
        paused = false;
    };
    mystream.pause = () => {
        paused = true;
    };
    mystream.readable = true;

    mystream.once("data", function(data) {
        if (res.headersSent) return;
        res.writeHead(200, {
            "Content-Disposition": "attachment; filename*=utf-8''test",
            "Content-Type": "application/zip"
        });
        res.write(data);
        mystream.pipe(res);
    });
	
    setInterval(() => {
        if (!paused) {
            mystream.emit("data", new Buffer(1024 * 1024));
        }
    }, 10);
}

const server = http.createServer(requestHandler);

server.listen(PORT, (err) => {
	if (err) {
		console.log("Error occurred", err) ;
	}
	console.log(`Server is listening on ${PORT}`);
	
	setInterval(() => {
		console.log(process.memoryUsage());
	}, 5000);
})

[andrewnester]

Seems very similar to this issue even though it’s closed

#6078

[andrewnester]

I did a bit more investigation and apparently the issue is caused by allocating Buffer.

In this particular case buffers are allocated a lot, V8 allocates them on external memory, frees the memory after Buffer is not used anymore but GC does not de-allocate the issue. So external memory is actually free just not de-allocated.

As a workaround to force memory de-allocation we can call gc() directly

[puzpuzpuz]

@andrewnester there is a chance that this issue may be related with glibc memory fragmentation? See #8871 and #33790

[andrewnester]

@puzpuzpuz #33790 doesn't look like relevant but #8871 might be but I don't know how to prove if it's relevant for us or not

[puzpuzpuz]

@andrewnester both of these issues are related with glibc memory fragmentation. To diagnose your issue you could try to switch to jemalloc (see #8871 (comment) for more details) and if that fixes your issue, it's most likely has something to do with memory fragmentation.

[gireeshpunathil]

pls evaluate the discussion in #31641 too - in that case, there is no real leak, instead incorrect accounting of memory against the owning process. (I am not claiming this one IS that, just a speculation)

[rudolf]

@andrewnester Using your script I can't reproduce this on v14.18.1 or v16.13.0. On both OSX and Ubuntu the external memory stays stable.

[bnoordhuis]

OP didn't follow up, @rudolf wasn't able to reproduce, and there's been no movement since late 2021 so I'll go ahead and close this out.