Memory usage is insanely high

[marvin-j97]

Problem

Process starts at ~24 MB.

await sharp("in.jpg")
  .resize({
    width: 1920,
    height: 1080,
  })
  .toFormat("jpg", {
    mozjpeg: true,
  })
  .toBuffer();

Process is now at ~120 MB. Memory does not decrease (no GC).

Running the above code in a loop, memory shoots up to ~620 MB, and does not decrease either.

Comparing with imagemagick

With the following code, using imagemagick, the memory stays constant at ~24 MB.
Tested with imagemagick 6.9.11.

const command = `convert in.jpg -resize 1920x1080^ -gravity center out.jpg`;

for (;;) {
  execSync(command);
}

System

Node 20
Sharp 0.32.6
Ubuntu 22.04 LTS

[lovell]

Please ensure you've read and understood everything in #955 relating to memory allocators on Linux.

In addition, it looks like you're trying to compare many short-lived processes with one long-running process, which is Äpfel mit Birnen vergleichen.

[marvin-j97]

Please ensure you've read and understood everything in #955 relating to memory allocators on Linux.

In addition, it looks like you're trying to compare many short-lived processes with one long-running process, which is Äpfel mit Birnen vergleichen.

This makes me wonder a bit about the Sharp internals. I haven't read too deep into the source code. How is libvips called exactly? If I spawned libvips as a child process, the process can be cleaned up by the OS, so my Node process won't get GC problems. I rolled a custom Rust binary now for cropping and resizing images, because it's much less overhead than both imagemagick & sharp/libvips - not too mention that deploying to serverless with Sharp currently is hard to pull off correctly, especially when using an alternative package manager. I had quite some problems with ESM and bundling Sharp and its dependencies; I'll see if I can document them some day. If there's one thing I always wished for Sharp to have is a way to skip the installation, give it a path to a libvips binary and let it call as child process with the (IMO) perfect Sharp API, similar to fluent-ffmpeg works.

I guess this is related to #3750.

[lovell]

sharp integrates with libvips via its C++ binding shared library. As you learnt when you read #955, memory management is controlled by the underlying allocator, which is usually the one provided by the OS (e.g. Ubuntu uses glibc).

The vips command line tool is intentionally simple, very much unlike the *magick and ffmpeg examples you've provided, and is therefore unsuitable for the more complex image processing pipelines that sharp allows. The approach we recommend to anyone integrating with libvips is to use one of its language bindings for anything other than very simple tasks.