Memory did not released back after process is complete

[ArkjuniorK]

Are you using the latest version? Is the version currently in use as reported by npm ls sharp the same as the latest version as reported by npm view sharp dist-tags.latest?

npm ls sharp
└── sharp@0.28.1 

npm view sharp dist-tags.latest
0.28.1

What are the steps to reproduce?

I'm using sharp to resize and convert an image to 1280 pixel and webp format.
I've done a research about this possible memory leak, I've tried to use sharp.concurrency(1) or use the LD_PRELOAD=/usr/lib/libjemalloc.so.2 but after each of image process is complete, memory was not released back.

Memory use before sharp is processing the image is about 90mb-110mb, but after processing 5-8 image, the memory increase to 280mb - 310mb and did not released back.

Basically that's how I use the sharp without use toBuffer() function since it would make memory consumption more bigger.

What is the expected behaviour?

Memory is released back after all the process is done.

Are you able to provide a minimal, standalone code sample, without other dependencies, that demonstrates this problem?

const sharp = require('sharp')

sharp.cache(false)
sharp.concurrency(1)

exports.upload_image = async (req, res) => {
   try {
      const file = req,files[0]
      
      await sharp('./' + file.path)
          .resize(1280)
          .webp({ quality: 30 })
          .toFile('./' + file.path + '.webp')
   }
   catch(err) {
      console.log(err)
   }
}

Are you able to provide a sample image that helps explain the problem?

What is the output of running npx envinfo --binaries --system?

On development machine

  System:
    OS: Linux 5.9 Manjaro Linux
    CPU: (8) x64 Intel(R) Core(TM) i7-7700HQ CPU @ 2.80GHz
    Memory: 243.14 MB / 7.66 GB
    Container: Yes
    Shell: 5.8 - /bin/zsh
  Binaries:
    Node: 14.16.1 - /usr/local/bin/node
    Yarn: 1.22.10 - /usr/bin/yarn
    npm: 6.14.13 - ~/Dev/estuary_surveyor_dev/node_modules/.bin/npm

On production machine

  System:
    OS: Linux 3.10 CentOS Linux 7 (Core)
    CPU: (3) x64 Intel(R) Xeon(R) Silver 4214 CPU @ 2.20GHz
    Memory: 719.18 MB / 3.00 GB
    Container: Yes
    Shell: 4.2.46 - /bin/bash
  Binaries:
    Node: 14.16.1 - /usr/local/bin/node
    npm: 6.14.12 - /usr/local/bin/npm

[lovell]

Please make sure you have read and understood everything in #955, including that RSS includes memory that has been freed but not yet returned to the OS.

If you still believe there is a problem, please can you provide a complete, standalone repo with all code and images that allows someone else to reproduce this consistently.

In addition, as you're using jemalloc, please can you set MALLOC_CONF in this repo to enable profiling to demonstrate what is consuming the memory. There are instructions in https://github.com/jemalloc/jemalloc/wiki/Use-Case%3A-Leak-Checking

[lovell]

@ArkjuniorK Were you able to make any progress with this?

[ArkjuniorK]

@lovell I was able to run the jemalloc.so.2 by passing it inside the pm2 ecosystem.config.js. But why sharp did not work with the default memory allocator in Linux distro such as manjaro and centos? Or it is actually issue with the pm2?

[lovell]

Glad you worked it out. There's more info at https://sharp.pixelplumbing.com/install#linux-memory-allocator

[ArkjuniorK]

@lovell On my code I used the sharp.concurrency(1) and jemalloc.so.2 as the memory allocator. Should I put the sharp.concurrency(1) on every file that I required the sharp module or it actually has nothing to do with it since I was used jemalloc.so.2 as memory allocator?

Here is the example

rotateImage.js

const sharp = require("sharp");

// sharp config
sharp.cache(false);
sharp.concurrency(1);

// sharp used here
function rotateImage() {
  ...
}

resizeImage.js

const sharp = require("sharp");

// sharp config
sharp.cache(false);
sharp.concurrency(1);

// sharp used here
function resizeImage() {
  ...
}

Thank you.

[lovell]

As of sharp v0.28.0, the use of jemalloc is automatically detected and the best concurrency is set for you, so perhaps try removing all the calls to sharp.concurrency.

[ArkjuniorK]

Thanks @lovell