Investigate memory usage with Image Optimization enabled

[timneutkens]

Verify canary release

• I verified that the issue exists in the latest Next.js canary release

Provide environment information

Latest version of Next.js.

Which area(s) of Next.js are affected? (leave empty if unsure)

Image optimization (next/image, next/legacy/image)

Link to the code that reproduces this issue or a replay of the bug

Reproduction was left out in reports in #49929

To Reproduce

Unfortunately we haven't received a reproduction of these reports so this needs a reproduction in order to investigate. We'll keep this issue open for the standard 30 days before closing it if there is no reproduction provided.

Describe the Bug

In #49929 some people reported high memory usage / some claim a memory leak when image optimization is enabled. Unfortunately we haven't been able to verify this as there is no reproduction provided for these, only monitoring tool screenshots.

We'll keep this issue open temporarily hoping that someone will provide a reproduction so that it can be investigated.

Expected Behavior

Unclear as of right now, as there is no reproduction yet.

Which browser are you using? (if relevant)

No response

How are you deploying your application? (if relevant)

No response

[AlphaBravoIndia]

Hey Tim,

I've been looking into this for our site that is deployed on Railway and realized that it may be purely driven by the fact that optimized images are stored in memory. If a site has a lot of images, I assume that the build up of optimized images could look like a memory leak? Our images aren't huge but we do have quite a few, say 10,000, and this is what our memory usage looks like over a few hours with optimized images enabled:

This builds up to like 7gb over a ~24 hour period which then forces us to restart the server.

A similar thing happens in this reproduction: https://github.com/AlphaBravoIndia/next-image-memory-leak

Whenever I refresh the page a bunch of times it generates more optimized images and memory usage spikes and stays higher than the previous point. This makes me think that it's purely the optimized images pushing memory usage up rather than a leak. If I stop refreshing the page, the memory usage stays consistent (at the new level).

Should optimized images be dumped from memory after a certain amount of time automatically or is this something that we need (or can) configure?

Cheers!

[miketdonahue]

(Digital Ocean App Platform - Next.js Deploy - v13.4.20-canary.16)

I have far less images, call it maybe 30 on the site. The more I click around (and the more images that load, presumably sitting in memory), the higher the memory becomes. It eventually grinds to a halt making <Image> unusable. I am using <Image> in a very straight forward way. The site is basically a blog (for all intents and purposes).

[csi-lk]

Can confirm we are seeing this issue in production on ^13.4.19, if we start curl'ing an image it will cause it to 500 after a short period of time

Setting unoptimised images in the next.config.js seems to workaround this for now

{
  images: { unoptimized: true },
}

Note: we are not using sharp

[MariuzM]

Thanks @csi-lk images: { unoptimized: true }, this is really good, i also noticed that images now loading faster with this.

[mengqing]

Any ideas on how to fix this issue other than setting unoptimized? We would still like to optimize the image but this memory leak is forcing us no choice but to turn optimization off

[MuhammadMinhaj]

Is there any update? Facing the same issue!

[rasfa98]

Do we have an update on this? As @mengqing mentioned it would be great to have the possibility to still optimize the images 😃

[khuezy]

It's probably better to offload the compute to its own service rather than forcing the main app server to handle it... which will block the main server and significantly reduce your throughput/latency.
See: https://nextjs.org/docs/app/api-reference/next-config-js/images
You can set up a distributed/scalable service on fly.io... fairly simple to implement it in rust or some other speedy language.

[houxd1992]

Can confirm we are seeing this issue in production on ^13.4.19, if we start curl'ing an image it will cause it to 500 after a short period of time

Setting unoptimised images in the next.config.js seems to workaround this for now

{
  images: { unoptimized: true },
}

Note: we are not using sharp

Do you guys think the cause of the memory leak is sharp or image optimised?
I'm confused.

[csi-lk]

Hmmm, as I mentioned above "Note: we are not using sharp" so I doubt it's related to sharp

I also haven't looked back into this for a while as we are just "pre-optimizing" our images manually (using img-optim then shipping those)

[cwatson272]

We have the same issue. We have thousands of images and very quickly our server gets crushed if we have optimization enabled across the site, even with very little traffic routed that server (lets say 100 page views/hour). We've since kept it to our ~30 or so static images and have offloaded everything else to Cloudinary, but it would be nice to use this function in more places.

Edit: turns out this is likely because we are on Heroku and it's filesystem is ephemeral. I think this looks for the file in the filesystem, can't find it, then generates a new image, but doesn't purge from memory until it can be found in or saved to the filesystem or something.