[Feature]: Add a setting to limit worker memory utilization

[iansu]

🚀 Feature Proposal

I'm proposing adding a flag to Jest that will control the maximum amount of memory that a worker uses. This setting could be either a number in megabytes or gigabytes or a percentage.

Motivation

I have experienced a lot of issues with Jest in various projects, especially projects using TypeScript, where Jest workers will run out of memory and crash. Fortunately Jest is able to recover from this issue and just spawns a new worker. However, when this happens you get a very long Node.js out-of-memory error dumped to the console which makes the test results hard to read. Multiply this by 4 or 8 or more workers and it makes quite a mess.

With my proposed changes the worker would check the provided threshold(s) and if it's exceeded them the worker would silently exit and free its memory, causing Jest to spawn a new worker.

I have implemented these changes without a flag in this repo: https://github.com/iansu/jest-runner-iansu

In this case I have hardcoded the worker to exit when its memory utilization exceeds 80%. I've tried this out in a number of projects that have Jest memory issues and it seems to work quite well. Respawning the worker does add a small overhead in terms of time but the benefits seem to far outweigh that.

Example

Here's an example using a workerMemoryThreshold flag. I just made up the name of this flag for this example. I don't have a strong opinion on what it should be called.

set a maximum of 1GB memory per worker

jest --workerMemoryThreshold=1G

set a maximum of 80% heap usage per worker

jest --workerMemoryThreshold=80%

Pitch

In a perfect world this wouldn't be an issue but in reality it very much is. If you Google for Jest memory issues or check the issues in this repo you find a lot of results. It's unclear if this issue ultimately lies with Jest, Node.js, V8, some other project or some combination/interaction between these projects.

I hope that the root cause of these issues is found and addressed in the future but in the meantime what I'm proposing is a very simple solution that will at least prevent Jest from running out of memory and logging out-of-memory errors to the console.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[iansu]

👋 not stale

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[iansu]

👋

[SimenB]

@iansu will this be solved by #13056?

[iansu]

@SimenB I think so. That looks very similar to what I'm proposing here. I'll take a closer look at that PR and leave some comments.

[phawxby]

@SimenB I think so. That looks very similar to what I'm proposing here. I'll take a closer look at that PR and leave some comments.

The main difference between your request and my PR is that it's not an absolute memory limit, it checks after a job is complete if the worker needs to be recycled to protect against memory leaks. So you could set the limit at 200MB but that would prevent it if from using 5GB while it's actually executing a job.

[iansu]

Closing this as it's implemented in #13056. Thanks @phawxby!

[SimenB]

Yay! 🎉

[SimenB]

The alpha works for your use case @iansu?

[iansu]

@SimenB I haven't tried it yet but I will

[iansu]

I am having trouble getting this work. I'm using v29.0.0-alpha.4 and the command jest --workerIdleMemoryLimit=200MB --logHeapUsage and the heap size just keeps growing well past 200MB.

[iansu]

Reopening this until I can get the new workerIdleMemoryLimit to work. I'll see if I can put together a repro.

[SimenB]

@iansu see #13107

[phawxby]

@iansu give it a try on the latest alpha

[iansu]

It's working now with alpha 5 so I'll close this. Thanks!

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.