Fix memory leaks of jest-worker #11187
Conversation
|
Deploy preview for jestjs ready! Built without sensitive environment variables with commit 698e78e |
|
Deploy preview for jestjs ready! Built without sensitive environment variables with commit 852ec1c |
There was a problem hiding this comment.
First of all, thanks a lot for tracking down the source and finding a fix for this!
I've added a regression test that fails on master but passes with your fix (pretty cool how easy to test this is with the leak detector IMO), and moved the changelog entry do the perf section where I think it fits better.
Let me know if you are ok with the changes :)
|
Thank you for adding the test @jeysal, it looks great! |
|
Great spot and thanks again! |
|
@MichaReiser FYI |
|
@shuding would a pre-release of this help? If not urgent a stable release should be out this month (🤞🤞🤞) |
|
@SimenB Yep we're good with a pre-release, thanks for your help! |
|
|
This PR upgrades `jest-worker` and `jest-cli` to the latest pre-release version, also removed `jest-circus` which is included in Jest by default. `jest-worker@next` includes a fix for memory leak that we need (jestjs/jest#11187). Fixes #22925. This will also improve the OOM issue for `next dev` #15855.
This PR upgrades `jest-worker` and `jest-cli` to the latest pre-release version, also removed `jest-circus` which is included in Jest by default. `jest-worker@next` includes a fix for memory leak that we need (jestjs/jest#11187). Fixes vercel#22925. This will also improve the OOM issue for `next dev` vercel#15855.
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Summary
We noticed some closure related memory leaks in the current implementation of
jest-workerwith v8. The arguments and result of the task of each worker can't be garbage collected until a new task is assigned to that worker. If the arguments/result sizes are large, it's gonna introduce a huge memory increase.This PR ensures that
argsandtaskare correctly scoped so they won't be claimed by other closures, and references can be cleaned once they become unneeded.Test plan
We have an app that passes image buffers as the argument of tasks which are pretty large, and we are seeing a huge memory leak after the tasks have finished (ref: vercel/next.js#22925). Here's a screenshot for the memory dump:
The memory usage increased from 112MB to 145MB (+29%) after processing the task, and most of the new allocated memory was taken by the arguments and result of that task, which should have been recollected after it ends.
With this fix we are no longer to reproduce this leak with the same app: