-
Notifications
You must be signed in to change notification settings - Fork 29.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
test: log before and after RSS in memory leak test #21080
Conversation
Suggesting to fast-track this. |
+1 to fast-track |
@Trott can I take this as you approving fast tracking? |
Re-run Linux CI because debugging another issue left around some root-owned files causing a build to fail: https://ci.nodejs.org/job/node-test-commit-linux/19255/ |
Yes. |
https://ci.nodejs.org/job/node-stress-single-test/1893/ once it starts (predicted: 2 hours from now.) |
Linux re-run is green. AIX might take a while to start because I'm currently using one (and possibly the only right now) AIX hosts for a stress test. But it will start! |
(By the way, I think @tniessen already got results from this in a CI run off a branch. So the delay in landing this hasn't affected it's usefulness.) |
This is the output from https://ci.nodejs.org/job/node-test-commit-linux-containered/4862/nodes=ubuntu1604_sharedlibs_debug_x64/console, its like 5.097MB, so just above the limit. My suggestion in the referenced issue should solve the problem, I can open a PR tomorrow. (Even though I am not sure whether the problem reproduces with the recent v8 update.)
|
It transpires that the extra bookkeeping in debug builds sometimes makes the increase in RSS go _just_ over the 5 MB limit, by fewer than 100 kB. Double the limit so we hopefully don't run into it any time again soon. The memory leak it tests for was one where RSS grew by hundreds of megabytes over the lifetime of the test; 5 vs. 10 MB is insignificant. Fixes: nodejs#21076
I added another commit that doubles the slop limit, PTAL. https://ci.nodejs.org/job/node-test-pull-request/15221/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I remember correctly, you originally wrote the regression test, so I'll trust your judgement on this one, but in my patch I also increased the iteration count: assuming there is a memory leak, it might go undetected in release builds if the amount of memory leaked per iteration is sufficiently small (which might not be the case, I didn't go into depth in the original issue). The patch which I showed to @Trott yesterday used 5e5
iterations and a limit of 8MB, which i assumed were reasonable choices.
Forgetting to free a DH key's memory leaks kilobytes at a time so 50k iterations should be plenty to flush that out. I picked 50k because otherwise it gets really slow with FIPS builds, because you can't use keys < 1024 bits. |
Okay, thanks for the explanation, I wasn't sure which part was being leaked. In that case this sounds reasonable. |
Refs: nodejs#21076 PR-URL: nodejs#21080 Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Anatoli Papirovski <apapirovski@mac.com> Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com> Reviewed-By: Yuta Hiroto <hello@hiroppy.me> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
It transpires that the extra bookkeeping in debug builds sometimes makes the increase in RSS go _just_ over the 5 MB limit, by fewer than 100 kB. Double the limit so we hopefully don't run into it any time again soon. The memory leak it tests for was one where RSS grew by hundreds of megabytes over the lifetime of the test; 5 vs. 10 MB is insignificant. Fixes: nodejs#21076 PR-URL: nodejs#21080 Refs: nodejs#21076 Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Anatoli Papirovski <apapirovski@mac.com> Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com> Reviewed-By: Yuta Hiroto <hello@hiroppy.me> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Landed in 7b6c428...997e97d |
Refs: #21076 PR-URL: #21080 Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Anatoli Papirovski <apapirovski@mac.com> Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com> Reviewed-By: Yuta Hiroto <hello@hiroppy.me> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
It transpires that the extra bookkeeping in debug builds sometimes makes the increase in RSS go _just_ over the 5 MB limit, by fewer than 100 kB. Double the limit so we hopefully don't run into it any time again soon. The memory leak it tests for was one where RSS grew by hundreds of megabytes over the lifetime of the test; 5 vs. 10 MB is insignificant. Fixes: #21076 PR-URL: #21080 Refs: #21076 Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Rich Trott <rtrott@gmail.com> Reviewed-By: Anatoli Papirovski <apapirovski@mac.com> Reviewed-By: Сковорода Никита Андреевич <chalkerx@gmail.com> Reviewed-By: Yuta Hiroto <hello@hiroppy.me> Reviewed-By: James M Snell <jasnell@gmail.com> Reviewed-By: Tiancheng "Timothy" Gu <timothygu99@gmail.com> Reviewed-By: Luigi Pinca <luigipinca@gmail.com>
Refs: #21076
To help investigate said bug.
CI: https://ci.nodejs.org/job/node-test-pull-request/15212/