Adjust external allocated memory less often #429
Conversation
Apparently calling v8::Isolate::AdjustAmountOfExternalAllocatedMemory frequently results in lots of wasted CPU cycles on garbage collection, per discussion here: meteor/meteor#10527 (comment) This fix was inspired by marudor/libxmljs2#22, which seems to have addressed nodejs/node#30995. Another project that benefitted from adjusting external allocated memory less often: mapnik/node-mapnik#136
Thanks to the allocation of the Coroutine stack (see set_stack_size), each Fiber has a size of approximately 1MB on a 64-bit machine, so using a threshold of 1MB gives no improvement over adjusting the memory every single time a fiber starts or exits. Instead (with this commit), we adjust memory whenever the gap exceeds 10x the size of a Fiber.
|
Doesn't v8 already have some threshold logic? It seems like v8 won't actually send a notification to the GC unless more than 32mb has been allocated. Each fiber is a 1mb so that would be a lot of fibers. Also did you experiment with just removing |
|
It looks like V8 has imposed a threshold since v6.1.58, which was first present in Node v9.0.0, which updated V8 to v6.1.534.36. The threshold was only 1MB until v6.1.94, when it increased to 32MB (its current value)… but that change should also have been included in the Node 9 changes (and thus Node 12.14.0 too). Hmm. Without doing a lot more digging, I don't have a good answer as to why the V8 thresholding was inadequate, though I would note that this area of the code is fairly complicated, with some significant known bugs like getting the subtraction backwards. Perhaps there are still unknown bugs that cause the memory pressure to be checked too often? It's either that, or multiple Meteor developers are misreading their CPU charts in the same way, which seems unlikely. In other words, I would be more than happy for Is there a particular experiment you would like to see to validate this removal, other than running a patched version of |
|
@laverdet Ok, from the Meteor perspective, simply not calling Would you be willing to release this (much simpler) change as Besides your argument about V8 not having much opportunity to collect |
|
Sounds good to me. Based on my experience the v8 garbage collector is perfectly capable of shooting itself in the foot without us helping. |
|
4.0.3 is on npm and untagged as suggested. I didn't test any of the binaries and rushed through the deployment but I had my fingers crossed so everything should be fine. |
This includes laverdet/node-fibers#429, fixing the CPU spikes reported and discussed here: #10527 (comment) Using an official fibers release rather than a GitHub URL is preferable because it doesn't require building fibers from source when deploying a Meteor app to production, and also doesn't rely on GitHub being operational, though of course it does rely on other networked services like npm.
|
I did notice in the Galaxy logs that However, this was also happening for @filipenevola If this ever becomes a problem, I think we just need to update our base Ubuntu image here: https://github.com/meteor/galaxy-images/blob/master/galaxy-app/Dockerfile#L1 (let me know if you need my help with that). |
|
No telling. The linux binaries come from a Gentoo server I have laying around that hasn't seen an update in 5 years or so. |
Calling
v8::Isolate::AdjustAmountOfExternalAllocatedMemoryfrequently appears to result in excessive garbage collections and wasted CPU cycles, per discussion here and as stated in the Node documentation: "Registering externally allocated memory will trigger global garbage collections more often than it would otherwise."While I think the V8 implementation could/should do a better job of handling requests to adjust external memory without over-reacting to each and every request, in the meantime the
fiberslibrary can work around the issue by not callinguni::AdjustAmountOfExternalAllocatedMemoryas often. You can see in the Meteor 1.9 PR discussion that this change seems to have fixed the CPU spikes that multiple developers encountered in production, and passes all of Meteor's tests, which exercise fibers pretty thoroughly.This fix was inspired by marudor/libxmljs2#22, which addressed nodejs/node#30995. I would note that the current implementation in this PR under-reports external memory usage, since the reporting lags behind the actual usage by up to ~10MB. I'm definitely open to switching things around a bit so that
uni::AdjustAmountOfExternalAllocatedMemorygets called with a larger number of bytes than necessary, thereby over-reporting the external memory usage, if you prefer that approach.Here's another project that benefitted from adjusting external allocated memory less often: mapnik/node-mapnik#136
It would be ideal to ship Meteor 1.9 with an official version of
fibers, rather than a GitHub URL, but I realize that's asking for your time at a busy/vacation time of year.