Run Webpack in separate process to reduce memory usage

[KyleAMathews]

Some people have complained about Gatsby running out of memory on small VPSs.

I just hit the node.js built-in heap limit while building a largeish site.

Apparently by design Webpack doesn't release memory after builds so while building Gatsby sites we accumulate memory with each build. Running Webpack in a separate process would ensure the memory is reclaimed after each build webpack/webpack#2137

[KyleAMathews]

By running webpack in separate process I mean we'd write out the config for each webpack build to a temp file and then run webpack as a child process so that when it finishes its memory gets released.

[aaronshaf]

I came here from #1829. I am having a similar problem: ~1500 pages in the site, crashes upon gatsby development or gatsby build

[szimek]

@KyleAMathews @m-allanson Any chances for some tips on how to move on with this ticket?

The blog I'm working on has 6 languages with ~600 pages in each language. During the development phase we were building only 2 languages, but now that we're finally getting closer to deployment, I've added a third one and I started getting these "JavaScript heap out of memory" errors when building HTML pages (#1651).

It works fine for 3 languages, if I use node --max_old_space_size=4096, but I'm not sure if it will work with 6 languages and as we're going to add more pages, there's always a risk that at some point we'll start running out of memory. Our Jenkins worker nodes have only 2GB RAM and Travis CI Ubuntu Trusty environment, which we're using to run our tests, has 4GB.

Long time ago I fixed another "out of memory" error we were having, which was caused by running GraphQL queries for all nodes at the same time (8517e09). I'm not sure if it can be as easily easily fixed, or if it requires some bigger changes in Gatsby architecture

[pieh]

@szimek you might be interested in checking out this PR for v2 (mainly this comment with memory usage changes) - #4912 (comment)

[KyleAMathews]

Perhaps still a good idea but v2 uncovered a bunch of other lower-hanging perf improvements to pick so closing for now.