Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Lock and update deps to fix build memory leak
- I've been to hell and back, and haven't managed to pinpoint all of the exact dependencies that cause the build to leak (or more likely: attempt to over-allocate) memory but after extensive testing, this is the set of dependencies that manages to build in an environment with max 3GB of memory available
- It could very well be that some of the dependencies blocked by this commit's locks simply take the build's memory consumption over the 3GB limit and aren't part of the same issue of over allocating memory -- but this is a decent compromise that ensures that the build should no longer intermittently fail in CI (which has 4GB, of which roughly 3,3GB is usable)
- The test setup:
- `docker run --rm -it --volume $(pwd):/home/circleci/project --memory=3G cimg/node:14.15` (same image as in CI, and 3 GB of memory ensures that the build will always work on the 4 GB available in CI)
1. `cd .. && cp -r project frontend` (just to ignore any file permission issues without touching the executor image too much)
2. `rm -rf dist node_modules`
3. `yarn install --immutable`
4. `yarn build`
5. -> repeat at least 3 times and if all pass (& CI passes) -> OK set of dependency versions
- 8 parallel CI jobs running to increase sample size with `/sys/fs/cgroup/memory/memory.max_usage_in_bytes` logged
- Findings:
- _Something_ (or a combination of things) attempts to over allocate memory (specifically new heap space) when running with `master`'s versions of frontend dependencies (likely culprits described below)
- With the versions used in this commit, the build will use all memory available to it but GC is early enough to never actually go over the limit **but** something in the newer dependencies goes over the line, causing reaping of the build processes and crashing the build
- The most likely explanation would be something "outside" the Node.js process (i.e. something creates additional processes), which doesn't obey the same heap limits
- TypeScript 4.3 has a few issues with relating to complex types which lead to at least significantly higher memory usage (likely not the same over-allocation issue as below) -> **lock to ~4.2.x**
- Related: microsoft/TypeScript#44299 and microsoft/TypeScript#44281
- This is actually a good idea anyway as TS makes breaking changes in "minor" releases (e.g. 4.2.x -> 4.3.x) and we shouldn't blindly update it
- webpack's dependency `terser-webpack-plugin@5.1.2` updates `jest-worker` 26 -> 27 which appears to cause memory over allocation, likely related to the way it creates new processes and/or threads for workers based on incorrect assumptions about the available amount of CPU cores inside a Docker container (`os.cpus()` vs. what's actually allocated in cgroups) -> **lock to 5.1.1**
- There's also a big issue with shared dependencies with `@storybook/*` modules also using webpack (but *4.x.x* instead of 5.x.x), which makes using `resolutions` pretty difficult without locking absolutely everything
- Related: webpack/webpack#13550 and webpack-contrib/terser-webpack-plugin#403
- `@types/node@16.x.x` is incompatible with Jest 27 (jestjs/jest#11640) -> update only to 15.x.x
- `webpack@5.38.0` updates _some_ dependency that either causes a similar memory over-allocation or simply increases the build's memory usage over the set 3GB limit
- My money's on `webpack-sources` as based on webpack's changelogs it appears to have had some memory leak issues between the versions used by `webpack@5.37.1` and `webpack@5.46.0` (latest at the time of this commit)
- `ts-node@10.x.x` could also be related but didn't manage to find anything conclusive here
- Additionally, update all dependencies _that don't break the build / cause memory issues_ to the latest non-major versions
- Excluded Chart.js (& related) dependencies as that whole repo seems in a very weird place now
- 3.x.x released a while ago and there hasn't been any release notes, tags or anything
- 3.0.0 diff is pretty massive with very non-descriptive commit/PR messages
- Excluded `react-select` and `react-datepicker` as they're going to be dropped at some point and both of them appeared to have pretty major changes
- Excluded Testcafe as that's always a pain & we're going to drop it when everything's been re-written for Playwright
- Update webpack loaders to their latest (incl. major) versions -- nothing breaking for us- Loading branch information
