Memory issue with large modules on Linux for Swift 4.2

[Fryie]

We have a server-side project (using SPM) and have a script to generate documentation for it. However, when trying to migrate to Swift 4.2 (from 4.1.2), we noticed that in our docker container on CI, the documentation couldn't be generated anymore for one of the modules (the other modules in our project worked fine). Basically, what happened was that the documentation generation started becoming very slow after ca. 30-40 files and then ended up being killed. That problem does not appear on macOS.

This behaviour suggests some kind of memory issue, either a leak, or just an inefficient use of memory (maybe on large modules, some intermediate information could be stored on disk?).

To reproduce this issue I created a silly repository with a single module that has 2001 source files—all of them very simple, one Base class and 2000 children inheriting from it. Our repository has of course fewer files, but they are more complicated; I would suspect that the underlying issue is the same.

Running sourcekitten doc --spm-module Dummy on a Linux machine seems to eat up loads of memory. I tested it on a Vagrant machine and sourcekitten was using about 84% of memory while >98% of CPU time was spent swapping. At some point the process is simply killed. Running htop shows that the memory usage is completely off the charts (it takes less than 5s to grow above 500MB).

In the linked repository, the issue can be reproduced by running everything inside a container (just by using make linux42). The container seems to be able to handle a little bit more than my Vagrant VM does, but at some point it gives up nonetheless.

This doesn't happen on my mac (just test with make mac): While the generation is slow and memory also increases, the progress seems more steady instead of everything becoming progressively slower, and towards the end Activity Monitor shows that < 600MB are being used by sourcekitten, while < 200MB are being used by SourceKit, but this takes a very long time to build up, as opposed to the Linux situation.

We can also test the issue on the Swift 4.1 release (make linux41). If I try that, I find out that sourcekitten will be able to process a slightly larger amount of files before grinding to a halt (on my machine, it's about 245 files on Swift 4.2 and about 300 on Swift 4.1 until things start to go bad). This might explain why in our project, it still worked fine with Swift 4.1, but now it doesn't anymore.

Now, of course I'm not sure if I'm on the right track here and/or if this is a sourcekitten issue or a bug in SourceKit.