Start a new slave per build

[alexcrichton]

It is far too common nowadays that one build on our buildbot will end up corrupting all future builds, requiring a manual login to the bot to fix the state of affairs. The most common reason this happens is a rogue process remains running on Windows, preventing recreation of the executable (e.g. causing the compiler to fail to link on all future test runs).

It would likely be much more robust to start a new slave each time we start a new build. This way we're guaranteed a 100% clean slate when starting a build (e.g. no lingering processes). This would, however, require caching LLVM builds elsewhere and having the first step of the build to download the most recent LLVM build.

As such, this is not an easy bug to implement at all, hence the wishlist status for our infrastructure.

cc @brson

[nagisa]

Can’t ccache caches be shared between machines somehow, perhaps by storing the cache on a network share? This way we could just ccache llvm.

[alexcrichton]

That sounds like an excellent solution to the problem!

[brson]

A distributed ccache does sound useful here.

Slave per build will take a few more minutes per cycle because of the time it takes to start slaves, but it's not a great cost.

I wonder if we could use a simple distributed rustc cache to speed up half of the auto builds.

[vadimcn]

Sounds a bit heavy-handed.
If this were only about Windows, I'd suggest looking into Windows Job Objects. They are basically like containers for process groups, and you can set them up such that all processes get killed when job object is destroyed.

[dotdash]

Is this still an issue or is this handled by the job group(?) stuff that was implemented for the windows buildbots?

[alexcrichton]

I believe that using dojob.exe has fixed some of our problems (I haven't seen any issues related to this recently), but I would also like to still do this. Right now if you kill a build too quickly it will corrupt the git directory, causing all future builds using that build directory to fail. This currently requires manual intervention to delete the git directory or rebooting the slave in question.

[steveklabnik]

Triage: not aware of any changes here.

[alexcrichton]

This likely isn't going to happen, we'd basically have to rewrite our entire CI system.