Cascading failure on bots when interrupted when performing git operations

[alexcrichton]

This is basically a best guess as to what's happening on the bots, but from time to time it appears that the build directory gets entirely corrupted with error messages like:

extracting C:\bot\slave\auto-win-msvc-64-opt-rustbuild\build\obj\build\cache\2016-05-24\rust-std-beta-x86_64-pc-windows-msvc.tar.gz
extracting C:\bot\slave\auto-win-msvc-64-opt-rustbuild\build\obj\build\cache\2016-05-24\rustc-beta-x86_64-pc-windows-msvc.tar.gz
extracting C:\bot\slave\auto-win-msvc-64-opt-rustbuild\build\obj\build\cache\2016-05-22\cargo-nightly-x86_64-pc-windows-msvc.tar.gz
Synchronizing submodule url for 'src/compiler-rt'
Synchronizing submodule url for 'src/jemalloc'
Synchronizing submodule url for 'src/liblibc'
Synchronizing submodule url for 'src/llvm'
Synchronizing submodule url for 'src/rt/hoedown'
Synchronizing submodule url for 'src/rust-installer'
fatal: Needed a single revision


command did not execute successfully: "git" "submodule" "update"
expected success, got: exit code: 1


Makefile:23: recipe for target 'all' failed
>> rustjob: found 0 remaining processes
Unable to find current revision in submodule path 'src/llvm'
make: *** [all] Error 1

This happened on a MSVC rustbuild bot, but I believe I've seen this before on basically all bots (but I feel the windows bots are affected more often).

My best guess as to what's happening here is:

• A build is running, and it's updating the LLVM submodule
• The build is canceled for any number of a few reasons
• The LLVM update process is interrupted, leaving the git directory in a corrupt state
• All future builds attempt to use this corrupt directory and fail

I... don't actually know what the best solution here is. May as well track it though!

[alexcrichton]

FWIW the fix now is to log into the bots and delete the obj directory and restart everything.