azure: Investigate seemingly spurious failures on CI

[alexcrichton]

Git checkout failures

• arm-android git checkout failure
• armhf-gnu git checkout failure
• dist-various-2 git checkout failure
• even more git checkout failures

Submodule weirdness on Windows

• https://dev.azure.com/rust-lang/rust/_build/results?buildId=295
• https://dev.azure.com/rust-lang/rust/_build/results?buildId=287

Other

• OSX connection lost during a try build
• Build failed to start but marked as successful

[pietroalbini]

• even more git checkout failures

fatal: reference is not a tree: 3b109e51cc8d9a3a833cf7e28db7911a8a0aa503
##[warning]Git checkout failed on shallow repository, this might because of git fetch with depth '2' doesn't include the checkout commit '3b109e51cc8d9a3a833cf7e28db7911a8a0aa503'. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=829603)
##[error]Git checkout failed with exit code: 128

This doesn't seem to be a spurious failure: these jobs started after bors marked the PR as failed due to other failures on Travis/AppVeyor, so the auto branch was already force-pushed to the next PR. We shouldn't see these anymore once we start gating on Azure.

[pietroalbini]

For the submodules weirdness I opened #61322 to investigate.

For the git checkout failures (logs) it seems the error is happening on the Azure side, and there is not much we can do on our end. Any idea on what could have caused it, or what steps we can take to investigate it @rylev @johnterickson?

[alexcrichton]

OSX connection lost during a try build

[johnterickson]

I see two trends - let me know if I'm missing another:

This looks like the auth to GitHub is not working and so git is falling back to a command prompt for creds. Hmm...

##[command]git -c http.extraheader="AUTHORIZATION: basic ***" fetch --tags --prune --progress --no-recurse-submodules --depth=2 origin
fatal: could not read Username for 'https://github.com': terminal prompts disabled
##[warning]Git fetch failed with exit code 128, back off 4.338 seconds before retry.

This sounds vaguely like a race we were seeing where we are getting a notification from GitHub before the commit has been replicated everywhere.

fatal: reference is not a tree: 3b109e51cc8d9a3a833cf7e28db7911a8a0aa503
##[warning]Git checkout failed on shallow repository, this might because of git fetch with depth '2' doesn't include the checkout commit '3b109e51cc8d9a3a833cf7e28db7911a8a0aa503'. Please reference documentation (http://go.microsoft.com/fwlink/?LinkId=829603)
##[error]Git checkout failed with exit code: 128

[pietroalbini]

This sounds vaguely like a race we were seeing where we are getting a notification from GitHub before the commit has been replicated everywhere.

@johnterickson I'm pretty sure that's our fault and "expected behavior".

Every time a new build needs to start bors force-pushes the merge commit on the auto branch, thus deleting the previous build's merge commit. If the new build is started shortly after the previous one (for example if someone cancels the build on the bors side to let an higher priority one start) the checkout step of the previous one might have not started yet and it will try to get the old HEAD, causing that error.

I've also seen the error only on builds that ran for ~5 minutes before being killed by our cancelbot because a newer build started, confirming the hypothesis.

[pietroalbini]

There is another spurious failure which is pretty bad: this build failed to "prepare" but was marked as successful on the GitHub Checks.

[johnterickson]

@pietroalbini I raised that "succeeded but actually failed" build with the right team - definitely concerning and I appreciate you reporting it.

[alexcrichton]

Another two that have come up:

• msvc-tools and gnu-tools on https://dev.azure.com/rust-lang/rust/_build/results?buildId=398 show:

+ /tmp/checktools.sh ../x.py /tmp/toolstates.json linux
fatal: ambiguous argument 'HEAD^': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

@pietroalbini or @kennytm do y'all know what might be causing that?

• All of a sudden our script to enable ipv6 in docker is failing. I don't know why this changed all of a sudden but it shouldn't be the hardest thing to work around (testing something on try now)

[kennytm]

Maybe the HEAD commit did not have a parent commit? Is the commit 322afaf associated to any branch during git fetch --depth=2?