Checksum failed for nightly channel

[dswd]

I just moved from multirust to rustup and experience the following problem when installing nightly:

$> rustup update nightly
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
error: checksum failed, expected: 'afb2094f716fb7167accccdbc9e213ad5e707e7b56b0175aa664108cf994eaa0', calculated: 'b60f814a446940366499a350769dab93bce25e91bb30825eebb3f74820fd8b74'

Stable and beta install just fine. Also I removed all remains of multirust from my system(removed ~/.multirust, cleared $PATH, fresh console) and tried again, no help.

To clarify, I do not have any version of nightly currently installed (it seems to work for people with nightly already installed):

$> rustup show
installed toolchains
--------------------

stable-x86_64-unknown-linux-gnu (default)
beta-x86_64-unknown-linux-gnu

active toolchain
----------------

stable-x86_64-unknown-linux-gnu (default)
rustc 1.9.0 (e4e8b6668 2016-05-18)

I am using the current version of rustup:

$> rustup -V
rustup 0.1.12 (c6e430a 2016-05-12)

[alexcrichton]

Someone was also reporting this yesterday where it was a checksum failure in the manifests. Definitely seems fishy, as especially the toolchains should never have bad sha256 sums, right @brson?

As this was happening (the failure), the nightly-dist-packaging-mac, nightly-dist-packaging-win-gnu-32, and nightly-dist-packaging-win-gnu-64 builders were all running. I don't think that should affect this, though, because the new TOML manifest wouldn't have been uploaded so none of the new artifacts should have been used.

I also checked and we had no cloudfront invalidations in flight when this happened.

[dswd]

Just a quick update: The expected checksum changed but it still does not match:

$> rustup install nightly
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
error: checksum failed, expected: '82d81d2254bcb2c088a0adcc06964b530fe34813930ec634119e43c19c43db64', calculated: 'b60f814a446940366499a350769dab93bce25e91bb30825eebb3f74820fd8b74'

[dswd]

Ok I finally found the problem! I found a folder called cargo-install.we9Ct4dun7pK in /tmp. Once I removed it, nightly installed just fine.
I will leave this issue open. Please close it if you don't consider this behavior a bug.

[alexcrichton]

Whoa! Sounds like we may not be invalidating a cache somewhere perhaps, definitely seems like a bug though.

[kamalmarhubi]

Someone was also reporting this yesterday where it was a checksum failure in the manifests.

Someone was me, and the checksums in the report today are the same checksums I was seeing yesterday, which seems bad.

[kamalmarhubi]

Oh cool, I still have the files in /tmp:

/tmp$ cat channel-rust-nightly.toml.sha256 
afb2094f716fb7167accccdbc9e213ad5e707e7b56b0175aa664108cf994eaa0  channel-rust-nightly.toml
/tmp$ sha256sum channel-rust-nightly.toml
b60f814a446940366499a350769dab93bce25e91bb30825eebb3f74820fd8b74  channel-rust-nightly.toml
/tmp$ ls -l channel-rust-nightly.toml*
-rw-r--r-- 1 kamal kamal 78430 Jun  9 15:13 channel-rust-nightly.toml
-rw-r--r-- 1 kamal kamal    92 Jun  8 14:57 channel-rust-nightly.toml.sha256

[alexcrichton]

Right! If you remove the old tmp files does an update work for you @kamalmarhubi?

[brson]

Definitely seems fishy, as especially the toolchains should never have bad sha256 sums, right @brson?

@alexcrichton I believe right now it's possible to get checksum drift on the manifests. The self-update will just warn, but if the manifests don't agree with their .sha256 file it is still an error.

[brson]

I'm surprised @kamalmarhubi has manifests sitting directly in their /tmp folder. rustup should be using TempDir to create subfolders.

I wouldn't expect the contents of /tmp would impact these checksums.

[kamalmarhubi]

@brson should have mentioned: those were files I downloaded directly from the dist site while investigating the issue. Basically, I was verifying that the mismatch I saw on Thursday was not from weird local caching.

Things work now, independently of those files being in /tmp. :-)

[kamalmarhubi]

@brson

I believe right now it's possible to get checksum drift on the manifests.

Is this something that can be fixed? The drift I saw lasted for at least tens of minutes, and up to many hours depending on when OP saw those same checksums. This seems a bit long.

I'm not sure what the serving infrastructure is behind cloudfront. Is it immediately backed by S3? Trying to figure out what's making this hard, and what could be done to change it.

[alexcrichton]

Yeah I'm curious how checksum drift is possible in the manifests. I thought it was only in the small window where an invalidation is in flight (or we're in the middle of an upload), but when this happened I confirmed and neither of those was happening.

[brson]

Let's switch both the self-update and manifest checksum checks to use the HTTP e-tags. The download_and_check function will need to change - instead of accepting a hash and downloading the paired .sha256 file it will accept an etag and check it against the HTTP etag header.

To facilitate the upgrade we'll need to maintain both code paths. We'll change the metadata format to store etags, like we currently do with update hashes. If there's an update hash (but no stored etag) dfor a particular artifact then use the old code path, otherwise use the etag path.

The way we store etags will need to be slightly different than update hashes. Right now we don't store update hashes for the self-updates - we just calculate them from the running bin. So we'll need to store them in a format that supports etags for self updates or channels. Maybe it makes sense to make it a hash table of URLs to etags. I don't think this will require a metadata version bump.

[gyscos]

It happens here, but only on some computers:

$ rustup update nightly
info: syncing channel updates for 'nightly-x86_64-unknown-linux-gnu'
error: checksum failed, expected: '7270d7499edc955910c9bcc8198584fc8cb6f96914097d30f2138a2fd6957cc1', calculated: 'b22674bc7474c33c30497ac040c6adbceac13a79b55f1fd147323c3a80bdd75f'

While others are fine.
Is there some cache somewhere that I'd need to clean? Or should I just try to re-install rustup entirely?
Note: ~/.multirust/tmp is empty. ~/.multirust/update-hashes only contains stable-x86_64-unknown-linux-gnu.

EDIT: I guess it was a temporary server-side issue? Anyway, I didn't change much, but it works again.