Invalidate artifacts on CloudFront more reliably

[brson]

We're only on cloudfront because it was an easy way to set up HTTPS, but it's been a huge pain because our files are always out of sync. Since for #20629 we probably need at HTTPS proxy anyway, we should just do it ourselves with nginx.

[brson]

Nominating because the cdn sync issues tend to cause a lot of confusion.

[pnkfelix]

polish issue, assigning to 1.0 (but not 1.0 beta).

[grid443]

Can't install rust on Debian 7. error: invalid sha2

rustup: Downloading https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz to /tmp/tmp.fVUn9l8nsY/rust-nightly-x86_64-unknown-linux-gnu.tar.gz
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 134M 100 134M 0 0 488k 0 0:04:42 0:04:42 --:--:-- 458k
rustup: Downloading https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz.sha256
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 111 100 111 0 0 81 0 0:00:01 0:00:01 --:--:-- 93
rustup: Verifying hash
rustup: error: invalid sha2

[wildlyinaccurate]

Is there a workaround for this? Currently getting hash mismatches on OSX:

rustup: Verifying hash
rustup: error: invalid sha256.
 14ea55058922d8bfd1b9bcea80c288472314f1590eddd90fe9cb310c1072c357   https://static.rust-lang.org/dist/rust-nightly-x86_64-apple-darwin.tar.gz
 b0a0daaaad94a4e4539b10aae87e37144d070c1eccad6760cfb1354f84f19ae2   /tmp/rustup-tmp-install.EIMy0euI/rust-nightly-x86_64-apple-darwin.tar.gz

[steveklabnik]

@wildlyinaccurate the workaround is 'wait a while' 😦

[wildlyinaccurate]

Managed to get it installed via Homebrew on OSX. I realise that's not very helpful for anybody on Linux, though...

[buster]

I had this issue for days now and i never had this for the last months... it's truely annoying.. :(

[5nyper]

Also experiencing this with Debian 😞 :

[amozz]

In ubuntu12.04, also encounter this problem.

rustup: Verifying hash
rustup: error: invalid sha256.
 ff91127472b58a7373a0d097bc16c35cb4d6b1f310e89c2b3611b09ad62cb0eb       https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz
 38add12a89c22c3518217ed631f22f7f2431af0867d496012af5d159fdbc8ad6       /tmp/tmp.7o8rujgQAn/rust-nightly-x86_64-unknown-linux-gnu.tar.gz

[5nyper]

Works here now

[davorb]

@wildlyinaccurate Download the rustup.sh-file, add a return-statement to the first line of the verify_hash()-function and then pipe that file to sudo sh. This will disable the sha256-check and let rust install. Keep in mind that doing this is a Bad Idea™.

[brson]

@brson you really must fix this by 1.0.

[mvdnes]

Just a suggestion, but maybe there is a way to automatically invalidate the cache?

$ curl -I https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz
(...)
Last-Modified: Fri, 27 Mar 2015 12:16:21 GMT
Age: 80012

$ curl -I https://static.rust-lang.org/dist/rust-nightly-x86_64-unknown-linux-gnu.tar.gz.asc
(...)
Last-Modified: Sat, 28 Mar 2015 13:16:25 GMT
Age: 1247

I now have to wait an hour before the tar is up-to-date with the pgp signature. I would recon that the script that automatically uploads the nightlies could also trigger something like describeded on this stackoverflow answer?

[Virtlink]

This bit me too. It's really annoying having to wait for several hours, and confuses new users. @mvdnes: The files seem to have been updated.

[brson]

@mvdnes Yeah, we could probably update our build infrastructure to generate an invalidation and submit it. Even if we do that there will be windows where things are not in sync, but it should be pretty short.

We do a daily invalidation now, but the timing is loose, and it's a fixed list of files that don't always correspond to what has actually changed.

[brson]

Removing from 1.0 milestone.

[brson]

Instead of moving off cloudfront we might just modify rust-buildbot to do the correct invalidations at upload time.

[brson]

cloudfront now supports wildcard invalidations, which should make it much easier to invalidate the correct things.

[brson]

Updated title to reflect current plan.

[brson]

Plan:

• Remove the invalidate.sh script and its cron job.
• Add code to rust-buildbot's finish_dist function to generate a precise invalidation and submit it to aws.

[edunham]

TODO: add master config step that runs invalidation at end of nightly build.

[alexcrichton]

Done!