Checksum error downloading com_github_google_protobuf archive

[nlacasse]

GitHub seems to have changed how they generate their archive.tar.gz files, resulting in them having different SHAs than previously.

As a result, using go_proto_library fails:

no such package '@com_github_google_protobuf//': Error downloading [https://github.com/google/protobuf/archive/v3.4.0.tar.gz] to /home/nlacasse/.cache/bazel/_bazel_nlacasse/db32ba9596c053450aeadce0671ee2d0/external/com_github_google_protobuf/v3.4.0.tar.gz: Checksum was f6600abeee3babfa18591961a0ff21e7db6a6d9ef82418a261ec4fee44ee6d44 but wanted cd55ee08e64a86cf12aaadd4672961813f592c194ed0c9ad94da0ec75acf219f 

See also protocolbuffers/protobuf#3619

[jayconrod]

Thanks for reporting! Are you still seeing this issue? I can't reproduce this, but it's late and maybe I'm missing something obvious.

Looks like this is affecting lots of repos. A comment in libgit2 says this was due to an OS upgrade at GitHub, but that it's been reverted.

[ianthehat]

I saw this earlier, but it's not happening now. I originally assumed they had just re-released 3.4.0 on us or something.

Directly accessing git tar-balls is always slightly fragile, it's just better than the alternatives.
What we should probably be doing is dumping specific versions into googlestorage and then getting them from there instead, and have a really easy way to add new ones.
We should talk to the Bazel team about doing this as a general service to users, and whether it would be allowed. If the guarantee was strong enough about keeping the files around forever people would be far more trusting about repeatable builds as well.

[nlacasse]

This is still happening for me. Try a bazel clean and then build something that depends on protos.

I think the revert mentioned in that issue was something that happened a while ago, the last time they tried to change how archives are constructed.

[jayconrod]

Confirmed this is happening again now, in the same test workspace that was working last night. Apparently GitHub has rolled forward the change.

I don't think there's an easy solution to this. We should stop relying on tarballs served through codeload.github.com, since we have no guarantee that the sha256 is stable. So moving forward:

• We should change our dependencies fetched this way to be git_repository instead of http_archive. Dependencies downloaded from other servers with stable sha256 don't need to change; the Go SDK won't need to change.
• We should do a point release on top of 0.5.4 with this change.
• We should include tarballs of this repository in our releases so that people can continue using http_archive with this repo.

@ianthehat I'll start making these changes this afternoon, but let me know if you have an idea for a better approach.

[ianthehat]

I would really rather we did not switch to git_repository, it is essentially un-cachable (it is possible, but it's a much higher barrier to entry than http archives, especially given --experimental_repository_cache)
I really think we should be dropping the tarball in google fast file serving and fetching it from that instead.

[jayconrod]

We can do that for our own releases (retroactively, even), but we can't do that for our dependencies (proto, buildtools, etc).

I'm preparing a change that removes the sha256 from http_archives retrieved from codeload.github.com. We only had these for half of our dependencies already, and we're fetching them all over https. While this is not perfectly hermetic, it should be as secure as git_repository and still faster.

[ixdy]

The bazel docs recommend against using git_repository, but I guess we don't have a better option given that we can't trust GitHub to have stable repo archives.

[jayconrod]

Closing this issue after the 0.5.5 release.