x/build: publish GPG signatures of downloads for releases

[brianredbeard]

While providing SHA256 sums is helpful, it would be preferable to also validate that binary blobs are created by a trusted entity. Utilizing a GPG signing key and providing detached signatures (in .asc or .sig format) would allow users to automatically validate the heritage of a Golang tarball/MSI.

[bradfitz]

We do sign our *.msi and *.pkg files, because those OSes basically require it, and Google has infrastructure for signing binary releases for those.

I don't think Google has an existing mechanism for doing GPG signing, but I'm likely wrong, and that's not really the point. We could make it work if required, but I don't think GPG signing adds much anyway. We already have an API now to let you fetch (over HTTPS, with Google certs) the SHA-256 of the downloads, and then you can verify your curl with sha256sum or whatever. But even that is a little redundant, since we offer the downloads over https too.

Put another way: what's the threat you're concerned about?

[dominikh]

(Dup of #12749)

[bradfitz]

would allow users to automatically validate the heritage of a Golang tarball/MSI.

If either

1. the download came from https://storage.googleapis.com/golang/*
2. the SHA-256 of any set of bytes downloaded from any mirror (are there any mirrors?) matches the SHA-256 published at https://golang.org/dl/,

then its heritage is the Go team.

You used the word "automatically".

For 1), use curl without --insecure and curl will automatically validate Google's cert.

For 2), just append ".sha256" to URLs. Here is how to verify a hypothetical HTTP download against the HTTPS-verified SHA-256:

$ curl --silent -O http://storage.googleapis.com/golang/go1.6.src.tar.gz
$ (curl --silent https://storage.googleapis.com/golang/go1.6.src.tar.gz.sha256; echo "  go1.6.src.tar.gz") > sums
$ sha256sum --check sums
go1.6.src.tar.gz: OK

It's hypothetical because you can also just download it over https.

[brianredbeard]

Sorry, my concern is that of a hostile network with a compromised CA certificate. Merely trusting the CA bundle alone has proved to be a questionable practice especially as the hashes are not signed either. Signing hashes would be sufficient, but at that point why not just sign the binaries as well.

State actors, Bluecoat, etc are all capable of targeting users.

[bradfitz]

The SSL is no longer added and removed there.

I'm going to close this. You might be legitimately concerned, but we're not going to change our process here at the current time. Even if we did sign our binaries with GPG, you'd then be concerned and filing more bugs asking whether our public key is really our public key and its heritage and how you know it's ours. I promise Google's network and CA infrastructure is leaps and bounds tighter than anybody on the Go team alone would be able to do. The Go team isn't going to manage our own keys. If we did, it might make you happy for a bit, but it wouldn't be more secure.

[brianredbeard]

Seriously though, while I have a strong faith in the ability of the Google cloud team to properly secure, update, and maintain their SSL certificates, most users have a number of arbitrary certs in their bundle. While it's good that Chrome specifically handles updates to this bundle in a timely manner this is not the case of many installs of curl. As this is a compiler and not simply a binary which will be executed by users, it is ripe to be the target of an attack. Imagine the value of compromising the golang compiler for a user like @cloudflare, @coreos, BBC, or basically anyone on this list.

[csirac2]

Speaking of trusting arbitrary CAs, I've encountered an environment that added the in-house/private root CA to the CA bundles on their build boxes in order to deal with various self-hosted internal services running over TLS. Gaining control of the local CA certainly would've been a lot easier than compromising an actual CA.

GPG keys let build maintainers (rather than whatever is in the ca cert bundle this week) have a say in the provenance of their sources. Sure, you've got bigger problems than wonky build farms if someone is rummaging around on your network with a level of access that could exploit the above scenario, but the point is that CA infrastructure is a very different animal to specific, hand-curated GPG trust relationships. And, yes: builds break when signing keys change. For some, this is an important signal, and a feature.

Having helped administer an open source project that offered them, I can confirm that the number of people that access checksums (let alone signatures) is a tiny fraction of 1% of downloads. For many projects, it's perfectly justifiable not to offer GPG sigs. Just, let's not pretend that https is equivalent.

[minux]

even if the binaries were signed, how could you be sure the
gpg key belongs to the Go team?

If Go team publishes the public key on a web page, then again
you have to trust the CA infrastructure to be certain the key is
genuine.

If you don't trust the current PKI infrastructure, then I don't think
you can trust gpg signed binary distributions either.

Or do you suggest another more trusted way to distribute the
public key?

If you don't trust the CA, then just mirror the Go repository in a
trusted machine and always build from source to make your
own binary distribution. It's that simple.

[csirac2]

I'm not arguing that the Go project should do gpg signed binaries. Not doing so is perfectly defensible. As you say, we can build ourselves.

I am saying that nominating a specific signing key once (every 12-24 months) to explicitly trust is completely different to trusting that hundreds of standard CAs plus arbitrary ones unique to each system and whatever intermidiate CAs in between, will enable TLS to really do its job on every single connection.

I only wish to counter the perception here that GPG signatures bring zero benefit. It may be negligible benefit (depending on where you sit), but it isn't zero.

[broady]

Could you not do the same thing with the certificate used to secure golang.org?

The cert's current sha1 signature is 250667e0d6707003a665439694301899d213dd26. It will rotate once per year.

[csirac2]

Almost: curl --pinnedpubkey is apparently a thing now, but it's new enough that most distros aren't shipping it yet. curl --cacert and wget --ca-certificate aren't quite the same, but probably close enough in practice. We could also use openssl to extract and verify the signature shasum, but this isn't as clean because once again, it makes no guarantees for the next TLS connection over which we presumably fetch something.

Traditionally though, it's considered best practice to sign an artefact where possible, not just the connection it arrives on (hence why most software distribution channels continue to bother signing things rather than just trusting TLS).

Apologies for generating issue tracker noise, this is admittedly a very minor nit-pick, I do appreciate all the great work done here.

[bradfitz]

@broady, yup. And here's a program to do that, with some curl-like flags:

https://gist.github.com/bradfitz/5c17a11e1f631e81073c

[adg]

Once we provide an official apt repository (#10965), users on Debian/Ubuntu will have signed binaries they can verify, just like OS X and Windows.

[brianredbeard]

For archive purposes, more evidence of threat from BlueCoat -

https://crt.sh/?id=19538258

[adg]

@brianredbeard I'm not sure what to derive from that link. Can you provide some context?

[csirac2]

It seems people are worried about the implications of Blue Coat Systems - who (among other things) sell TLS MiTM proxy solutions - being made an official intermediate CA on the open internet by Symantec: https://news.ycombinator.com/item?id=11781915

[gtank]

I'll chime in to mention that this policy also makes it difficult to mirror or cache the Go binaries securely. With Linux distros I can serve an ISO image and detached signature anywhere. To offer the same security guarantees with Go I have to maintain a chain of custody all the way to a trusted server that uses TLS as well as Google does, or re-sign with my own (not widely trusted) GPG keys.

The Go team itself doesn't have to be involved with the signing to improve the situation. Just being able to verify that a particular artifact came from within Google without consulting Google servers each time would help. And Google does have at least one existing mechanism for doing GPG signing: https://www.google.com/linuxrepositories/

[thomsh]

No update on this thread sice May 2016, @brianredbeard raise a good point.

[bradfitz]

@theznx, there are no updates on this thread because it's closed.

But I have aspirations of redoing a few pieces of our release tooling for Go 1.9 anyway, so I'll reopen this to track.

If Google's signing infrastructure supports GPG, we could published a detached signature as well. @broady, do you know? If so, it wouldn't add much pain to our release, since we already do all the binary signings in one step today.

[bradfitz]

@broady, see internal docs on how to do this. I created shortlink http://go/gpg-sign.

If you have time to do this for Go 1.9, that'd be cool. Otherwise move this to Go 1.10.

[bradfitz]

@broady, do you want to do this for this release?

[broady]

Sorry for the inaction. Yes, will look this week, hopefully get it done before 1.9 RC2.

[gopherbot]

Change https://golang.org/cl/58270 mentions this issue: release: upload asc files

[broady]

I've signed go1.9rc2 for linux-amd64 only. I assume this is the artifact people need signed the most. Is that right?

wget https://storage.googleapis.com/golang/go1.9rc2.linux-amd64.tar.gz
wget https://storage.googleapis.com/golang/go1.9rc2.linux-amd64.tar.gz.asc
gpg --verify *.asc *.tar.gz

I'm sorry, I don't know much about practical use of GPG. Apparently the key that's used for signing is rotated yearly, and the latest is available here:
https://dl.google.com/dl/linux/linux_signing_key.pub

Are there any questions or concerns?

If there's demand, I'll work on signing every binary, not just linux-amd64.tar.gz

edit: ah, I think the subkey (1397 BC53 640D B551) will rotate out, but not 7721 F63B D38B 4796?

[broady]

go1.9rc2 now has asc files for all the tar.gz files.

releases will be signed in this manner going forward. (msi and pkg installers continue to be signed)

[yogo1212]

@broady thank you very much!
is this documented anywhere but here?
maybe it could go here here?