GPG Signatures in the Warehouse UI

[mschwager]

Hey all,

I'm going to mostly copy from an issue raised elsewhere (pypi/legacy#762):

First off, congrats on the move to the new PyPI, the new UI and workflows are so much better now!

I'm curious why GPG functionality was removed from PyPI? I'll concede that it was sparingly used within the Python community before, but the functionality was available to those who wanted it. Speaking to your point:

... an attacker who uploaded the package can just as easily have uploaded a signature with a key they control.

This is true, but someone attempting to verify a PyPI package would notice that the key used to sign the package has changed. I'm assuming that we're talking about a subsequent verification here, or an upgrade, and not the first verification of the package. If that's the case, then yes, we'll be assuming trust on first use.

My idea of a typical workflow with PyPI GPG signatures:

1. Visit Github page, or README of project, and obtain the GPG public key used for the Python package. Note that we can't use key information from an already downloaded package since, in the above attacker scenario, the package could be compromised. Essentially, we somehow obtain the public key out-of-band.
2. Import the GPG to our internal key chain.
3. Download Python package and GPG signature from PyPI.
4. Verify Python package against the GPG signature using the previously obtained key.
5. Repeat steps 3-4 when upgrading a package.

Now, step 3 was much easier in the previous PyPI service, as the GPG keys were in plain view when looking at a package release.

Note that we're protecting against compromises in steps 3-5 in this scenario, as you mentioned an attacker can upload a package and corresponding signature. Protecting against compromise in steps 1-2 is a different story :)

So, I guess I'm curious about your thoughts on the above threat model, and why GPG functionality was removed from PyPI.

Thanks!

This discussion also provides some interesting points regarding GPG signatures and PyPI: https://mail.python.org/pipermail/distutils-sig/2016-May/028933.html

I'm definitely interested in discussing this further, and figuring out how we can improve the security of packages in the new Warehouse!

[meejah]

Maybe this bug is the "relevant one"? I see that the new (and very nice looking!) PyPI UI no longer even lists links to signature files for releases that have them. I know random GPG signatures aren't a panacea, but also I know at least one of my users (the Debian project) does use and check them properly, so it would be very nice if these links could appear again.

You can get the signatures if you know they're there by appending .asc to files but this is .. non-ideal.

Two projects that definitely have signatures but aren't showing up in the Web UI are: Tahoe-LAFS and txtorcon.

[meejah]

To clarify my point above: some of my users know what key to check for txtorcon releases. How are they supposed to get signature files if Warehouse doesn't serve them?

The end-game here is that every Python package that cares to sign their releases then has to put signatures "somewhere else" and every consumer has to customize their (already existing) tools to go and find those signatures. This seems like a step backwards.

[dstufft]

They're still being served, they're just not being linked to from the UI.

[meejah]

Cool, I do see that. Is there a good reason not to link them in the UI, though?

(Also just to clarify, when I twine upload a new release with signatures, those will be served as well?)

[hellais]

(Also just to clarify, when I twine upload a new release with signatures, those will be served as well?)

Yes I confirmed that this does happen with some packages I maintain.

Here is a PR that exposes a download link for the signature if a package has it: #4314

[dstufft]

Hey. Sorry I didn't see the ask from May 18th and so missed responding to this.

The basic tl;dr here is that we don't believe that gpg to be sufficient for package signatures on a repository like PyPI. I went into a lot more detail on my blog, but the basic problem is that the Python packaging tool chain does not provide any sort of mechanism for determining what keys to trust for a specific package, which means that the signatures are largely theater unless someone goes and does extra work to determine the list of valid keys for their project.

In my approximation, the signature files uploaded to PyPI are almost entirely used by downstream distributors, primarily debian. In Debian's case, they have a redirector written on top of PyPI that exposes this information already (see for example, http://pypi.debian.net/txtorcon) which is likely going to be a better workflow for them anyways (it even includes a link to generate the watch file with the relevant pgpsigurlmangle directive). There may be other distros that need this, and my recommendation would be for them to have something similar.

Quite frankly, I'm pretty down on the idea that the signatures provide value even for downstream distributors. My experience with them is that is that distros have a high likelihood of just blindly accepting a new key and/or disabling signature verification when keys change. Obviously that is going to vary based on who is actually doing the packaging, but the fact that there is a non trivial amount of cases where that is happening suggests that the value provided isn't super high there either.

Obviously we still allow uploading the signatures, and will still serve them if uploaded. This is done so that downstream distributors that really do want it can continue to have it, however we explicitly chose not to expose this in the UI, largely because a lot of people felt compelled to upload signatures because they felt vaguely like they should get that signature URL to show up on their page, even if nobody was actually using it. It also provided consumers with a false sense of security, I know of multiple people who preferred to pip install ... signed packages because they thought they were more secure, even though pip doesn't verify them (and we don't have a trust model that we could use to verify them if we wanted to). This ended up effectively guilting OSS developers into doing extra work for, in my opinion, little to no extra benefit.

Thus we removed them from the UI to de-emphasize the fact that they exist, leaving exposing them up to more specialized UIs/tools that can be layered on top of PyPI and I'm not really convinced that this isn't a perfectly acceptable state to be in currently.

Please note that this doesn't mean that we're against package signing in general. There are plans to add package signing which will actually be generally useful that will come with a trust model, will be able to be mandated (ideally), and pip etc will also be able to validate it. We just have not yet gotten to that point. That desire also influences the decision to de-emphasize gpg signatures here, since having two signature mechanisms exposed would be more, not less confusing!

So with all of that, I'm going to close this issue, and the related PR, because I don't think that this functionality is worth it nor does it fit into the overall vision for where we want to take PyPI into the future.

[nlhkabu]

Just noting that this was also a featured request in the 'buy a feature' design research I conducted. Maybe it is worth explaining our position on this on our help page?

[LucidOne]

Has the API endpoint been disabled for uploading signatures? Everything seems to provide as little information as possible about why signature uploads are failing.