Verify Release URLs using publish attestations

[facutuesca]

This PR adds a new field to Release, trusted_publisher_url, which stores the publisher URL used to create that release.

More specifically, this field is only populated when the first file uploaded (the one that creates the release) is uploaded using Trusted Publishing and has a PEP-740 publish attestation.

This allows PyPI to verify a release's URLs (the ones returned by the Release.urls property), by comparing them with the trusted_publisher_url. This means we put those URLs in the Verified details section of the project page:

(compare with the mockup provided in this comment: pypi#8635 (comment))

This is an alternative implementation to these two PRs:
pypi#15862
pypi#15891

cc @woodruffw

[woodruffw]

Nitpick: might be good to have some ActiveState URL examples in here as well 🙂

[facutuesca]

Done!

[woodruffw]

NB: This doesn't do case normalization or anything else, which is probably fine but might be worth calling out in a comment.

[facutuesca]

With the new implementation, now we do normalization. I added a detailed docstring to document the behavior

[facutuesca]

@woodruffw While looking into URL normalization, I realized this implementation has a pretty bad security bug:

publisher_url = "https://github.com/org/project"
release_url = "https://github.com/org/project22"
verify_url(release_url) == True  # because release_url.startswith(publisher_url) is True

Which led me to look into urllib to see if we can parse and compare URL elements. But urllib doesn't do any validation on inputs, which means there's no guarantee a malicious input will not parse as a valid URL that matches the publisher URL.

Any ideas on how we can tackle URL comparison here?
edit: Would something like this be enough?:

    def verify_url(self, url: str) -> bool:
        if not self.trusted_publisher_url:
            return False
        return (url == self.trusted_publisher_url or 
                url.startswith(self.trusted_publisher_url + '/'))  # if checking with startswith, ensure the character after the publisher URL is a forward slash

[woodruffw]

Which led me to look into urllib to see if we can parse and compare URL elements. But urllib doesn't do any validation on inputs, which means there's no guarantee a malicious input will not parse as a valid URL that matches the publisher URL.

Yeah, I think we should avoid urllib entirely for this, and use a more standards-conformant URL parser that offers strong validation. One good candidate is rfc3986 -- technically that'll be stricter than WHATWG requires for URL validation, but this shouldn't be an issue in practice (and when it is, it'll be a good validation check).

TL;DR: I think we should use rfc3986 + test the bejeezus out of it, including negative and backstop cases 🙂

[facutuesca]

TL;DR: I think we should use rfc3986 + test the bejeezus out of it, including negative and backstop cases 🙂

@woodruffw Done! The Release.verify_url now does more exhaustive checks, and I added several more tests for the error cases I could think of.

[woodruffw]

Nitpick: api is a pretty generic import, so maybe we do import rfc3986.api instead and refer to things with that fully qualified path 🙂

[facutuesca]

fixed

[woodruffw]

NB: Probably need to check what happens when either URL doesn't have a path component; I suspect path will be None in that case 🙂

[facutuesca]

Good catch! Added a check, and a test

[woodruffw]

A testcase for the inverse (TP has path, expected has no path) would also be good! And same for both having no path.

[facutuesca]

Done!

[facutuesca]

Superseeded by pypi#16205