Resolve normalised vs original distribution names #97
Labels
component: sources
Related to WheelSource and friends
type: task
Something that needs to be done that is not a bug or feature
Comments
pradyunsg
added
the
type: task
mattoberle
added a commit
to mattoberle/rules_python
that referenced
this issue
Jun 7, 2022
`pypa/installer` is used to parse Wheel metadata, but does not currently provide a method for normalizing distribution names: - pypa/installer#97 `pypa/pkg_resources` provides `Requirement.parse` which returns an instance of `Requirement` where `.key` is the canonical distribution name per PEP 503. The `Requirement` class can also parse `extras`, but it returns a normalized form that I believe could break the installation of the extras.
12 tasks
alexeagle
added a commit
to bazelbuild/rules_python
that referenced
this issue
Jun 21, 2022
* Use PEP 426 rules when setting deps from extras This commit addresses issue #720. [PEP 426](https://peps.python.org/pep-0426/#name) states that distribution names are case-insensitive and "-" is interchangeable with "_". The `pip-compile` command creates a lockfile where all package names are lowercase. The tool may also modify interchangeable characters. The following examples are all valid `requirements.txt` or `requirements_lock.txt` entries: ``` SQLAlchemy[postgresql_psycopg2binary]==1.4.36 sqlalchemy[postgresql_psycopg2binary]==1.4.36 sentry_sdk[flask]==1.5.8 sentry-sdk[flask]==1.5.8 ``` A distribution's `METADATA` file contains the stylization chosen by the publisher. By applying a "sanitise" function when building the `extras` dict and when performing lookups we can eliminate this difference as a concern. * Use PEP 503 rules when sanitising extras * Normalize distribution name with pkg_resources `pypa/installer` is used to parse Wheel metadata, but does not currently provide a method for normalizing distribution names: - pypa/installer#97 `pypa/pkg_resources` provides `Requirement.parse` which returns an instance of `Requirement` where `.key` is the canonical distribution name per PEP 503. The `Requirement` class can also parse `extras`, but it returns a normalized form that I believe could break the installation of the extras. * Use Requirement.parse to populate extra reqs * Revert "Use Requirement.parse to populate extra reqs" This reverts commit f0faa97. * Test for distribution name normalization in extras * Replace pkg_resources with packaging.utils This replaces `pkg_resources.Requirement.parse` with `packaging.utils.canonicalize_name`. Doing this pulls in a vendored requirement from `pip`, which may be undesirable. The code we want is just: ``` re.sub(r"[-_.]+", "-", name).lower() ``` This commit also leaves a reference to `pkg_resources` in `wheel.py` which does not canonicalize the name. Co-authored-by: Jonathon Belotti <jonathon@canva.com> Co-authored-by: Alex Eagle <alex@aspect.dev>
mattem
pushed a commit
to mattem/rules_python
that referenced
this issue
Jul 7, 2022
* Use PEP 426 rules when setting deps from extras This commit addresses issue bazelbuild#720. [PEP 426](https://peps.python.org/pep-0426/#name) states that distribution names are case-insensitive and "-" is interchangeable with "_". The `pip-compile` command creates a lockfile where all package names are lowercase. The tool may also modify interchangeable characters. The following examples are all valid `requirements.txt` or `requirements_lock.txt` entries: ``` SQLAlchemy[postgresql_psycopg2binary]==1.4.36 sqlalchemy[postgresql_psycopg2binary]==1.4.36 sentry_sdk[flask]==1.5.8 sentry-sdk[flask]==1.5.8 ``` A distribution's `METADATA` file contains the stylization chosen by the publisher. By applying a "sanitise" function when building the `extras` dict and when performing lookups we can eliminate this difference as a concern. * Use PEP 503 rules when sanitising extras * Normalize distribution name with pkg_resources `pypa/installer` is used to parse Wheel metadata, but does not currently provide a method for normalizing distribution names: - pypa/installer#97 `pypa/pkg_resources` provides `Requirement.parse` which returns an instance of `Requirement` where `.key` is the canonical distribution name per PEP 503. The `Requirement` class can also parse `extras`, but it returns a normalized form that I believe could break the installation of the extras. * Use Requirement.parse to populate extra reqs * Revert "Use Requirement.parse to populate extra reqs" This reverts commit f0faa97. * Test for distribution name normalization in extras * Replace pkg_resources with packaging.utils This replaces `pkg_resources.Requirement.parse` with `packaging.utils.canonicalize_name`. Doing this pulls in a vendored requirement from `pip`, which may be undesirable. The code we want is just: ``` re.sub(r"[-_.]+", "-", name).lower() ``` This commit also leaves a reference to `pkg_resources` in `wheel.py` which does not canonicalize the name. Co-authored-by: Jonathon Belotti <jonathon@canva.com> Co-authored-by: Alex Eagle <alex@aspect.dev>
stefanor
added a commit
to stefanor/installer
that referenced
this issue
Aug 15, 2022
Allowing us to unpack wheels with non-normalized filenames. Part of pypa#97, but doesn't retrieve the names from distribution metadata, yet.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
At present, the distribution name comes from the wheel filename, which can be normalised according to the new rule (PEP 503 normalisation, but with
_in place of-) or according to an older rule (any character exceptA-Za-z0-9.is converted to_).If we want the original distribution name for anything, we should get that from wheel metadata. If we want the normalised name, we should normalise it ourselves for consistency.
The text was updated successfully, but these errors were encountered: