Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve normalised vs original distribution names #97

Open
takluyver opened this issue Jan 14, 2022 · 0 comments
Open

Resolve normalised vs original distribution names #97

takluyver opened this issue Jan 14, 2022 · 0 comments
Labels
component: sources Related to WheelSource and friends type: task Something that needs to be done that is not a bug or feature

Comments

@takluyver
Copy link
Member

At present, the distribution name comes from the wheel filename, which can be normalised according to the new rule (PEP 503 normalisation, but with _ in place of -) or according to an older rule (any character except A-Za-z0-9. is converted to _).

If we want the original distribution name for anything, we should get that from wheel metadata. If we want the normalised name, we should normalise it ourselves for consistency.

@pradyunsg pradyunsg added the type: task Something that needs to be done that is not a bug or feature label Feb 17, 2022
mattoberle added a commit to mattoberle/rules_python that referenced this issue Jun 7, 2022
`pypa/installer` is used to parse Wheel metadata, but does not currently
provide a method for normalizing distribution names:

- pypa/installer#97

`pypa/pkg_resources` provides `Requirement.parse` which returns an instance
of `Requirement` where `.key` is the canonical distribution name per PEP 503.

The `Requirement` class can also parse `extras`, but it returns a normalized
form that I believe could break the installation of the extras.
alexeagle added a commit to bazelbuild/rules_python that referenced this issue Jun 21, 2022
* Use PEP 426 rules when setting deps from extras

This commit addresses issue #720.

[PEP 426](https://peps.python.org/pep-0426/#name) states that distribution
names are case-insensitive and "-" is interchangeable with "_".

The `pip-compile` command creates a lockfile where all package names are lowercase.
The tool may also modify interchangeable characters.

The following examples are all valid `requirements.txt` or `requirements_lock.txt` entries:

```
SQLAlchemy[postgresql_psycopg2binary]==1.4.36
sqlalchemy[postgresql_psycopg2binary]==1.4.36
sentry_sdk[flask]==1.5.8
sentry-sdk[flask]==1.5.8
```

A distribution's `METADATA` file contains the stylization chosen by the publisher.
By applying a "sanitise" function when building the `extras` dict and when performing lookups
we can eliminate this difference as a concern.

* Use PEP 503 rules when sanitising extras

* Normalize distribution name with pkg_resources

`pypa/installer` is used to parse Wheel metadata, but does not currently
provide a method for normalizing distribution names:

- pypa/installer#97

`pypa/pkg_resources` provides `Requirement.parse` which returns an instance
of `Requirement` where `.key` is the canonical distribution name per PEP 503.

The `Requirement` class can also parse `extras`, but it returns a normalized
form that I believe could break the installation of the extras.

* Use Requirement.parse to populate extra reqs

* Revert "Use Requirement.parse to populate extra reqs"

This reverts commit f0faa97.

* Test for distribution name normalization in extras

* Replace pkg_resources with packaging.utils

This replaces `pkg_resources.Requirement.parse` with
`packaging.utils.canonicalize_name`. Doing this pulls in a vendored
requirement from `pip`, which may be undesirable.

The code we want is just:

```
re.sub(r"[-_.]+", "-", name).lower()
```

This commit also leaves a reference to `pkg_resources` in `wheel.py` which
does not canonicalize the name.

Co-authored-by: Jonathon Belotti <jonathon@canva.com>
Co-authored-by: Alex Eagle <alex@aspect.dev>
mattem pushed a commit to mattem/rules_python that referenced this issue Jul 7, 2022
* Use PEP 426 rules when setting deps from extras

This commit addresses issue bazelbuild#720.

[PEP 426](https://peps.python.org/pep-0426/#name) states that distribution
names are case-insensitive and "-" is interchangeable with "_".

The `pip-compile` command creates a lockfile where all package names are lowercase.
The tool may also modify interchangeable characters.

The following examples are all valid `requirements.txt` or `requirements_lock.txt` entries:

```
SQLAlchemy[postgresql_psycopg2binary]==1.4.36
sqlalchemy[postgresql_psycopg2binary]==1.4.36
sentry_sdk[flask]==1.5.8
sentry-sdk[flask]==1.5.8
```

A distribution's `METADATA` file contains the stylization chosen by the publisher.
By applying a "sanitise" function when building the `extras` dict and when performing lookups
we can eliminate this difference as a concern.

* Use PEP 503 rules when sanitising extras

* Normalize distribution name with pkg_resources

`pypa/installer` is used to parse Wheel metadata, but does not currently
provide a method for normalizing distribution names:

- pypa/installer#97

`pypa/pkg_resources` provides `Requirement.parse` which returns an instance
of `Requirement` where `.key` is the canonical distribution name per PEP 503.

The `Requirement` class can also parse `extras`, but it returns a normalized
form that I believe could break the installation of the extras.

* Use Requirement.parse to populate extra reqs

* Revert "Use Requirement.parse to populate extra reqs"

This reverts commit f0faa97.

* Test for distribution name normalization in extras

* Replace pkg_resources with packaging.utils

This replaces `pkg_resources.Requirement.parse` with
`packaging.utils.canonicalize_name`. Doing this pulls in a vendored
requirement from `pip`, which may be undesirable.

The code we want is just:

```
re.sub(r"[-_.]+", "-", name).lower()
```

This commit also leaves a reference to `pkg_resources` in `wheel.py` which
does not canonicalize the name.

Co-authored-by: Jonathon Belotti <jonathon@canva.com>
Co-authored-by: Alex Eagle <alex@aspect.dev>
stefanor added a commit to stefanor/installer that referenced this issue Aug 15, 2022
Allowing us to unpack wheels with non-normalized filenames.

Part of pypa#97, but doesn't retrieve the names from distribution metadata,
yet.
stefanor added a commit to stefanor/installer that referenced this issue Aug 15, 2022
Allowing us to unpack wheels with non-normalized filenames.

Part of pypa#97, but doesn't retrieve the names from distribution metadata,
yet.

Fixes: pypa#134
stefanor added a commit to stefanor/installer that referenced this issue Aug 15, 2022
Allowing us to unpack wheels with non-normalized filenames.

Part of pypa#97, but doesn't retrieve the names from distribution metadata,
yet.

Fixes: pypa#134
@pradyunsg pradyunsg added the component: sources Related to WheelSource and friends label Dec 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
component: sources Related to WheelSource and friends type: task Something that needs to be done that is not a bug or feature
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants