[BUG] `pyproject.toml`: Unicode character in a license file prevents dependencies from being installed

[yurinnick]

Description

Some licenses, in this specific example, LGPL-2.1, contains Unicode characters. In case of LGPL 2.1 it is U+000c - Form Feed character.

In pyproject.toml having a license file with those characters silently breaks package dependencies installation.

Expected behavior

• pip install -e . installs package dependencies
• pip install -e .[dev] installs optional dev dependencies group

pip version

23.2.1

Python version

3.10, 3.11

OS

Fedora Linux 38

How to Reproduce

1. Create pyproject.toml

[project]
name = "unicode-bug"
version = "0.1.0"
requires-python = ">=3.10"
license = {file = "LICENSE"}
dependencies = [
    "requests"
]
[project.optional-dependencies]
dev = [
  "pytest"
]

2. Create LICENSE file with unicode character

$ echo -e '\U000b' > LICENSE

3. Try to install the package and optional dependencies - doesn't install dependencies

$ pip install -e .
...
Successfully built unicode-bug
Installing collected packages: unicode-bug
  Attempting uninstall: unicode-bug
    Found existing installation: unicode-bug 0.1.0
    Uninstalling unicode-bug-0.1.0:
      Successfully uninstalled unicode-bug-0.1.0
Successfully installed unicode-bug-0.1.0

$ pip install -e .[dev]
...
WARNING: unicode-bug 0.1.0 does not provide the extra 'dev'
...

4. Remove unicode character

$ echo "" > LICENSE

3. Try to install the package and optional dependencies - installation works

$ pip install -e .
...
Successfully built unicode-bug
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests, unicode-bug
  Attempting uninstall: unicode-bug
    Found existing installation: unicode-bug 0.1.0
    Uninstalling unicode-bug-0.1.0:
      Successfully uninstalled unicode-bug-0.1.0
Successfully installed certifi-2023.7.22 charset-normalizer-3.2.0 idna-3.4 requests-2.31.0 unicode-bug-0.1.0 urllib3-2.0.4

$ pip install -e .[dev]
Successfully built unicode-bug
Installing collected packages: pluggy, packaging, iniconfig, unicode-bug, pytest
  Attempting uninstall: unicode-bug
    Found existing installation: unicode-bug 0.1.0
    Uninstalling unicode-bug-0.1.0:
      Successfully uninstalled unicode-bug-0.1.0
Successfully installed iniconfig-2.0.0 packaging-23.1 pluggy-1.3.0 pytest-7.4.0 unicode-bug-0.1.0

Output

No response

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[RonnyPfannschmidt]

What's the output when using a valid build-system section, and what warnings get shown when using build to make the wheel?

[yurinnick]

Add build-system section:

...
[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

Install and build the wheel with broken license file:

$ pip install -e .
Obtaining file:///home/yurinnick/Documents/test
  Installing build dependencies ... done
  Checking if build backend supports build_editable ... done
  Getting requirements to build editable ... done
  Preparing editable metadata (pyproject.toml) ... done
Building wheels for collected packages: unicode-bug
  Building editable for unicode-bug (pyproject.toml) ... done
  Created wheel for unicode-bug: filename=unicode_bug-0.1.0-0.editable-py3-none-any.whl size=2922 sha256=4277b75c0ebbf7978dc5a79089f9d5b8cce84d2179e10aa837c2069139ff977d
  Stored in directory: /tmp/pip-ephem-wheel-cache-6sqe4pu4/wheels/5e/bb/78/14fd4ec013baed754dd266aec43868a061011260e628caafc4
Successfully built unicode-bug
Installing collected packages: unicode-bug
  Attempting uninstall: unicode-bug
    Found existing installation: unicode-bug 0.1.0
    Uninstalling unicode-bug-0.1.0:
      Successfully uninstalled unicode-bug-0.1.0
Successfully installed unicode-bug-0.1.0

$ pip wheel .
Processing /home/yurinnick/Documents/test
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: unicode-bug
  Building wheel for unicode-bug (pyproject.toml) ... done
  Created wheel for unicode-bug: filename=unicode_bug-0.1.0-py3-none-any.whl size=1425 sha256=ff6fb74aa5124491036b8ab8f8788910f7a6ba9b5d65285db8568f45e99975a9
  Stored in directory: /tmp/pip-ephem-wheel-cache-mk_5fqz9/wheels/5e/bb/78/14fd4ec013baed754dd266aec43868a061011260e628caafc4
Successfully built unicode-bug

So no new warnings or errors as far as I can tell.

[RonnyPfannschmidt]

I meant the build package,which has easier opt out of build output hiding

[yurinnick]

python -m build yields this output: https://gist.github.com/yurinnick/bf06c9d0f97f9c361b38038f5fe8e072

[pradyunsg]

FWIW, omitting build-system is definitely valid, and passing -v to pip will present the build output. There is no need to use build for it.

[pfmoore]

Works fine for me on Windows:

❯ pip install -vvv -e ".[dev]"
Using pip 23.2.1 from C:\Work\Scratch\foof\.venv\Lib\site-packages\pip (python 3.11)
Non-user install because user site-packages disabled
...
Successfully installed certifi-2023.7.22 charset-normalizer-3.2.0 colorama-0.4.6 idna-3.4 iniconfig-2.0.0 packaging-23.1 pluggy-1.3.0 pytest-7.4.0 requests-2.31.0 unicode-bug-0.1.0 urllib3-2.0.4
Remote version of pip: 23.2.1
Local version of pip:  23.2.1
Was pip installed by pip? True
Removed build tracker: 'C:\\Users\\xxx\\AppData\\Local\\Temp\\pip-build-tracker-7gbkreko'

❯ pip list
Package            Version   Editable project location
------------------ --------- -------------------------
certifi            2023.7.22
charset-normalizer 3.2.0
colorama           0.4.6
idna               3.4
iniconfig          2.0.0
packaging          23.1
pip                23.2.1
pluggy             1.3.0
pytest             7.4.0
requests           2.31.0
unicode-bug        0.1.0     C:\Work\Scratch\foof
urllib3            2.0.4

(note pytest is installed, so the dev extra was recognised).

[yurinnick]

Verbose pip install output: https://gist.github.com/yurinnick/9128d68e5f9be9b0b50362122d79346c

[yurinnick]

I did some research and my findings so far:

• The problem should be somewhere in the metadata parsing logic in pip._internal.metadata.importlib._dists.Distribution class
• Distribution loaded from unicode_bug.egg_info directory has valid metadata [1]
• Discribution loaded from temporary /tmp/pip-modern-metadata-0q88s1hj/unicode_bug-0.1.0.dist-info has invalid metadata [2]

[1]

>>> d.metadata.values()
['2.1', 'unicode-bug', '0.1.0', '\n        ', '>=3.10', 'dev', 'LICENSE', 'requests', 'pytest ; extra == "dev"']
>>> d.metadata.keys()
['Metadata-Version', 'Name', 'Version', 'License', 'Requires-Python', 'Provides-Extra', 'License-File', 'Requires-Dist', 'Requires-Dist']

[2]

[('Metadata-Version', '2.1'), ('Name', 'unicode-bug'), ('Version', '0.1.0'), ('License', ''), ('Description', "        \nRequires-Python: >=3.10\nLicense-File: LICENSE\nRequires-Dist: requests\nProvides-Extra: dev\nRequires-Dist: pytest ; extra == 'dev'\n\n")]

[uranusjr]

This sounds like a setuptools bug to me. The dist-info directory in the temporary directory is generated by the build backend (setuptools) and pip only reads the result.

[pradyunsg]

This is definitely a setuptools bug. Transferring the issue over. :)

[abravalheri]

Hi @yurinnick thank you very much for reporting, I will have a look on this.

Distribution loaded from unicode_bug.egg_info directory has valid metadata [1]\nDiscribution loaded from temporary /tmp/pip-modern-metadata-0q88s1hj/unicode_bug-0.1.0.dist-info has invalid metadata [2]

This is interesting. Is the PKG-INFO file in the top of the sdist OK? I had a quick view in the code and it seems that we do use UTF-8 to read the file pointed out in pyproject.toml. And then we use UTF-8 again to write PKG-INFO.

One thing that might be related is that setuptools produces PKG-INFO files and those are later transformed into METADATA by pypa/wheel. But then pypa/wheel also seems to deal with the files using UTF-8.

The code contains some places that we use email.message_from_file (via distutils) to read PKG-INFO when building from the sdist, this might be where UTF-8 is not explicitly used. But I am not sure this is used, I need to investigate better.

If you build the project skipping the sdist step (e.g. python -m build --wheel) is the metadata file still ill-formed?

[yurinnick]

PKG-INFO in egg-info directory has the unicode character, but gets parsed correctly.
LICENSE file from dist-info directory also contains unicode, but METADATA file doesn't.

Here are the files in the hex form: https://gist.github.com/yurinnick/94fc3d9b19e0bf6270f7dd332f4b0e61

[pfmoore]

One thing confuses me. You keep referring to a "unicode character", but the reproducer you gave (which, as I said, works fine for me on Windows) uses the character \U000b, which is a perfectly ordinary ASCII character - 'LINE TABULATION' (U+000B).

Is the problem here caused by the fact that this character is whitespace, and as such is getting stripped at some point due to something removing trailing whitespace from lines?