Pip ignores extras_require when description in setup.py contains \n #7010

OK-UNDP · 2019-09-13T01:20:52Z

Environment

pip version: 19.2.3
Python version: 3.6.8
Poetry version 1.0.0b1

Description
This issue was discovered while publishing docs from undp/MsGraphExporter on ReadTheDocs. When pyproject.toml option description has a multi-line string enclosed in """ (which is perfectly legit for a string in TOML), it causes poetry to build the package with the description parameter in the setup.py file containing string like 'this is\na short description' which apparently causes pip install pkg.whl[docs] to fail picking up extras_require option.

Expected behavior
Having \n in long_description of setup_kwargs in setup.py does not cause such issue. So, I would expect having it in description should not affect pip either.

How to re-create the issue

Have the description in the pyproject.toml as a multi-line value enclosed in """.

# ...snip...
[tool.poetry]
name = "ms-graph-exporter"
version = "0.1.0-rc.1"
description = """A distributed Celery application to export time-domain data
periodically from Microsoft Graph API into a buffer key-value store."""
# ...snip...

Check pyproject.toml syntax with poery.
```
$ poetry check
All set!
```

Build the package with poetry

$ poetry build
Building ms-graph-exporter (0.1.0-rc.1)
- Building sdist
- Built ms-graph-exporter-0.1.0rc1.tar.gz

- Building wheel
- Built ms_graph_exporter-0.1.0rc1-py3-none-any.whl

Check setup.py from the package tar.gz to ensure it has all extras_require defined and also has \n in the description.

# -*- coding: utf-8 -*-
from distutils.core import setup

packages = \
['ms_graph_exporter', 'ms_graph_exporter.celery', 'ms_graph_exporter.ms_graph']

package_data = \
{'': ['*']}

install_requires = \
['adal>=1.2,<2.0',
# ...snip...
'typing>=3.7,<4.0']

extras_require = \
{'code-format': ['black>=19.3b0,<20.0', 'blacken-docs>=0.5.0,<0.6.0'],
'code-lint': ['flake8>=3.7,<4.0',
# ...snip...
              'pygments>=2.4,<3.0'],
'docs': ['recommonmark>=0.5.0,<0.6.0',
          'sphinx>=2.0,<3.0',
          'sphinx_rtd_theme>=0.4.3,<0.5.0',
          'sphinx-autodoc-typehints>=1.7,<2.0'],
'test': ['pytest>=5.0.1,<6.0.0',
# ...snip...
          'pytest-variables[yaml]>=1.7,<2.0']}

setup_kwargs = {
    'name': 'ms-graph-exporter',
    'version': '0.1.0rc1',
    'description': 'A distributed Celery application to export time-domain data\nperiodically from Microsoft Graph API into a buffer key-value store.',
    # ...snip...
    'extras_require': extras_require,
    'python_requires': '>=3.6,<4.0',
}


setup(**setup_kwargs)

Try to install docs extra requirements with the package.

$ pip install ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl[docs]
Processing ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl
WARNING: ms-graph-exporter 0.1.0rc1 does not provide the extra 'docs'
Installing collected packages: ms-graph-exporter
Successfully installed ms-graph-exporter-0.1.0rc1

How to resolve the issue

Ensure pyproject.toml has a single-line description.

# ...snip...
[tool.poetry]
name = "ms-graph-exporter"
version = "0.1.0-rc.1"
description = "A distributed Celery application to export time-domain data periodically from Microsoft Graph API into a buffer key-value store."
# ...snip...

Check pyproject.toml syntax with poery.
```
$ poetry check
All set!
```

Build the package with poetry

$ poetry build
Building ms-graph-exporter (0.1.0-rc.1)
- Building sdist
- Built ms-graph-exporter-0.1.0rc1.tar.gz

- Building wheel
- Built ms_graph_exporter-0.1.0rc1-py3-none-any.whl

Check setup.py from the package tar.gz to ensure it has all extras_require defined and description without \n.

# -*- coding: utf-8 -*-
from distutils.core import setup

packages = \
['ms_graph_exporter', 'ms_graph_exporter.celery', 'ms_graph_exporter.ms_graph']

package_data = \
{'': ['*']}

install_requires = \
['adal>=1.2,<2.0',
# ...snip...
'typing>=3.7,<4.0']

extras_require = \
{'code-format': ['black>=19.3b0,<20.0', 'blacken-docs>=0.5.0,<0.6.0'],
'code-lint': ['flake8>=3.7,<4.0',
# ...snip...
              'pygments>=2.4,<3.0'],
'docs': ['recommonmark>=0.5.0,<0.6.0',
          'sphinx>=2.0,<3.0',
          'sphinx_rtd_theme>=0.4.3,<0.5.0',
          'sphinx-autodoc-typehints>=1.7,<2.0'],
'test': ['pytest>=5.0.1,<6.0.0',
# ...snip...
          'pytest-variables[yaml]>=1.7,<2.0']}

setup_kwargs = {
    'name': 'ms-graph-exporter',
    'version': '0.1.0rc1',
    'description': 'A distributed Celery application to export time-domain data periodically from Microsoft Graph API into a buffer key-value store.',
    # ...snip...
    'extras_require': extras_require,
    'python_requires': '>=3.6,<4.0',
}


setup(**setup_kwargs)

Install docs extra requirements with the package without any issue.

$ pip install ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl[docs]
Processing ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl
[... snip ...]
Successfully installed Jinja2-2.10.1 MarkupSafe-1.1.1 PyJWT-1.7.1 Pygments-2.4.2
adal-1.2.2 alabaster-0.7.12 amqp-2.5.1 asn1crypto-0.24.0 attrs-19.1.0 babel-2.7.0
billiard-3.6.1.0 celery-4.3.0 celery-redbeat-0.13.0 certifi-2019.9.11 cffi-1.12.3
chardet-3.0.4 commonmark-0.9.0 cryptography-2.7 docutils-0.15.2 future-0.17.1 gevent-1.4.0
greenlet-0.4.15 idna-2.8 imagesize-1.1.0 importlib-metadata-0.22 kombu-4.6.4
more-itertools-7.2.0 ms-graph-exporter-0.1.0rc1 packaging-19.1 pycparser-2.19
pyparsing-2.4.2 python-dateutil-2.8.0 pytz-2019.2 pyyaml-3.13 recommonmark-0.5.0
redis-3.3.8 requests-2.22.0 six-1.12.0 snowballstemmer-1.9.1 sphinx-2.2.0
sphinx-autodoc-typehints-1.8.0 sphinx-rtd-theme-0.4.3 sphinxcontrib-applehelp-1.0.1
sphinxcontrib-devhelp-1.0.1 sphinxcontrib-htmlhelp-1.0.2 sphinxcontrib-jsmath-1.0.1
sphinxcontrib-qthelp-1.0.2 sphinxcontrib-serializinghtml-1.1.3 tenacity-5.1.1
typing-3.7.4.1 urllib3-1.25.3 vine-1.3.0 zipp-0.6.0

Originally posted by @OK-UNDP in readthedocs/readthedocs.org#6151 (comment)

The text was updated successfully, but these errors were encountered:

chrahunt · 2019-09-13T01:43:59Z

The description gets written to the "Summary" field in ms_graph_exporter-0.1.0rc1.dist-info/METADATA. The multi-line version looks like:

Metadata-Version: 2.1
Name: ms-graph-exporter
Version: 0.1.0rc1
Summary: A distributed Celery application to export time-domain data
periodically from Microsoft Graph API into a buffer key-value store.
Home-page: https://github.com/undp/MsGraphExporter
...

I don't think that the metadata is valid. Consider:

The "Summary" field is described as a "one-line summary of what the distribution does" ([ref])(https://packaging.python.org/specifications/core-metadata/#summary)
The only multi-line field within the metadata specification has specific instructions for how to encode it (see here)
Not only is a warning traced about the "missing" extra, but none of the dependencies are installed. I verified this locally after making sure that I hadn't already installed them. So there's a bigger issue parsing that file than just the extras getting missed.

After replacing the multi-line summary with an encoded version like

Metadata-Version: 2.1
Name: ms-graph-exporter
Version: 0.1.0rc1
Summary: A distributed Celery application to export time-domain data
       |periodically from Microsoft Graph API into a buffer key-value store.
Home-page: https://github.com/undp/MsGraphExporter
...

and re-zipping, the resulting wheel seems to be installed correctly (with dependencies and extras).

chrahunt · 2019-09-13T01:50:51Z

When building with the poetry-generated setup.py using setuptools (by extracting the sdist, deleting the pyproject.toml, and running pip wheel .), the generated wheel has a METADATA file with:

Version: 0.1.0rc1
Summary: A distributed Celery application to export time-domain data
Requires-Dist: adal (<2.0,>=1.2)

Truncating like this may be better than encoding like I did above, since any other tools may be expecting literally a single line. If you file an issue with Poetry you may want to mention it.

OK-UNDP · 2019-09-13T01:57:28Z

@chrahunt, thanks fort digging deeper into this! So, it seems I'd better file an issue with Poetry to truncate multi-line descriptions, or flagging it by running poetry check.

chrahunt · 2019-09-13T02:01:16Z

You did most of the work - it was very clear what to check. :) Thank you for that.

I'd better file an issue with Poetry to truncate multi-line descriptions, or flagging it by running poetry check.

Yup, that's what I would do.

chrahunt · 2019-09-13T03:37:12Z

I will close this now since the issue has been reported over in poetry. Thanks again for such a thorough report!

triage-new-issues bot added the S: needs triage Issues/PRs that need to be triaged label Sep 13, 2019

chrahunt added S: awaiting response Waiting for a response/more information type: support User Support labels Sep 13, 2019

triage-new-issues bot removed S: needs triage Issues/PRs that need to be triaged labels Sep 13, 2019

no-response bot removed the S: awaiting response Waiting for a response/more information label Sep 13, 2019

chrahunt added the resolution: wrong project Should be reported elsewhere label Sep 13, 2019

OK-UNDP mentioned this issue Sep 13, 2019

Poetry allows multiline descriptions which causes problems with pip installing extras python-poetry/poetry#1372

Closed

3 tasks

chrahunt closed this as completed Sep 13, 2019

lock bot added the auto-locked Outdated issues that have been locked by automation label Oct 13, 2019

lock bot locked as resolved and limited conversation to collaborators Oct 13, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pip ignores extras_require when description in setup.py contains \n #7010

Pip ignores extras_require when description in setup.py contains \n #7010

OK-UNDP commented Sep 13, 2019

chrahunt commented Sep 13, 2019

chrahunt commented Sep 13, 2019 •

edited

Loading

OK-UNDP commented Sep 13, 2019

chrahunt commented Sep 13, 2019

chrahunt commented Sep 13, 2019

Pip ignores extras_require when description in setup.py contains \n #7010

Pip ignores extras_require when description in setup.py contains \n #7010

Comments

OK-UNDP commented Sep 13, 2019

How to re-create the issue

How to resolve the issue

chrahunt commented Sep 13, 2019

chrahunt commented Sep 13, 2019 • edited Loading

OK-UNDP commented Sep 13, 2019

chrahunt commented Sep 13, 2019

chrahunt commented Sep 13, 2019

chrahunt commented Sep 13, 2019 •

edited

Loading