Unable to build docs with poetry

[OK-UNDP]

Details

• Read the Docs project URL: MsGraphExporter

Expected Result

I have the V2 .readthedocs.yaml in a feature branch of my project that defines:

python:
  version: 3.6
  install:
    - method: pip
      path: .
      extra_requirements:
        - docs

Since I use poetry and extras defined in pyproject.toml, I would expect RTD to install [docs] extras and build to succeed.

Actual Result

Instead, builds for the feature branch version fail due to unavailable dependencies. Package install step shows this:

Processing /home/docs/checkouts/readthedocs.org/user_builds/msgraphexporter/checkouts/feature-readthedocs
  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
  Installing backend dependencies: finished with status 'done'
    Preparing wheel metadata: started
    Preparing wheel metadata: finished with status 'done'
  WARNING: ms-graph-exporter-0.1.0rc1 0.1.0rc1 does not provide the extra 'docs'
Building wheels for collected packages: ms-graph-exporter
  Building wheel for ms-graph-exporter (PEP 517): started
  Building wheel for ms-graph-exporter (PEP 517): finished with status 'done'
  Created wheel for ms-graph-exporter: filename=ms_graph_exporter-0.1.0rc1-cp36-none-any.whl size=51051 sha256=19b410ac5ffa8a917cadbd1856095833467b0fc1dc2639e17f209764671fff90
  Stored in directory: /home/docs/checkouts/readthedocs.org/user_builds/msgraphexporter/.cache/pip/wheels/89/c3/93/213500c1a1f7c203865c9318360deb9d465a6fb26cd651eebb
Successfully built ms-graph-exporter
Installing collected packages: ms-graph-exporter
Successfully installed ms-graph-exporter-0.1.0rc1

Perhaps I am doing something wrong, so I would be grateful if somebody could point me to the right direction.

[humitos]

Hi @OK-UNDP! Can you link your project URL and build URL that it's failing?

We do install the packages with pip not with poetry. Using extra_requirements: - docs will produce this command pip install .[docs].

There were some ideas to support poetry and pipfile (#3181) but considering the amount of work, we are not ready for it.

[stsewd]

@OK-UNDP please check #4912 (comment)

[OK-UNDP]

Hi @humitos, the latest build URL is this one. I also fixed the RTD project URL in the original post above.

Pip 19.2.3 docs mention regarding PEP-517 that:

Furthermore, from version 19.0 onwards, pip supports projects specifying the build backend they use in pyproject.toml, in the form described in PEP 517.

My pyproject.toml has the following backend declaration:

[build-system]
requires = ["poetry>=0.12"]
build-backend = "poetry.masonry.api"

However, when I do pip install .[docs] in my project folder locally I get the same error does not provide the extra 'docs'. Perhaps, there is a way to specify docs extras in some other place in pyproject.toml?

Hi @stsewd, you would have to spell it out for me. I have wiped the environment multiple times but it didn't help.

[stsewd]

@OK-UNDP can you try updating/recreating the poetry.lock file?

This repo works on rtd, and the lock file looks kind of different https://github.com/zalando-incubator/transformer/blob/master/poetry.lock (I'm not familiar with poetry)

[OK-UNDP]

@stsewd, I figured it out. Here is the issue, which is not related to RTD it seems (would be glad for suggestions where I should file it (pip? poetry?).

Issue Summary

The initial issues was that ReadTheDocs build environment was not able to build my docs saying that package does not have docs extra requirement defined.

Further troubleshooting identified that when pyproject.toml option description has a multi-line string enclosed in """ (which is perfectly legit for a string in TOML), it causes poetry to build the package with the description parameter in the setup.py file containing string like 'this is\na short description' which apparently causes pip install .[docs] to fail picking up extras_require option.

So, this is not the RTD's issue, but some intricacy of poetry and/or pip.

How to re-create the issue

1. Have the description in the pyproject.toml as a multi-line value enclosed in """.

   # ...snip...
   [tool.poetry]
   name = "ms-graph-exporter"
   version = "0.1.0-rc.1"
   description = """A distributed Celery application to export time-domain data
   periodically from Microsoft Graph API into a buffer key-value store."""
   # ...snip...

2. Check pyproject.toml syntax with poery.

   $ poetry check
   All set!

3. Build the package with poetry

   $ poetry build
   Building ms-graph-exporter (0.1.0-rc.1)
   - Building sdist
   - Built ms-graph-exporter-0.1.0rc1.tar.gz
   
   - Building wheel
   - Built ms_graph_exporter-0.1.0rc1-py3-none-any.whl

4. Check setup.py from the package tar.gz to ensure it has all extras_require defined and also has \n in the description.

   # -*- coding: utf-8 -*-
   from distutils.core import setup
   
   packages = \
   ['ms_graph_exporter', 'ms_graph_exporter.celery', 'ms_graph_exporter.ms_graph']
   
   package_data = \
   {'': ['*']}
   
   install_requires = \
   ['adal>=1.2,<2.0',
   # ...snip...
   'typing>=3.7,<4.0']
   
   extras_require = \
   {'code-format': ['black>=19.3b0,<20.0', 'blacken-docs>=0.5.0,<0.6.0'],
   'code-lint': ['flake8>=3.7,<4.0',
   # ...snip...
                 'pygments>=2.4,<3.0'],
   'docs': ['recommonmark>=0.5.0,<0.6.0',
             'sphinx>=2.0,<3.0',
             'sphinx_rtd_theme>=0.4.3,<0.5.0',
             'sphinx-autodoc-typehints>=1.7,<2.0'],
   'test': ['pytest>=5.0.1,<6.0.0',
   # ...snip...
             'pytest-variables[yaml]>=1.7,<2.0']}
   
   setup_kwargs = {
       'name': 'ms-graph-exporter',
       'version': '0.1.0rc1',
       'description': 'A distributed Celery application to export time-domain data\nperiodically from Microsoft Graph API into a buffer key-value store.',
       # ...snip...
       'extras_require': extras_require,
       'python_requires': '>=3.6,<4.0',
   }
   
   
   setup(**setup_kwargs)

5. Try to install docs extra requirements with the package.

   $ pip install ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl[docs]
   Processing ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl
   WARNING: ms-graph-exporter 0.1.0rc1 does not provide the extra 'docs'
   Installing collected packages: ms-graph-exporter
   Successfully installed ms-graph-exporter-0.1.0rc1

How to resolve the issue

1. Ensure pyproject.toml has a single-line description.

   # ...snip...
   [tool.poetry]
   name = "ms-graph-exporter"
   version = "0.1.0-rc.1"
   description = "A distributed Celery application to export time-domain data periodically from Microsoft Graph API into a buffer key-value store."
   # ...snip...

2. Check pyproject.toml syntax with poery.

   $ poetry check
   All set!

3. Build the package with poetry

   $ poetry build
   Building ms-graph-exporter (0.1.0-rc.1)
   - Building sdist
   - Built ms-graph-exporter-0.1.0rc1.tar.gz
   
   - Building wheel
   - Built ms_graph_exporter-0.1.0rc1-py3-none-any.whl

4. Check setup.py from the package tar.gz to ensure it has all extras_require defined and description without \n.

   # -*- coding: utf-8 -*-
   from distutils.core import setup
   
   packages = \
   ['ms_graph_exporter', 'ms_graph_exporter.celery', 'ms_graph_exporter.ms_graph']
   
   package_data = \
   {'': ['*']}
   
   install_requires = \
   ['adal>=1.2,<2.0',
   # ...snip...
   'typing>=3.7,<4.0']
   
   extras_require = \
   {'code-format': ['black>=19.3b0,<20.0', 'blacken-docs>=0.5.0,<0.6.0'],
   'code-lint': ['flake8>=3.7,<4.0',
   # ...snip...
                 'pygments>=2.4,<3.0'],
   'docs': ['recommonmark>=0.5.0,<0.6.0',
             'sphinx>=2.0,<3.0',
             'sphinx_rtd_theme>=0.4.3,<0.5.0',
             'sphinx-autodoc-typehints>=1.7,<2.0'],
   'test': ['pytest>=5.0.1,<6.0.0',
   # ...snip...
             'pytest-variables[yaml]>=1.7,<2.0']}
   
   setup_kwargs = {
       'name': 'ms-graph-exporter',
       'version': '0.1.0rc1',
       'description': 'A distributed Celery application to export time-domain data periodically from Microsoft Graph API into a buffer key-value store.',
       # ...snip...
       'extras_require': extras_require,
       'python_requires': '>=3.6,<4.0',
   }
   
   
   setup(**setup_kwargs)

5. Install docs extra requirements with the package without any issue.

   $ pip install ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl[docs]
   Processing ./dist/ms_graph_exporter-0.1.0rc1-py3-none-any.whl
   [... snip ...]
   Successfully installed Jinja2-2.10.1 MarkupSafe-1.1.1 PyJWT-1.7.1 Pygments-2.4.2
   adal-1.2.2 alabaster-0.7.12 amqp-2.5.1 asn1crypto-0.24.0 attrs-19.1.0 babel-2.7.0
   billiard-3.6.1.0 celery-4.3.0 celery-redbeat-0.13.0 certifi-2019.9.11 cffi-1.12.3
   chardet-3.0.4 commonmark-0.9.0 cryptography-2.7 docutils-0.15.2 future-0.17.1 gevent-1.4.0
   greenlet-0.4.15 idna-2.8 imagesize-1.1.0 importlib-metadata-0.22 kombu-4.6.4
   more-itertools-7.2.0 ms-graph-exporter-0.1.0rc1 packaging-19.1 pycparser-2.19
   pyparsing-2.4.2 python-dateutil-2.8.0 pytz-2019.2 pyyaml-3.13 recommonmark-0.5.0
   redis-3.3.8 requests-2.22.0 six-1.12.0 snowballstemmer-1.9.1 sphinx-2.2.0
   sphinx-autodoc-typehints-1.8.0 sphinx-rtd-theme-0.4.3 sphinxcontrib-applehelp-1.0.1
   sphinxcontrib-devhelp-1.0.1 sphinxcontrib-htmlhelp-1.0.2 sphinxcontrib-jsmath-1.0.1
   sphinxcontrib-qthelp-1.0.2 sphinxcontrib-serializinghtml-1.1.3 tenacity-5.1.1
   typing-3.7.4.1 urllib3-1.25.3 vine-1.3.0 zipp-0.6.0

[stsewd]

@OK-UNDP wow, thanks for so descriptive solution :)

I think you should check if the setup.py file generated by poetry doesn't have any syntax errors or missing something that would make python omit the next values. If it's valid, you should report to pip. Also, you could try with any project that uses a setup.py file and put a new line and see if it breaks.