Poetry build does not install build dependencies (affects both install and build)

[mdgoldberg]

• I am on the latest Poetry version.
• I have searched the issues of this repo and believe that this is not a duplicate.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: OS X 10.15.5 (also tested in docker on ubuntu 16.04)
• Poetry version: 1.1.0b2
• Link of a Gist with the contents of your pyproject.toml file: https://gist.github.com/mdgoldberg/96a9955c8c823f2127dbc471a954b898

Issue

I created a repo with a minimal reproduction: https://github.com/mdgoldberg/poetry-cython-example

The problem here is that Cython isn't installed prior to building/installing the root package, even though Cython is specified in build-system.requires in the pyproject.toml. Here's how I am reproducing the issues, for both poetry install and poetry build:

$ git clone git@github.com:mdgoldberg/poetry-cython-example.git
$ cd poetry-cython-example
$ poetry install -vvv
Creating virtualenv poetry-cython-example in /Users/mattgoldberg/poetry-cython-example/.venv
Using virtualenv: /Users/mattgoldberg/poetry-cython-example/.venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Installing the current project: poetry-cython-example (0.1.0)
  - Building package poetry-cython-example in editable mode
  - Executing build script: build.py
Traceback (most recent call last):
  File "/Users/mattgoldberg/poetry-cython-example/build.py", line 6, in <module>
    from Cython.Build import cythonize
ModuleNotFoundError: No module named 'Cython'
  - Adding poetry_cython_example.pth to /Users/mattgoldberg/poetry-cython-example/.venv/lib/python3.6/site-packages for /Users/mattgoldberg/poetry-cython-example
  - Adding the poetry_cython_example-0.1.0.dist-info directory to /Users/mattgoldberg/poetry-cython-example/.venv/lib/python3.6/site-packages
$
$ poetry build -f wheel -vvv
Building poetry-cython-example (0.1.0)
  - Building wheel
Traceback (most recent call last):
  File "build.py", line 6, in <module>
    from Cython.Build import cythonize
ModuleNotFoundError: No module named 'Cython'

  CalledProcessError

  Command '['/Users/mattgoldberg/poetry-cython-example/.venv/bin/python', 'build.py']' returned non-zero exit status 1.

  at ~/.pyenv/versions/3.6.10/lib/python3.6/subprocess.py:311 in check_call
       307│     if retcode:
       308│         cmd = kwargs.get("args")
       309│         if cmd is None:
       310│             cmd = popenargs[0]
    →  311│         raise CalledProcessError(retcode, cmd)
       312│     return 0
       313│
       314│
       315│ def check_output(*popenargs, timeout=None, **kwargs):
$ # now install Cython and they work
$ pip install Cython
Collecting Cython
  Using cached Cython-0.29.21-cp36-cp36m-macosx_10_9_x86_64.whl (1.9 MB)
Installing collected packages: Cython
Successfully installed Cython-0.29.21
$ poetry install -vvv
Using virtualenv: /Users/mattgoldberg/poetry-cython-example/.venv
Installing dependencies from lock file

Finding the necessary packages for the current system

Installing the current project: poetry-cython-example (0.1.0)
  - Building package poetry-cython-example in editable mode
  - Executing build script: build.py
  - Adding poetry_cython_example.pth to /Users/mattgoldberg/poetry-cython-example/.venv/lib/python3.6/site-packages for /Users/mattgoldberg/poetry-cython-example
  - Adding the poetry_cython_example-0.1.0.dist-info directory to /Users/mattgoldberg/poetry-cython-example/.venv/lib/python3.6/site-packages
$ poetry build -f wheel -vvv
Building poetry-cython-example (0.1.0)
  - Building wheel
  - Built poetry_cython_example-0.1.0-cp36-cp36m-macosx_10_15_x86_64.whl

Note that for the fix, all that matters is whether Cython is installed in the "current" Python environment. If I install Cython globally but then activate the virtual env (which doesn't have Cython), then poetry install and poetry build still fail. Conversely, if I install Cython in the virtualenv and try to poetry install or poetry build from the directory but outside the virtualenv, it also fails.

Please, let me know if I'm misusing build.py or any other features, I know some of them may not be stabilized. For now, I'm just installing Cython before running poetry build or poetry install` in my Dockerfile, but it really seems like I shouldn't need to, per PEP 518. Let me know if I'm misunderstanding, and thank you for developing Poetry!

[abn]

@mdgoldberg can you give the branch in #2794 a try?

[mdgoldberg]

@abn Yeah that works, thank you! One thing is that it leaves the build dependencies installed after poetry build / poetry install completes. Is that intended behavior? My understanding is that build-system.requires is specifically for packages that are needed at build time but not runtime (if they were needed at runtime, they'd be included in tool.poetry.dependencies). Does it make sense to uninstall them after the build has succeeded/failed? This may sound trivial, but at least one benefit that would apply to me would be smaller docker image sizes.

[abn]

The decision to keep them installed was intentional. While, these dependencies are not required for runtime, it is still a development dependency of the package (as in, it is required for the development of the package). Without it, editable installs of the project would not be possible as that (poetry install) executes the build script, which in turn requires the dependency. Uninstalling and reinstalling every time would be rather inefficient.

As for the docker layer size, doubt the addition of these dependencies will blow up the size. If that is really a concern, I'd recommend building a wheel and installing that instead of having poetry manage the installation in the runtime container. Or do pip install /path/to/project to trigger a PEP 517 build and subsequent installation of the package.

[mdgoldberg]

@abn Yeah, that makes sense. Thanks for your help. When do you think the functionality in that PR will be available on a release accessible via poetry self update?

[abn]

No guarantees on that one unfortunately. I'd like to get feedback from the other maintainers regarding the approach I have taken first. It might require rework.

[mdgoldberg]

Sounds good. Definitely looking forward to that update (and 1.1 in general), for now I will just pip install Cython before building.

[espdev]

... for now I will just pip install Cython before building.

@mdgoldberg,

Unfortunately, this does not work if you want to publish and distribute your package. A user will need to install build dependencies for your package manually. It is not a good way.

For example, I need to publish a library with C++ pybind11 based extension modules. The library is distributed in prebuilt wheels for many platforms. I can add pybind11 to dev dependencies for development and building wheels. But also I want to provide sdist with source code, so that a user can install it from the source. And I don't want to require pybind11 dependency always. Only for installing from the sources. pybind11 is not required for prebuilt wheels. I need to build dependencies here, dev dependencies do not work in this case.

[tombh]

The PR still isn't in a public release right? I'm still seeing the Cython issue in the latest Poetry 1.1.12.

[clintonroy]

At least what's working for me is adding cython to tool.poetry.dev-dependencies. That shouldn't be necessary, i don't think, but it's kinda true at least.

[matteosantama]

At least what's working for me is adding cython to tool.poetry.dev-dependencies. That shouldn't be necessary, i don't think, but it's kinda true at least.

I would like to add that this workaround no longer works in 1.2.0a2. Hopefully the 1.2.x series will address this issue properly!

[ndevenish]

I've resigned myself to:

• Edit some runtime dependencies out of pyproject.toml (numpy wheels not available on platform and not needed for build)
• poetry install
• Revert the pyproject.toml (so that the package metadata is correct)
• poetry build

[ziyuang]

The PR still isn't in a public release right? I'm still seeing the Cython issue in the latest Poetry 1.1.12.

Still an issue in 1.6.1

[dimbleby]

Still an issue in 1.6.1

no it's not, as we can easily verify thanks to the repo with a reproduction provided at the top of this thread

[ziyuang]

Still an issue in 1.6.1

no it's not, as we can easily verify thanks to the repo with a reproduction provided at the top of this thread

Is build.py relevant when I add a package (like h5py) that requires Cython?
OK I was in fact installing deepface. After separately installing the failed packages and adjusting the version ranges of some dependencies, I can do poetry add deepface ultimately after a couple of rounds. Thanks.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.