Error in setup.py "No module named 'torch'" when installing with Poetry

[sisp]

When I try to install torch-sparse using Poetry, I'm getting the following error which occurs in setup.py:

ModuleNotFoundError: No module named 'torch'

The reason is that torch-sparse imports torch in setup.py while torch is not yet installed. Since those torch imports are only needed to build compiled extensions, it should be possible to avoid importing torch when installing the torch-sparse wheel package.

These are commands to reproduce the problem (tested using Poetry v1.1.7):

$ poetry init -n --python '^3.6.2' --dependency torch --dependency torch-sparse
$ poetry install
Creating virtualenv torch-sparse-poetry in /tmp/torch-sparse-poetry/.venv
Updating dependencies
Resolving dependencies... (77.1s)

Writing lock file

Package operations: 6 installs, 0 updates, 0 removals

  • Installing numpy (1.19.5)
  • Installing dataclasses (0.8)
  • Installing scipy (1.5.4)
  • Installing typing-extensions (3.10.0.0)
  • Installing torch (1.9.0)
  • Installing torch-sparse (0.6.11): Failed

  EnvCommandError

  Command ['/tmp/torch-sparse-poetry/.venv/bin/pip', 'install', '--no-deps', '$HOME/.cache/pypoetry/artifacts/59/cf/7b/23094d3d3aa79d571458529d8031882ce27d36db73083987acdab34868/torch_sparse-0.6.11.tar.gz'] errored with the following return code 1, and output: 
  Processing $HOME/.cache/pypoetry/artifacts/59/cf/7b/23094d3d3aa79d571458529d8031882ce27d36db73083987acdab34868/torch_sparse-0.6.11.tar.gz
      ERROR: Command errored out with exit status 1:
       command: /tmp/torch-sparse-poetry/.venv/bin/python -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-req-build-vk1oqsni/setup.py'"'"'; __file__='"'"'/tmp/pip-req-build-vk1oqsni/setup.py'"'"';f = getattr(tokenize, '"'"'open'"'"', open)(__file__) if os.path.exists(__file__) else io.StringIO('"'"'from setuptools import setup; setup()'"'"');code = f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-2tsa72d0
           cwd: /tmp/pip-req-build-vk1oqsni/
      Complete output (5 lines):
      Traceback (most recent call last):
        File "<string>", line 1, in <module>
        File "/tmp/pip-req-build-vk1oqsni/setup.py", line 8, in <module>
          import torch
      ModuleNotFoundError: No module named 'torch'
      ----------------------------------------
  WARNING: Discarding file://$HOME/.cache/pypoetry/artifacts/59/cf/7b/23094d3d3aa79d571458529d8031882ce27d36db73083987acdab34868/torch_sparse-0.6.11.tar.gz. Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  WARNING: You are using pip version 21.1.3; however, version 21.2.2 is available.
  You should consider upgrading via the '/tmp/torch-sparse-poetry/.venv/bin/python -m pip install --upgrade pip' command.
  

  at ~/.local/share/pypoetry/venv/lib/python3.6/site-packages/poetry/utils/env.py:1101 in _run
      1097│                 output = subprocess.check_output(
      1098│                     cmd, stderr=subprocess.STDOUT, **kwargs
      1099│                 )
      1100│         except CalledProcessError as e:
    → 1101│             raise EnvCommandError(e, input=input_)
      1102│ 
      1103│         return decode(output)
      1104│ 
      1105│     def execute(self, bin, *args, **kwargs):

[rusty1s]

This is a current limitation, indeed. The bad thing is that I do not think there exists a workaround for this. Any ideas?

[sisp]

I would need to verify some assumptions, but I believe it is possible to strictly separate build steps from runtime installation. Perhaps instead of passing torch's BuildExtension class to cmdclass, you could create a custom class and import from the torch package inside the run method (which I think is the one that is run when a command is executed). Something like:

from setuptools.command.install import install


class BuildExtensionCommand(install):
    def run(self):
        from torch.utils.cpp_extension import BuildExtension
        return BuildExtension.with_options(no_python_abi_suffix=True, use_ninja=False).run()


setup(
    # ...
    cmdclass={
        'build_ext': BuildExtensionCommand
    }
)

This snippet is completely unverified though, so just the sketch of an idea. I'm not sure yet how to create the ext_modules list without importing torch globally though.

[rusty1s]

Thanks for digging into this. If you are interested, please feel free to contribute :)

[Abhishaike]

I'm confused, if torch is a dependency for this library, why is it not included in setup.py as a dependency?

[rusty1s]

We need to import torch in setup.py for compilation, so we cannot add it as a dependency. It needs to be installed in advance :(

[JacobHayes]

I think this should might be possible with a pyproject.toml [build-system] / requires section from PEP 517, I'll put up a PR!

While pyproject.toml can indeed define build-time deps, that's not sufficient to match the host's CUDA version.

[abrahme]

what is the consensus workaround here if there is one?

[JacobHayes]

We currently have an install script that installs torch and then these packages. After that, we run poetry install. Since the installed versions of torch* don't match what poetry has locked (poetry expects eg: X.X.X, but sees X.X.X+cu116 or whatever) and would try to reinstall them, we have some hacky code that renames the installed packages (in site-packages) to remove the +cuXYZ from the folder/metadata so it matches poetry's expectations.

TL;DR pretty hacky. 😅 I think others may just avoid placing torch* in their pyproject.toml (assuming they don't have any transitive deps with it)?

[abrahme]

We currently have an install script that installs torch and then these packages. After that, we run poetry install. Since the installed versions of torch* don't match what poetry has locked (poetry expects eg: X.X.X, but sees X.X.X+cu116 or whatever) and would try to reinstall them, we have some hacky code that renames the installed packages (in site-packages) to remove the +cuXYZ from the folder/metadata so it matches poetry's expectations.

TL;DR pretty hacky. 😅 I think others may just avoid placing torch* in their pyproject.toml (assuming they don't have any transitive deps with it)?

sorry, i'm pretty new to this and also my first time responding to a github issue so forgive me if this is the wrong way to go about it. however, i'm unsure what is meant by "avoiding placing torch* in their pyproject.toml" means in this context. is it that those torch dependencies are installed without the use of Poetry, and other non-torch dependencies go through poetry?

[JacobHayes]

@abrahme no worries, your response seems like the right way to go!

is it that those torch dependencies are installed without the use of Poetry, and other non-torch dependencies go through poetry?

Yeah, I'd guess that's what others do. ie: the [tool.poetry.dependencies] would contain most deps but omit torch, torch-sparse, etc. Then, install those separately before or after the poetry install.

Another approach I've seen people do is hard code the URLs to pull wheels/etc from. You can specify markers so the right file is used for each OS/CPU, but you would have to just hard code the cuda version (eg: cu116) in the URL and ensure it aligns with your host(s).