Experimental installer and Pyspark ~2.4: AttributeError: module 'pypandoc' has no attribute 'convert'

[Aremaki]

• Poetry version: 1.2.2
• Python version: 3.7.13
• OS version and name: WSL2 with distro Ubuntu 20.04.4 LTS
• pyproject.toml: https://gist.github.com/Aremaki/9d4b5cf7d821774dcaef8192147d7738

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

Reproduction steps:

I juste created a new virtual environnement with python 3.7.13:

pip list
Package    Version
---------- -------
pip        22.2.2
setuptools 47.1.0

I checked that poetry is using this virtual environnement:

$ poetry env info

Virtualenv
Python:         3.7.13
Implementation: CPython
Path:           /mnt/c/Users/4192941/Documents/Projets/EDS-Temporal-Variability/edsteva/venv
Executable:     /mnt/c/Users/4192941/Documents/Projets/EDS-Temporal-Variability/edsteva/venv/bin/python
Valid:          True

System
Platform:   linux
OS:         posix
Python:     3.7.13
Path:       /home/adam/miniconda3/envs/py37
Executable: /home/adam/miniconda3/envs/py37/bin/python3.7

I checked poetry config:

$ poetry config --list

cache-dir = "/home/adam/.cache/pypoetry"
experimental.new-installer = true
experimental.system-git-client = false
installer.max-workers = null
installer.no-binary = null
installer.parallel = true
repositories.datasciencetools.url = "https://gitlab.eds.aphp.fr/datasciencetools/"
repositories.edsteva.url = "https://gitlab.eds.aphp.fr/datasciencetools/edsteva/"
repositories.equipedatascience.url = "https://gitlab.eds.aphp.fr/equipedatascience/"
virtualenvs.create = true
virtualenvs.in-project = true
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.no-setuptools = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}/virtualenvs"  # /home/adam/.cache/pypoetry/virtualenvs
virtualenvs.prefer-active-python = false
virtualenvs.prompt = "{project_name}-py{python_version}"

Then I run poetry install and it fails to install pyspark with the following error:

$ poetry install

Updating dependencies
Resolving dependencies... (0.4s)

Writing lock file

Package operations: 2 installs, 0 updates, 0 removals

  • Installing py4j (0.10.7)
  • Installing pyspark (2.4.8): Failed

  CalledProcessError

  Command '['/mnt/c/Users/4192941/Documents/Projets/EDS-Temporal-Variability/edsteva/venv/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/mnt/c/Users/4192941/Documents/Projets/EDS-Temporal-Variability/edsteva/venv', '--no-deps', '/home/adam/.cache/pypoetry/artifacts/11/a7/ca/a969dff384e5444fdea49d15c9d54f017af4359bf00741b8aa90cd331f/pyspark-2.4.8.tar.gz']' returned non-zero exit status 1.

  at ~/miniconda3/lib/python3.9/subprocess.py:528 in run
       524│             # We don't call process.wait() as .__exit__ does that for us.
       525│             raise
       526│         retcode = process.poll()
       527│         if check and retcode:
    →  528│             raise CalledProcessError(retcode, process.args,
       529│                                      output=stdout, stderr=stderr)
       530│     return CompletedProcess(process.args, retcode, stdout, stderr)
       531│ 
       532│ 

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/mnt/c/Users/4192941/Documents/Projets/EDS-Temporal-Variability/edsteva/venv/bin/python', '-m', 'pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/mnt/c/Users/4192941/Documents/Projets/EDS-Temporal-Variability/edsteva/venv', '--no-deps', '/home/adam/.cache/pypoetry/artifacts/11/a7/ca/a969dff384e5444fdea49d15c9d54f017af4359bf00741b8aa90cd331f/pyspark-2.4.8.tar.gz'] errored with the following return code 1, and output: 
  Processing /home/adam/.cache/pypoetry/artifacts/11/a7/ca/a969dff384e5444fdea49d15c9d54f017af4359bf00741b8aa90cd331f/pyspark-2.4.8.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'done'
    Installing backend dependencies: started
    Installing backend dependencies: finished with status 'done'
    Preparing metadata (pyproject.toml): started
    Preparing metadata (pyproject.toml): finished with status 'error'
    error: subprocess-exited-with-error
    
    × Preparing metadata (pyproject.toml) did not run successfully.
    │ exit code: 1
    ╰─> [15 lines of output]
        Traceback (most recent call last):
          File "/mnt/c/Users/4192941/Documents/Projets/EDS-Temporal-Variability/edsteva/venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 363, in <module>
            main()
          File "/mnt/c/Users/4192941/Documents/Projets/EDS-Temporal-Variability/edsteva/venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 345, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/mnt/c/Users/4192941/Documents/Projets/EDS-Temporal-Variability/edsteva/venv/lib/python3.7/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 164, in prepare_metadata_for_build_wheel
            return hook(metadata_directory, config_settings)
          File "/tmp/pip-build-env-ybyrrroq/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 377, in prepare_metadata_for_build_wheel
            self.run_setup()
          File "/tmp/pip-build-env-ybyrrroq/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 483, in run_setup
            self).run_setup(setup_script=setup_script)
          File "/tmp/pip-build-env-ybyrrroq/overlay/lib/python3.7/site-packages/setuptools/build_meta.py", line 335, in run_setup
            exec(code, locals())
          File "<string>", line 156, in <module>
        AttributeError: module 'pypandoc' has no attribute 'convert'
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: metadata-generation-failed
  
  × Encountered error while generating package metadata.
  ╰─> See above for output.
  
  note: This is an issue with the package mentioned above, not pip.
  hint: See above for details.
  

  at ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/utils/env.py:1476 in _run
      1472│                 output = subprocess.check_output(
      1473│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1474│                 )
      1475│         except CalledProcessError as e:
    → 1476│             raise EnvCommandError(e, input=input_)
      1477│ 
      1478│         return decode(output)
      1479│ 
      1480│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  PoetryException

  Failed to install /home/adam/.cache/pypoetry/artifacts/11/a7/ca/a969dff384e5444fdea49d15c9d54f017af4359bf00741b8aa90cd331f/pyspark-2.4.8.tar.gz

  at ~/.local/pipx/venvs/poetry/lib/python3.9/site-packages/poetry/utils/pip.py:51 in pip_install
       47│ 
       48│     try:
       49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
    →  51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       52│ 

This error normally occurs when a version of pypandoc >= 1.8 is installed becasue the method convert is deprecated (cf. pyspark.setup.py and pypandoc. But it does not occur when pypandoc is not installed and in this case pypandoc is not installed in my virtual environnement. When I do a simple:

pip install "pysaprk==2.4.8"

It works fine !

Moreover when i set the experimental installer to false it also works fine:

$ poetry config experimental.new-installer false
$ poetry install
                          
Updating dependencies
Resolving dependencies... (0.2s)

Writing lock file

Package operations: 2 installs, 0 updates, 0 removals

  - Installing py4j (0.10.7)
  - Installing pyspark (2.4.8)

Installing the current project: edsteva (1.0.0)

Thus, I suspect the experimental new installer to use a version of pypandoc >= 1.8. Would it be possible to add an option to remove this.

Best regards,

[neersighted]

Looks like a issue with an unbounded build-time dependency for Pyspark -- pip install --use-pep517 pyspark==2.4.8 reproduces the exact same error. Because pypandoc made a breaking change and there was no upper bound in Pyspark, a newer version that drops the convert() function gets pulled in during a build. As Poetry doesn't mix install dependencies with build-time dependencies, your pyproject.toml has no influence on the version used during the build of Spark.

For what it's worth, it looks like current versions of Pyspark (e.g. ==3.3.0) build just fine; upstream appears to have resolved the issue some time ago. If you are able to migrate to a newer version, things should just work. If not, you may be best off forking Pyspark and using a URL dependency to depend on your patched version.

[neersighted]

Dug a little deeper and it appears that they ended up dropping the dependency entirely in 3.0: apache/spark#18981

[Aremaki]

Thanks for the reply. This was very helpful

[carlotardivo]

Hi, in the past I ran into the same issue while trying to install pyspark 2.4.7 on python 3.7.10 environment. The solution for me was to deactivate the experimental new installer, for example by adding the following lines to the poetry.toml file:

[experimental]
new-installer = false

Now I've updated poetry to 1.6.0, but with the same poetry.toml file this error is appearing again. I see that a modern-installation is now available so I deactivated it, but the error is still raised.

I now that this issue is not a poetry bug, but is it possible to recover the old workaround and let poetry execute successfully?

Thanks.

[carlotardivo]

@neersighted tagging you just to be sure that you get notified (I don't know notifications policy on closed issues, sorry).

[carlotardivo]

I now see that old installer has been removed in 1.5.0 version, so I think that the old workaround cannot be used anymore, and the only way to let poetry > 1.4.2 successfully build with pyspark 2.4 is to fork the pyspark repo.

@neersighted can you confirm my conclusions?

Thanks.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.