Installation Error: Failed to build installable wheels for some pyproject.toml based projects (pendulum)

[bennwei]

I tried to install PyAirbyte, but installation failed due to pendulum==2.1.2:

poetry add airbyte 

• Installing pendulum (2.1.2): Failed

  ChefBuildError

  Backend subprocess exited when trying to invoke build_wheel
  
  Traceback (most recent call last):
    File "/tmp/tmpe0dmrq00/pendulum-2.1.2/build.py", line 5, in <module>
      from distutils.command.build_ext import build_ext
  ModuleNotFoundError: No module named 'distutils'
  Traceback (most recent call last):
    File "/home/eliao/.local/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
      main()
    File "/home/eliao/.local/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 335, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/home/eliao/.local/pipx/venvs/poetry/lib/python3.10/site-packages/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
      return _build_backend().build_wheel(wheel_directory, config_settings,
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/tmpqzlbwpex/.venv/lib/python3.12/site-packages/poetry/core/masonry/api.py", line 58, in build_wheel
      return WheelBuilder.make_in(
             ^^^^^^^^^^^^^^^^^^^^^
    File "/tmp/tmpqzlbwpex/.venv/lib/python3.12/site-packages/poetry/core/masonry/builders/wheel.py", line 88, in make_in
      wb.build(target_dir=directory)
    File "/tmp/tmpqzlbwpex/.venv/lib/python3.12/site-packages/poetry/core/masonry/builders/wheel.py", line 123, in build
      self._build(zip_file)
    File "/tmp/tmpqzlbwpex/.venv/lib/python3.12/site-packages/poetry/core/masonry/builders/wheel.py", line 172, in _build
      self._run_build_script(self._package.build_script)
    File "/tmp/tmpqzlbwpex/.venv/lib/python3.12/site-packages/poetry/core/masonry/builders/wheel.py", line 262, in _run_build_script
      subprocess.check_call([self.executable.as_posix(), build_script])
    File "/home/eliao/.pyenv/versions/3.12.1/lib/python3.12/subprocess.py", line 413, in check_call
      raise CalledProcessError(retcode, cmd)
  subprocess.CalledProcessError: Command '['/tmp/tmpqzlbwpex/.venv/bin/python', 'build.py']' returned non-zero exit status 1.
  

  at [~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry/installation/chef.py](https://vscode-remote+wsl-002bubuntu.vscode-resource.vscode-cdn.net/home/eliao/projects/IMAO_DS/python_dev/ds_sandbox/sharepoint_automate/~/.local/pipx/venvs/poetry/lib/python3.10/site-packages/poetry/installation/chef.py):164 in _prepare
      160│ 
      161│                 error = ChefBuildError("\n\n".join(message_parts))
      162│ 
      163│             if error is not None:
    → 164│                 raise error from None
      165│ 
      166│             return path
      167│ 
      168│     def _prepare_sdist(self, archive: Path, destination: Path | None = None) -> Path:

Note: This error originates from the build backend, and is likely not a problem with poetry but with pendulum (2.1.2) not supporting PEP 517 builds. You can verify this by running 'pip wheel --no-cache-dir --use-pep517 "pendulum (==2.1.2)"'.

Also failed trying using pip install:

pip install airbyte

Building wheels for collected packages: pendulum
  Building wheel for pendulum (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for pendulum (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [27 lines of output]
      Traceback (most recent call last):
        File "/tmp/pip-install-a1cc5gy7/pendulum_2d4989dfcedd41dc827c4c2e67120bd7/build.py", line 5, in <module>
          from distutils.command.build_ext import build_ext
      ModuleNotFoundError: No module named 'distutils'
      Traceback (most recent call last):
        File "/home/eliao/.cache/pypoetry/virtualenvs/sharepoint-automate-jCMQFMG6-py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/eliao/.cache/pypoetry/virtualenvs/sharepoint-automate-jCMQFMG6-py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/eliao/.cache/pypoetry/virtualenvs/sharepoint-automate-jCMQFMG6-py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-g042xrh4/overlay/lib/python3.12/site-packages/poetry/core/masonry/api.py", line 58, in build_wheel
          return WheelBuilder.make_in(
                 ^^^^^^^^^^^^^^^^^^^^^
        File "/tmp/pip-build-env-g042xrh4/overlay/lib/python3.12/site-packages/poetry/core/masonry/builders/wheel.py", line 88, in make_in
          wb.build(target_dir=directory)
        File "/tmp/pip-build-env-g042xrh4/overlay/lib/python3.12/site-packages/poetry/core/masonry/builders/wheel.py", line 123, in build
          self._build(zip_file)
        File "/tmp/pip-build-env-g042xrh4/overlay/lib/python3.12/site-packages/poetry/core/masonry/builders/wheel.py", line 172, in _build
          self._run_build_script(self._package.build_script)
        File "/tmp/pip-build-env-g042xrh4/overlay/lib/python3.12/site-packages/poetry/core/masonry/builders/wheel.py", line 262, in _run_build_script
          subprocess.check_call([self.executable.as_posix(), build_script])
        File "/home/eliao/.pyenv/versions/3.12.1/lib/python3.12/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['/home/eliao/.cache/pypoetry/virtualenvs/sharepoint-automate-jCMQFMG6-py3.12/bin/python', 'build.py']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for pendulum
Failed to build pendulum
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (pendulum)

[tcboles]

I am having the same issue. It looks like airbyte-cdk (https://github.com/tcboles/airbyte/blob/master/airbyte-cdk/python/pyproject.toml) and a few connectors would need to be updated to use the latest pendulum version to make this work with projects that already use the latest pendulum version or have dependencies that do.

[neerajgupta2407]

I am also facing the same error.

[bennwei]

I had tried many ways in my local env and can not bypass the pendulum==2.1.2 issue. But when I tried a colab demo example and airbyte was able to installed without issue:
https://colab.research.google.com/github/airbytehq/quickstarts/blob/master/pyairbyte_notebooks/PyAirbyte_Basic_Features_Demo.ipynb

[notpushkin]

This happens because distutils was deprecated in Python 3.12. The dependency seems to be fixed in Pendulum 3.0, so bumping the dependency should do the trick.

[bennwei]

@notpushkin How did you resolve this on your side? I tried to bump Pendulum==3.0, but the issues is on the SDK airbyte-cdk (https://github.com/tcboles/airbyte/blob/master/airbyte-cdk/python/pyproject.toml) as @tcboles mentioned above that SDK is not compatible with Pendulum==3.0 install. Let me know if any solution you can bypass this issue when there is no package updates from the author.

[aaronsteers]

@notpushkin, @neerajgupta2407, @bennwei, @tcboles - Thanks everyone for reporting this and sharing your observations here.

It sounds like this issue is specific to Python 3.12+, where distutils is no longer available, while at the same time being required by Pendulum 2.x. As others have noted, the CDK constrains Pendulum to <3.0 so I think there are just two solutions:

1. Update the CDK constraint to allow 3.x versions of Pendulum.
2. Use Python 3.11 (or 3.10) where distutils is still available.

Does that feel like an accurate summary? If so, I'll open an issue shortly to request a Pendulum version bump in the CDK upstream.

I will look into breaking changes... UPDATE: I have looked into breaking changes in Pendulum 3.0 and one change that I was able to identify as impactful to some connectors is the renaming of the Period class to Interval. Some of our connectors use this type and it would be non-trivial to make them forwards and backwards compatible across major versions of Pendulum. I'm curious if there are any other workarounds 2.x to make Pendulum install correctly. I'll look deeper into this and report back...

Please feel free to chime in if this sounds like an appropriate path forward or if you have any other ideas or feedback. Thanks!

[aaronsteers]

A very highly esteemed ex-colleague of mine @edgarrmondragon posted this comment:

Pendulum 3.0.0b1 has wheels for M1 macs and Python 3.12, but builds of Pendulum 2.* on Python 3.12+ seem like they'll be forever broken.

In terms of upgrading, I worry about supporting only Pendulum 3.x, since this will break side-by-side installations if users are using libraries that might still be on Pendulum 2.x. For a while, at least, it seems the safest option is to downgrade to Python 3.11 if possible. The longer-term option seem to be (1) update the CDK and PyAirbyte to allow >=2.0,<4.0, and then (2) update the connectors to use the more restrictive, but forward-looking >=3.0,<4.0.

[notpushkin]

@bennwei

How did you resolve this on your side?

I didn't (yet). I was trying to play around with PyAirbyte to see if it's a good fit for a project I'm working on, so when it didn't work I just moved on ¯\_(ツ)_/¯

[neerajgupta2407]

@aaronsteers Thanks for the support.

Now I am using python version 3.10 and it's working fine.