No module named 'setuptools' when installing through pre-commit #1874

CarlosRDomin · 2021-12-28T13:53:37Z

We recently (within the past 5 days max) started having our CI pipelines fail due to pre-commit + isort raising a ModuleNotFoundError: No module named 'setuptools'. When we comment out the isort hook and leave e.g. black, flake8, etc on, the error goes away.

Steps to reproduce (e.g. inside docker run -it python:3.8 bash, also tested with 3.8.5, 3.8.12, 3.9):

# Create an empty folder and initialize a git repo in it, so pre-commit doesn't complain
mkdir test && cd test
git init

# Install pre-commit
pip install pre-commit

# Create a pre-commit config
cat > .pre-commit-config.yaml <<EOF
repos:
  - repo: https://github.com/pycqa/isort
    rev: 5.10.1  # We have tried with several versions, like 5.7.0, 5.5.2, 5.0.0, all with similar results
    hooks:
      - id: isort
EOF

# Run pre-commit (should succeed, there aren't any files!)
pre-commit run -a

Error details (cat /root/.cache/pre-commit/pre-commit.log)

version information

pre-commit version: 2.16.0
sys.version:
    3.8.12 (default, Dec 21 2021, 10:45:09)
    [GCC 10.2.1 20210110]
sys.executable: /usr/local/bin/python
os.name: posix
sys.platform: linux

error information

An unexpected error has occurred: CalledProcessError: command: ('/root/.cache/pre-commit/repor8uvua1d/py_env-python3/bin/python', '-mpip', 'install', '.')
return code: 2
expected return code: 0
stdout:
    Processing /root/.cache/pre-commit/repor8uvua1d
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'

stderr:
    ERROR: Exception:
    Traceback (most recent call last):
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
        status = run_func(*args)
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
        return func(self, options, args)
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 338, in run
        requirement_set = resolver.resolve(
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
        collected = self.factory.collect_root_requirements(root_reqs)
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 468, in collect_root_requirements
        req = self._make_requirement_from_install_req(
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 430, in _make_requirement_from_install_req
        cand = self._make_candidate_from_link(
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 201, in _make_candidate_from_link
        self._link_candidate_cache[link] = LinkCandidate(
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 281, in __init__
        super().__init__(
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
        self.dist = self._prepare()
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
        dist = self._prepare_distribution()
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 292, in _prepare_distribution
        return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
        return self._prepare_linked_requirement(req, parallel_builds)
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 546, in _prepare_linked_requirement
        dist = _get_prepared_distribution(
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 58, in _get_prepared_distribution
        abstract_dist.prepare_distribution_metadata(finder, build_isolation)
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 47, in prepare_distribution_metadata
        self._install_build_reqs(finder)
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 106, in _install_build_reqs
        build_reqs = self._get_build_requires_wheel()
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 83, in _get_build_requires_wheel
        return backend.get_requires_for_build_wheel()
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 172, in get_requires_for_build_wheel
        return self._call_hook('get_requires_for_build_wheel', {
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 332, in _call_hook
        raise BackendUnavailable(data.get('traceback', ''))
    pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 89, in _build_backend
        obj = import_module(mod_path)
      File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
      File "<frozen importlib._bootstrap>", line 991, in _find_and_load
      File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
      File "<frozen importlib._bootstrap>", line 991, in _find_and_load
      File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 843, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/tmp/pip-build-env-v50_yrw1/overlay/lib/python3.8/site-packages/poetry/core/masonry/__init__.py", line 10, in <module>
        from .builder import Builder
      File "/tmp/pip-build-env-v50_yrw1/overlay/lib/python3.8/site-packages/poetry/core/masonry/builder.py", line 7, in <module>
        from .builders.sdist import SdistBuilder
      File "/tmp/pip-build-env-v50_yrw1/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/__init__.py", line 2, in <module>
        from .wheel import WheelBuilder
      File "/tmp/pip-build-env-v50_yrw1/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 23, in <module>
        from packaging.tags import sys_tags
      File "/tmp/pip-build-env-v50_yrw1/overlay/lib/python3.8/site-packages/poetry/core/_vendor/packaging/tags.py", line 7, in <module>
        import distutils.util
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/_distutils_hack/__init__.py", line 92, in create_module
        return importlib.import_module('setuptools._distutils')
      File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
    ModuleNotFoundError: No module named 'setuptools'

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/pre_commit/error_handler.py", line 65, in error_handler
    yield
  File "/usr/local/lib/python3.8/site-packages/pre_commit/main.py", line 396, in main
    return run(args.config, store, args)
  File "/usr/local/lib/python3.8/site-packages/pre_commit/commands/run.py", line 417, in run
    install_hook_envs(to_install, store)
  File "/usr/local/lib/python3.8/site-packages/pre_commit/repository.py", line 224, in install_hook_envs
    _hook_install(hook)
  File "/usr/local/lib/python3.8/site-packages/pre_commit/repository.py", line 82, in _hook_install
    lang.install_environment(
  File "/usr/local/lib/python3.8/site-packages/pre_commit/languages/python.py", line 205, in install_environment
    helpers.run_setup_cmd(prefix, install_cmd)
  File "/usr/local/lib/python3.8/site-packages/pre_commit/languages/helpers.py", line 52, in run_setup_cmd
    cmd_output_b(*cmd, cwd=prefix.prefix_dir, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/pre_commit/util.py", line 154, in cmd_output_b
    raise CalledProcessError(returncode, cmd, retcode, stdout_b, stderr_b)
pre_commit.util.CalledProcessError: command: ('/root/.cache/pre-commit/repor8uvua1d/py_env-python3/bin/python', '-mpip', 'install', '.')
return code: 2
expected return code: 0
stdout:
    Processing /root/.cache/pre-commit/repor8uvua1d
      Installing build dependencies: started
      Installing build dependencies: finished with status 'done'
      Getting requirements to build wheel: started
      Getting requirements to build wheel: finished with status 'done'

stderr:
    ERROR: Exception:
    Traceback (most recent call last):
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/cli/base_command.py", line 164, in exc_logging_wrapper
        status = run_func(*args)
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/cli/req_command.py", line 205, in wrapper
        return func(self, options, args)
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/commands/install.py", line 338, in run
        requirement_set = resolver.resolve(
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/resolver.py", line 73, in resolve
        collected = self.factory.collect_root_requirements(root_reqs)
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 468, in collect_root_requirements
        req = self._make_requirement_from_install_req(
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 430, in _make_requirement_from_install_req
        cand = self._make_candidate_from_link(
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/factory.py", line 201, in _make_candidate_from_link
        self._link_candidate_cache[link] = LinkCandidate(
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 281, in __init__
        super().__init__(
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 156, in __init__
        self.dist = self._prepare()
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 225, in _prepare
        dist = self._prepare_distribution()
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/resolution/resolvelib/candidates.py", line 292, in _prepare_distribution
        return preparer.prepare_linked_requirement(self._ireq, parallel_builds=True)
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 482, in prepare_linked_requirement
        return self._prepare_linked_requirement(req, parallel_builds)
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 546, in _prepare_linked_requirement
        dist = _get_prepared_distribution(
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/operations/prepare.py", line 58, in _get_prepared_distribution
        abstract_dist.prepare_distribution_metadata(finder, build_isolation)
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 47, in prepare_distribution_metadata
        self._install_build_reqs(finder)
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 106, in _install_build_reqs
        build_reqs = self._get_build_requires_wheel()
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_internal/distributions/sdist.py", line 83, in _get_build_requires_wheel
        return backend.get_requires_for_build_wheel()
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 172, in get_requires_for_build_wheel
        return self._call_hook('get_requires_for_build_wheel', {
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_vendor/pep517/wrappers.py", line 332, in _call_hook
        raise BackendUnavailable(data.get('traceback', ''))
    pip._vendor.pep517.wrappers.BackendUnavailable: Traceback (most recent call last):
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 89, in _build_backend
        obj = import_module(mod_path)
      File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
      File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
      File "<frozen importlib._bootstrap>", line 991, in _find_and_load
      File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
      File "<frozen importlib._bootstrap>", line 991, in _find_and_load
      File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 843, in exec_module
      File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
      File "/tmp/pip-build-env-v50_yrw1/overlay/lib/python3.8/site-packages/poetry/core/masonry/__init__.py", line 10, in <module>
        from .builder import Builder
      File "/tmp/pip-build-env-v50_yrw1/overlay/lib/python3.8/site-packages/poetry/core/masonry/builder.py", line 7, in <module>
        from .builders.sdist import SdistBuilder
      File "/tmp/pip-build-env-v50_yrw1/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/__init__.py", line 2, in <module>
        from .wheel import WheelBuilder
      File "/tmp/pip-build-env-v50_yrw1/overlay/lib/python3.8/site-packages/poetry/core/masonry/builders/wheel.py", line 23, in <module>
        from packaging.tags import sys_tags
      File "/tmp/pip-build-env-v50_yrw1/overlay/lib/python3.8/site-packages/poetry/core/_vendor/packaging/tags.py", line 7, in <module>
        import distutils.util
      File "/root/.cache/pre-commit/repor8uvua1d/py_env-python3/lib/python3.8/site-packages/_distutils_hack/__init__.py", line 92, in create_module
        return importlib.import_module('setuptools._distutils')
      File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
    ModuleNotFoundError: No module named 'setuptools'

The text was updated successfully, but these errors were encountered:

mik-laj · 2021-12-28T14:17:55Z

I also encourage this problem. As workaround, I used the following config:

  - repo: local
    hooks:
      - id: isort
        name: isort
        entry: isort
        require_serial: true
        language: python
        language_version: python3
        types_or: [cython, pyi, python]
        args: ['--filter-files']
        minimum_pre_commit_version: '2.9.2'
        additional_dependencies: ['isort==5.10.1']

asottile · 2021-12-28T17:38:49Z

the root cause is a bug in pip coupled with some changes in setuptools -- here's proper workarounds: pre-commit/pre-commit#2178 (comment)

CarlosRDomin · 2021-12-28T18:14:36Z

Thank you for the quick responses everyone! For the readers who get here looking for a solution, I can confirm that all three of the workarounds proposed above work!

That is (either of these will do):

Using a "local" isort pre-commit hook (thanks @mik-laj for the quick workaround!)
The MR Depend on setuptools #1875 by @m-kuhn (tested with pre-commit try-repo), should it get merged? Sounds like a proper long-term solution
Downgrading virtualenv: pip install 'virtualenv<20.11' (sounds like this will stop working in 28 days, so Jan 21st 2022)

That pre-commit hook is currently failing to install because of PyCQA/isort#1874, and this change fixes that.

This PR resolves PyCQA/isort#1874, which is currently biting us when we attempt to install the isort pre-commit hook.

PyCQA/isort#1874 (comment)

Following along on PyCQA/isort#1874 , seems like this is the "real" fix.

timothycrosley · 2021-12-28T22:02:48Z

This is not an isort specific issue. isort hasn't made any releases that change this behavior, and in fact hasn't made any release in some months. It also has no external dependencies. isort uses poetry intentionally and not setuptools.This breakage is occurring upstream and I encourage you to reach out to the projects responsible, and in the meantime utilize SETUPTOOLS_USE_DISTUTILS=stdlib or one of the other workarounds @asottile has helpfully cataloged here: pre-commit/pre-commit#2178 (comment)

Will keep this issue open and pinned until it is resolved upstream so that users that run into this issue can quickly find appropriate workarounds.

m-kuhn · 2021-12-29T06:28:44Z

The underlying issue has been fixed upstream pypa/setuptools@9c9c91c

m-kuhn mentioned this issue Dec 28, 2021

Depend on setuptools #1875

Closed

browniebroke mentioned this issue Dec 28, 2021

Add generated files to .gitignore when selecting Gulp cookiecutter/cookiecutter-django#3500

Merged

2 tasks

jsf9k added a commit to cisagov/log4j-md-yml that referenced this issue Dec 28, 2021

Add setuptools as an additional dependency for the isort pre-commit hook

e88ce39

That pre-commit hook is currently failing to install because of PyCQA/isort#1874, and this change fixes that.

jsf9k added a commit to cisagov/log4j-md-yml that referenced this issue Dec 28, 2021

Use isort from PyCQA/isort#1875

51752ee

This PR resolves PyCQA/isort#1874, which is currently biting us when we attempt to install the isort pre-commit hook.

jsf9k mentioned this issue Dec 28, 2021

Revert to using the official isort repo cisagov/log4j-md-yml#2

Closed

mildbyte added a commit to splitgraph/sgr that referenced this issue Dec 28, 2021

Fix isort pre-commit hook

846e83b

PyCQA/isort#1874 (comment)

mildbyte added a commit to splitgraph/sgr that referenced this issue Dec 28, 2021

Fix isort pre-commit hook

ecf4d61

PyCQA/isort#1874 (comment)

mildbyte added a commit to splitgraph/sgr that referenced this issue Dec 28, 2021

Fix isort pre-commit hook

3c1bd4f

PyCQA/isort#1874 (comment)

dhalperi added a commit to dhalperi/pybatfish that referenced this issue Dec 28, 2021

ci: pin isort instead of virtualenv

0701ac0

Following along on PyCQA/isort#1874 , seems like this is the "real" fix.

timothycrosley added question Further information is requested upstream Not a bug in isort but in an upstream repository labels Dec 28, 2021

timothycrosley pinned this issue Dec 28, 2021

tomaarsen added a commit to tomaarsen/nltk that referenced this issue Dec 28, 2021

Update isort pre-commit due to issues: see PyCQA/isort#1874

9f4de18

BreadGenie mentioned this issue Dec 29, 2021

feat: Add offline command line option (#1452) intel/cve-bin-tool#1480

Merged

timothycrosley unpinned this issue Jan 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

No module named 'setuptools' when installing through pre-commit #1874

No module named 'setuptools' when installing through pre-commit #1874

CarlosRDomin commented Dec 28, 2021

version information

error information

mik-laj commented Dec 28, 2021

asottile commented Dec 28, 2021

CarlosRDomin commented Dec 28, 2021

timothycrosley commented Dec 28, 2021

m-kuhn commented Dec 29, 2021

No module named 'setuptools' when installing through pre-commit #1874

No module named 'setuptools' when installing through pre-commit #1874

Comments

CarlosRDomin commented Dec 28, 2021

version information

error information

mik-laj commented Dec 28, 2021

asottile commented Dec 28, 2021

CarlosRDomin commented Dec 28, 2021

timothycrosley commented Dec 28, 2021

m-kuhn commented Dec 29, 2021