Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FileExistsError while building requirements.pex #15862

Closed
danxmoran opened this issue Jun 17, 2022 · 6 comments
Closed

FileExistsError while building requirements.pex #15862

danxmoran opened this issue Jun 17, 2022 · 6 comments
Labels
bug

Comments

@danxmoran
Copy link
Contributor

Describe the bug

While running ./pants lint <dir>, a teammate is hitting:

Engine traceback:
  in select
  in pants.core.goals.lint.lint
  in pants.backend.python.lint.pylint.rules.pylint_lint (pylint)
  in pants.backend.python.lint.pylint.rules.pylint_lint_partition
  in pants.backend.python.util_rules.pex.create_pex
  in pants.backend.python.util_rules.pex.build_pex (requirements.pex)
  in pants.engine.process.fallible_to_exec_result_or_raise
Traceback (most recent call last):
  File "/Users/erica.stephan/.cache/pants/setup/bootstrap-Darwin-x86_64/pants.BV1PAP/install/lib/python3.9/site-packages/pants/engine/process.py", line 272, in fallible_to_exec_result_or_raise
    raise ProcessExecutionFailure(
pants.engine.process.ProcessExecutionFailure: Process 'Building 86 requirements for requirements.pex from the 3rdparty/lockfiles/resolves/default.lockfile resolve: biopython==1.76, bioutils==0.5.5, bleach==4.1.0, boto3==1.20.52, botocore==1.23.52, braintree==3.37.2, business-rules==1.0.1, cachetools==4.2.4, celery[sqs]==4.4.7, clrenv@ git+https://github.com/color/clrenv.git@v0.2.2#c2be7e7c9a23d0d203daa0515d5c2df0ad270786, clrriskmodels@ git+https://github.com/color/risk-models.git@v1.1.1#f2bbe428da49f9d141795bee47469de83e6c28e7, clrypt@ git+https://github.com/color/clrypt.git@v0.2.3#861efad297750138e25d18c4867009419fe39459, dataclasses==0.6; python_version < "3.7", ddtrace==0.48.0, django-countries==7.2.1, django-filter==2.4.0, django-fsm-admin==1.2.4, django-fsm==2.8.0, django-object-actions==3.0.2, django-simple-history==3.0.0, django-timezone-field==4.2.1, django-waffle==2.2.1, django[bcrypt]==3.2.13, djangoql==0.16.0, djangorestframework-api-key==2.0.0, djangorestframework-queryfields==1.0.0, djangorestframework==3.13.1, easypost==5.1.3, filetype==1.0.8, google-api-python-client==1.6.4, google-auth-httplib2==0.0.4, gtfparse==1.2.0, hgvs==1.5.2, hl7apy==1.3.4, honeycomb-beeline==3.1.0, html2text==2020.1.16, intervaltree==3.0.2, jinja2==2.11.3, jira==3.1.1, jsonfield==2.1.1, jsonschema==3.0.2, markdown==3.3.4, memoize@ git+https://github.com/color/memoize.git@5c2712b301b07aee9fc7dc5a6548459488a5ad44, moto[ec2,s3,sqs,ssm]==2.3.1, munch==2.2.0, numpy==1.19.5, openpyxl==3.0.9, pandas==1.1.5, paramiko==2.8.0, parsley==1.3, pgpy==0.5.4, phonenumberslite==8.12.44, pillow==8.3.2, premailer==3.10.0, psutil==5.9.0, pybedtools==0.9.0, pycryptodome==3.6.6, pygments==2.10.0, pyhgvs@ git+https://github.com/counsyl/hgvs.git@ab9b95f21466fbb3265b5bd818ccab0c926ca59f, pyjwt==2.4.0, pylibdmtx==0.1.9, pypdf2==2.0.0, pysam==0.15.4, pytest==6.2.4, python-dateutil==2.8.2, python-intervals==1.10.0, pytz==2021.3, pyvcf==0.6.8, pyyaml==5.4, raven==6.10.0, redis==3.5.3, requests==2.27.1, responses==0.17.0, scipy==1.5.4, setuptools==45.3.0, six==1.16.0, slack-sdk==3.12.0, stripe==2.60.0, text-unidecode==1.3, twilio==7.0.0, typing-extensions==3.10.0.2, us==2.0.2, uszipcode==0.2.6, weasyprint==52.5, xmltodict==0.12.0, zeep==4.1.0' failed with exit code 1.
stdout:

stderr:
  ERROR: Command errored out with exit status 1:
   command: /Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/erica.stephan/.cache/pants/named_caches/pex_root/tmp/pip-req-build-g334q3ej/setup.py'"'"'; __file__='"'"'/Users/erica.stephan/.cache/pants/named_caches/pex_root/tmp/pip-req-build-g334q3ej/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /Users/erica.stephan/.cache/pants/named_caches/pex_root/tmp/pip-wheel-c03klxij
       cwd: /Users/erica.stephan/.cache/pants/named_caches/pex_root/tmp/pip-req-build-g334q3ej/
  Complete output (27 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/tmp/pip-req-build-g334q3ej/setup.py", line 10, in <module>
      setup(
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/__init__.py", line 168, in setup
      _install_setup_requires(attrs)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/__init__.py", line 163, in _install_setup_requires
      dist.fetch_build_eggs(dist.setup_requires)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/dist.py", line 766, in fetch_build_eggs
      resolved_dists = pkg_resources.working_set.resolve(
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 812, in resolve
      dist = best[req.key] = env.best_match(
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1097, in best_match
      return self.obtain(req, installer)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1109, in obtain
      return installer(requirement)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/dist.py", line 829, in fetch_build_egg
      return fetch_build_egg(self, req)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/installer.py", line 149, in fetch_build_egg
      wheel.install_as_egg(dist_location)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/wheel.py", line 130, in install_as_egg
      self._install_as_egg(destination_eggdir, zf)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/wheel.py", line 138, in _install_as_egg
      self._convert_metadata(zf, destination_eggdir, dist_info, egg_info)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/wheel.py", line 159, in _convert_metadata
      os.mkdir(destination_eggdir)
  FileExistsError: [Errno 17] File exists: '/Users/erica.stephan/.cache/pants/named_caches/pex_root/tmp/pip-req-build-g334q3ej/.eggs/cffi-1.15.0-py3.8-macosx-11.6-x86_64.egg'
  ----------------------------------------
  ERROR: Failed building wheel for cairocffi
  ERROR: Command errored out with exit status 1:
   command: /Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/bin/python3.8 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/erica.stephan/.cache/pants/named_caches/pex_root/tmp/pip-req-build-g334q3ej/setup.py'"'"'; __file__='"'"'/Users/erica.stephan/.cache/pants/named_caches/pex_root/tmp/pip-req-build-g334q3ej/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
       cwd: /Users/erica.stephan/.cache/pants/named_caches/pex_root/tmp/pip-req-build-g334q3ej
  Complete output (27 lines):

  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/tmp/pip-req-build-g334q3ej/setup.py", line 10, in <module>
      setup(
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/__init__.py", line 168, in setup
      _install_setup_requires(attrs)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/__init__.py", line 163, in _install_setup_requires
      dist.fetch_build_eggs(dist.setup_requires)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/dist.py", line 766, in fetch_build_eggs
      resolved_dists = pkg_resources.working_set.resolve(
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 812, in resolve
      dist = best[req.key] = env.best_match(
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1097, in best_match
      return self.obtain(req, installer)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1109, in obtain
      return installer(requirement)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/dist.py", line 829, in fetch_build_egg
      return fetch_build_egg(self, req)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/installer.py", line 149, in fetch_build_egg
      wheel.install_as_egg(dist_location)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/wheel.py", line 130, in install_as_egg
      self._install_as_egg(destination_eggdir, zf)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/wheel.py", line 138, in _install_as_egg
      self._convert_metadata(zf, destination_eggdir, dist_info, egg_info)
    File "/Users/erica.stephan/.cache/pants/named_caches/pex_root/venvs/s/1280d463/venv/lib/python3.8/site-packages/setuptools/wheel.py", line 159, in _convert_metadata
      os.mkdir(destination_eggdir)
  FileExistsError: [Errno 17] File exists: '/Users/erica.stephan/.cache/pants/named_caches/pex_root/tmp/pip-req-build-g334q3ej/.eggs/cffi-1.15.0-py3.8-macosx-11.6-x86_64.egg'
  ----------------------------------------

Unfortunately others haven't been able to reproduce 😞 but it looks like a potential race condition coming out of pex.

Pants version

PANTS_SHA=19fe4e3cec21067791dea8f1351c033c7132f579

(living dangerously so we can try out the improvements / new features slated for 2.13)

OS

MacOS

Additional info

The day we jumped to the 2.13 prerelease was (unfortunately) the same day Pants was added by default to many eng workflows at my company, so it's unclear to me if this problem is new to Pants 2.13 or if it's always been there and we just haven't had enough people running Pants to hit it. We're still debugging on our end.
@danxmoran danxmoran added the bug label Jun 17, 2022
@danxmoran
Copy link
Contributor Author

pypa/cibuildwheel#813 has some discussion of what looks like the same error. It was closed with a work-around - not sure if any of the discussion could apply to Pants / pex

@jsirois
Copy link
Contributor

jsirois commented Jun 17, 2022 •
edited
Loading

The work-around Pip option is plumbed through PEX and that is:

--force-pep517, --use-pep517, --no-use-pep517
                        Whether to force use of PEP 517 for building source
                        distributions into wheels
                        (https://www.python.org/dev/peps/pep-0518) or force
                        direct invocation of`setup.py bdist_wheel` (which
                        requires all source distributions have a `setup.py`
                        based build). Defaults to using PEP-517 only when a
                        `pyproject.toml` file is present with a `build-system`
                        section. If PEP-517 is forced (--use-pep517 is passed)
                        and no `pyproject.toml` file is present or one is but
                        does not have a `build-system` section defined, then
                        the build is executed as if a `pyproject.toml` was
                        present with a `build-system` section comprised of
                        `requires = ["setuptools>=40.8.0", "wheel"]` and
                        `build-backend = "setuptools.build_meta:__legacy__"`.
                        (default: None)

It looks like Pants does not plumb this though.

You might also try https://www.pantsbuild.org/docs/reference-python#section-macos-big-sur-compatibility but that almost certainly won't help.

@jsirois
Copy link
Contributor

jsirois commented Jun 17, 2022

@danxmoran If you want to dig here on the machine where the problem occurs, the steps would be:

pex -r requirements.txt --resolver-version pip-2020-resolver -o /tmp/test.pex

Where requirements.txt is here.

That should hopefully just reproduce the error, but with Pants removed.

Then try:

pex -r requirements.txt --resolver-version pip-2020-resolver -o /tmp/test.pex --force-pep517

If this fixes things, then we can move on to considering how to plumb this Pex option through Pants.

@danxmoran
Copy link
Contributor Author

Just realized this was the very first time we saw pex-tool/pex#2025

@aakog
Copy link

aakog commented Mar 10, 2023

Some of our devs hit this issue consistently on M1 Mac. The issue went away after they removed all python versions and installed only 3.9.16 using pyenv.

@danxmoran
Copy link
Contributor Author

Going to close this one as it seems like a problem with the underlying dependencies, not Pants

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jsirois @danxmoran @aakog