Nondeterministic "Failed cleaning build dir" error when creating a packed Pex

[danxmoran]

This is a weird one that's been floating around Color for a few months, and it finally reproduced on my local machine. I think the top-level command that's failing is a Pex process.

---

We've occasionally been hitting failures from ./pants commands, caused by what looks like a race condition building / installing dependencies. The error always looks like:

pid 47365 -> /Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/bin/python -sE /Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/pex --disable-pip-version-check --no-python-version-warning --exists-action a --no-input --use-deprecated legacy-resolver --isolated -q --cache-dir /Users/dan.moran/.cache/pants/named_caches/pex_root/pip_cache wheel --no-deps --wheel-dir /Users/dan.moran/.cache/pants/named_caches/pex_root/built_wheels/sdists/cairocffi-1.4.0.tar.gz/509339b32ccd8d7b00c2204c32736cde78db53a32e6a162d312478d25626cd9a/cp38-cp38-macosx_13_0_x86_64.cc8653789c1e4bfab9ae4cf487719fcf.work /Users/dan.moran/.cache/pants/named_caches/pex_root/downloads/509339b32ccd8d7b00c2204c32736cde78db53a32e6a162d312478d25626cd9a/cairocffi-1.4.0.tar.gz --index-url https://pypi.org/simple/ --retries 5 --timeout 15 exited with 1 and STDERR:
  ERROR: Command errored out with exit status 1:
   command: /Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/dan.moran/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-hr3qjlai/setup.py'"'"'; __file__='"'"'/Users/dan.moran/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-hr3qjlai/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /Users/dan.moran/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-wheel-p9l79e9w
       cwd: /Users/dan.moran/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-hr3qjlai/
  Complete output (27 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-hr3qjlai/setup.py", line 10, in <module>
      setup(
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/__init__.py", line 168, in setup
      _install_setup_requires(attrs)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/__init__.py", line 163, in _install_setup_requires
      dist.fetch_build_eggs(dist.setup_requires)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/dist.py", line 766, in fetch_build_eggs
      resolved_dists = pkg_resources.working_set.resolve(
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/pkg_resources/__init__.py", line 812, in resolve
      dist = best[req.key] = env.best_match(
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1097, in best_match
      return self.obtain(req, installer)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1109, in obtain
      return installer(requirement)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/dist.py", line 829, in fetch_build_egg
      return fetch_build_egg(self, req)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/installer.py", line 149, in fetch_build_egg
      wheel.install_as_egg(dist_location)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/wheel.py", line 130, in install_as_egg
      self._install_as_egg(destination_eggdir, zf)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/wheel.py", line 138, in _install_as_egg
      self._convert_metadata(zf, destination_eggdir, dist_info, egg_info)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/wheel.py", line 159, in _convert_metadata
      os.mkdir(destination_eggdir)
  FileExistsError: [Errno 17] File exists: '/Users/dan.moran/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-hr3qjlai/.eggs/cffi-1.15.1-py3.8-macosx-12.6-x86_64.egg'
  ----------------------------------------
  ERROR: Failed building wheel for cairocffi
  ERROR: Command errored out with exit status 1:
   command: /Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/Users/dan.moran/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-hr3qjlai/setup.py'"'"'; __file__='"'"'/Users/dan.moran/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-hr3qjlai/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' clean --all
       cwd: /Users/dan.moran/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-hr3qjlai
  Complete output (27 lines):
  Traceback (most recent call last):
    File "<string>", line 1, in <module>
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-hr3qjlai/setup.py", line 10, in <module>
      setup(
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/__init__.py", line 168, in setup
      _install_setup_requires(attrs)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/__init__.py", line 163, in _install_setup_requires
      dist.fetch_build_eggs(dist.setup_requires)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/dist.py", line 766, in fetch_build_eggs
      resolved_dists = pkg_resources.working_set.resolve(
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/pkg_resources/__init__.py", line 812, in resolve
      dist = best[req.key] = env.best_match(
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1097, in best_match
      return self.obtain(req, installer)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1109, in obtain
      return installer(requirement)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/dist.py", line 829, in fetch_build_egg
      return fetch_build_egg(self, req)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/installer.py", line 149, in fetch_build_egg
      wheel.install_as_egg(dist_location)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/wheel.py", line 130, in install_as_egg
      self._install_as_egg(destination_eggdir, zf)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/wheel.py", line 138, in _install_as_egg
      self._convert_metadata(zf, destination_eggdir, dist_info, egg_info)
    File "/Users/dan.moran/.cache/pants/named_caches/pex_root/venvs/a9ec98fb3f155f4e8b5de6c1ff190309ce15121b/0796ea8d0cb05fb74572064697475fd9cfdfa756/lib/python3.8/site-packages/setuptools/wheel.py", line 159, in _convert_metadata
      os.mkdir(destination_eggdir)
  FileExistsError: [Errno 17] File exists: '/Users/dan.moran/.cache/pants/named_caches/pex_root/pip_cache/.tmp/pip-req-build-hr3qjlai/.eggs/cffi-1.15.1-py3.8-macosx-12.6-x86_64.egg'
  ----------------------------------------
  ERROR: Failed cleaning build dir for cairocffi
ERROR: Failed to build one or more wheels

Once the error occurs it reliably reproduces, which is odd because ~/.cache/pants/named_caches/pex_root/pip_cache/.tmp/ is an empty dir in-between the failed runs. The only reliable way we've found to get un-stuck is to delete & reinstall Python (we use asdf so we rm ~/.asdf and then asdf install to get it back).

[danxmoran]

Hmmm pypa/cibuildwheel#813 discusses seeing the same error in another context, apparently fixed by switching to a pep517 build.

[jsirois]

@danxmoran I was hoping you noticed setup requires - those are always trouble. If you can at all find a way to prebuild wheels you should.