pex 1.6.1 cannot build cryptography 2.5 #661

jsirois · 2019-01-31T18:33:02Z

The mechanics here are subtle - setuptools does a build isolation dance involving unimports that interacts in a fragile way with its monkey-patching of distutils. We see a failure like so due to the unimporting interaction with the monkeypatches:

$ pex -vvvvvvvvv cryptography==2.5
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7f2fe54f2fd0>
pex: 
Modifying given platform of 'current':
Using the current platform of Platform(platform='linux_x86_64', impl='cp', version='37', abi='cp37m')
Under current interpreter PythonInterpreter('/usr/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 2), {})

To match given interpreter PythonInterpreter('/usr/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 2), {}).

Calculated platform: Platform(platform='linux_x86_64', impl='cp', version='37', abi='cp37m')
pex: R: tags for Platform(platform='linux_x86_64', impl='cp', version='37', abi='cp37m') x PythonInterpreter('/usr/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 2), {}) -> [('cp37', 'cp37m', 'manylinux1_x86_64'), ('cp37', 'cp37m', 'linux_x86_64'), ('cp37', 'abi3', 'manylinux1_x86_64'), ('cp37', 'abi3', 'linux_x86_64'), ('cp37', 'none', 'manylinux1_x86_64'), ('cp37', 'none', 'linux_x86_64'), ('py3', 'none', 'manylinux1_x86_64'), ('py3', 'none', 'linux_x86_64'), ('cp37', 'none', 'any'), ('cp3', 'none', 'any'), ('py37', 'none', 'any'), ('py3', 'none', 'any')]
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7f2fe5497ac8>
pex: crawling link i=0 link=Link('file:///home/jsirois/.pex/build') follow_links=False
pex: crawling link i=0 link=Link('https://pypi.org/simple/cryptography/') follow_links=False
**** Failed to install cryptography-2.5 (caused by: NonZeroExit("received exit code 1 during execution of `['/usr/bin/python3.7', '-sE', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpbq0630tn']` while trying to execute `['/usr/bin/python3.7', '-sE', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpbq0630tn']`")
):
stdout:

stderr:
Traceback (most recent call last):
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-j1m6g1u8/pycparser-2.19/setup.py", line 65, in <module>
    # If there's no vectors locally that probably means we are in a tarball and
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 142, in setup
    _install_setup_requires(attrs)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 135, in _install_setup_requires
    dist.parse_config_files(ignore_option_errors=True)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 564, in parse_config_files
    _Distribution.parse_config_files(self, filenames=filenames)
  File "/usr/lib/python3.7/distutils/dist.py", line 397, in parse_config_files
    filenames = self.find_config_files()
  File "/usr/lib/python3.7/distutils/dist.py", line 352, in find_config_files
    sys_dir = os.path.dirname(sys.modules['distutils'].__file__)
KeyError: 'distutils'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 14, in <module>
  File "<string>", line 324, in <module>
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 142, in setup
    _install_setup_requires(attrs)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 137, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 586, in fetch_build_eggs
    replace_conflicting=True,
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 780, in resolve
    replace_conflicting=replace_conflicting
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 1063, in best_match
    return self.obtain(req, installer)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 1075, in obtain
    return installer(requirement)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 653, in fetch_build_egg
    return cmd.easy_install(req)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 679, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 705, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 890, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 1158, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 1144, in run_setup
    run_setup(setup_script, args)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 253, in run_setup
    raise
  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/_vendor/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-j1m6g1u8/pycparser-2.19/setup.py", line 65, in <module>
    # If there's no vectors locally that probably means we are in a tarball and
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 142, in setup
    _install_setup_requires(attrs)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 135, in _install_setup_requires
    dist.parse_config_files(ignore_option_errors=True)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 564, in parse_config_files
    _Distribution.parse_config_files(self, filenames=filenames)
  File "/usr/lib/python3.7/distutils/dist.py", line 397, in parse_config_files
    filenames = self.find_config_files()
  File "/usr/lib/python3.7/distutils/dist.py", line 352, in find_config_files
    sys_dir = os.path.dirname(sys.modules['distutils'].__file__)
KeyError: 'distutils'


pex: Failed to install package at /tmp/tmppt__qt4e/cryptography-2.5: Failed to install /tmp/tmppt__qt4e/cryptography-2.5                                                                                                          
Traceback (most recent call last):
  File "/home/jsirois/.venv/pex/bin/pex", line 10, in <module>
    sys.exit(main())
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/bin/pex.py", line 662, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/bin/pex.py", line 592, in build_pex
    for resolved_dist in resolveds:
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 565, in resolve_multi
    use_manylinux=use_manylinux):
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 501, in resolve
    return resolver.resolve(resolvables_from_iterable(requirements, builder, interpreter=interpreter))
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 299, in resolve
    dist = self.build(package, resolvable.options)
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 378, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 257, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage('file:///home/jsirois/.pex/build/cryptography-2.5.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

The text was updated successfully, but these errors were encountered:

jsirois · 2019-01-31T18:37:37Z

I spiked on this yesterday and ammended the venording code to omit conditional vendored imports of the form:

if "__PEX_UNVENDORED__" in __import__("os").environ:
  import pkg_resources
else:
  import pex.third_party.pkg_resources as pkg_resources

This allows exposing the vendored dists in un-vendored mode to run installs with by:

Chrooting the vendored code - we already do this but now need to expose the chroot paths for 2 below.
Adding the chrooted code to the PYTHONPATH of the installer process.
Invoking the installer process with __PEX_UNVENDORED__=1

This will also remove one of the hardest to grok bits of pex where it concocts a substitute setup.py that sets up sys.path and then hands off to the underlying setup.py.

I'll cleanup that spike and prep a PR.

Now that pex vendors setuptools and wheel, we build / install sdists with potentially fragile import semantics for these two distributions. In particular, we trip up against this when trying to build cryptography. Robustify installs by de-vendoring setuptools and wheel when used in an install context. In order to support de-vendoring make the method of import conditional on an environment variable instead of attempting to re-write vendored imports at runtime. Fixes pex-tool#661

Now that pex vendors setuptools and wheel, we build / install sdists with potentially fragile import semantics for these two distributions. In particular, we trip up against this when trying to build cryptography. Robustify installs by de-vendoring setuptools and wheel when used in an install context. In order to support de-vendoring make the method of import conditional on an environment variable instead of attempting to re-write vendored imports at runtime. Fixes #661

jsirois added the bug label Jan 31, 2019

jsirois self-assigned this Jan 31, 2019

jsirois added the in progress label Jan 31, 2019

jsirois mentioned this issue Jan 31, 2019

Release 1.6.2 #652

Closed

3 tasks

jsirois mentioned this issue Jan 31, 2019

Upgrade to pex 1.6.6. pantsbuild/pants#7186

Merged

jsirois mentioned this issue Feb 6, 2019

Support de-vendoring for installs. #666

Merged

jsirois mentioned this issue Feb 7, 2019

pex does not resolve abi3 wheels correctly #539

Closed

jsirois closed this as completed in #666 Feb 7, 2019

jsirois removed the in progress label Feb 7, 2019

jsirois mentioned this issue Feb 14, 2019

pex 1.6.2 cannot build cryptography 2.5 #671

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pex 1.6.1 cannot build cryptography 2.5 #661

pex 1.6.1 cannot build cryptography 2.5 #661

jsirois commented Jan 31, 2019 •

edited

Loading

jsirois commented Jan 31, 2019 •

edited

Loading

pex 1.6.1 cannot build cryptography 2.5 #661

pex 1.6.1 cannot build cryptography 2.5 #661

Comments

jsirois commented Jan 31, 2019 • edited Loading

jsirois commented Jan 31, 2019 • edited Loading

jsirois commented Jan 31, 2019 •

edited

Loading

jsirois commented Jan 31, 2019 •

edited

Loading