Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pex 1.6.1 cannot build cryptography 2.5 #661

Closed
jsirois opened this issue Jan 31, 2019 · 1 comment
Closed

pex 1.6.1 cannot build cryptography 2.5 #661

jsirois opened this issue Jan 31, 2019 · 1 comment
Assignees
Labels

Comments

@jsirois
Copy link
Member

jsirois commented Jan 31, 2019

The mechanics here are subtle - setuptools does a build isolation dance involving unimports that interacts in a fragile way with its monkey-patching of distutils. We see a failure like so due to the unimporting interaction with the monkeypatches:

$ pex -vvvvvvvvv cryptography==2.5
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7f2fe54f2fd0>
pex: 
Modifying given platform of 'current':
Using the current platform of Platform(platform='linux_x86_64', impl='cp', version='37', abi='cp37m')
Under current interpreter PythonInterpreter('/usr/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 2), {})

To match given interpreter PythonInterpreter('/usr/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 2), {}).

Calculated platform: Platform(platform='linux_x86_64', impl='cp', version='37', abi='cp37m')
pex: R: tags for Platform(platform='linux_x86_64', impl='cp', version='37', abi='cp37m') x PythonInterpreter('/usr/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 2), {}) -> [('cp37', 'cp37m', 'manylinux1_x86_64'), ('cp37', 'cp37m', 'linux_x86_64'), ('cp37', 'abi3', 'manylinux1_x86_64'), ('cp37', 'abi3', 'linux_x86_64'), ('cp37', 'none', 'manylinux1_x86_64'), ('cp37', 'none', 'linux_x86_64'), ('py3', 'none', 'manylinux1_x86_64'), ('py3', 'none', 'linux_x86_64'), ('cp37', 'none', 'any'), ('cp3', 'none', 'any'), ('py37', 'none', 'any'), ('py3', 'none', 'any')]
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7f2fe5497ac8>
pex: crawling link i=0 link=Link('file:///home/jsirois/.pex/build') follow_links=False
pex: crawling link i=0 link=Link('https://pypi.org/simple/cryptography/') follow_links=False
**** Failed to install cryptography-2.5 (caused by: NonZeroExit("received exit code 1 during execution of `['/usr/bin/python3.7', '-sE', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpbq0630tn']` while trying to execute `['/usr/bin/python3.7', '-sE', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpbq0630tn']`")
):
stdout:

stderr:
Traceback (most recent call last):
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-j1m6g1u8/pycparser-2.19/setup.py", line 65, in <module>
    # If there's no vectors locally that probably means we are in a tarball and
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 142, in setup
    _install_setup_requires(attrs)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 135, in _install_setup_requires
    dist.parse_config_files(ignore_option_errors=True)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 564, in parse_config_files
    _Distribution.parse_config_files(self, filenames=filenames)
  File "/usr/lib/python3.7/distutils/dist.py", line 397, in parse_config_files
    filenames = self.find_config_files()
  File "/usr/lib/python3.7/distutils/dist.py", line 352, in find_config_files
    sys_dir = os.path.dirname(sys.modules['distutils'].__file__)
KeyError: 'distutils'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 14, in <module>
  File "<string>", line 324, in <module>
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 142, in setup
    _install_setup_requires(attrs)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 137, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 586, in fetch_build_eggs
    replace_conflicting=True,
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 780, in resolve
    replace_conflicting=replace_conflicting
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 1063, in best_match
    return self.obtain(req, installer)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 1075, in obtain
    return installer(requirement)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 653, in fetch_build_egg
    return cmd.easy_install(req)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 679, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 705, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 890, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 1158, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 1144, in run_setup
    run_setup(setup_script, args)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 253, in run_setup
    raise
  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/_vendor/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-j1m6g1u8/pycparser-2.19/setup.py", line 65, in <module>
    # If there's no vectors locally that probably means we are in a tarball and
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 142, in setup
    _install_setup_requires(attrs)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 135, in _install_setup_requires
    dist.parse_config_files(ignore_option_errors=True)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 564, in parse_config_files
    _Distribution.parse_config_files(self, filenames=filenames)
  File "/usr/lib/python3.7/distutils/dist.py", line 397, in parse_config_files
    filenames = self.find_config_files()
  File "/usr/lib/python3.7/distutils/dist.py", line 352, in find_config_files
    sys_dir = os.path.dirname(sys.modules['distutils'].__file__)
KeyError: 'distutils'


pex: Failed to install package at /tmp/tmppt__qt4e/cryptography-2.5: Failed to install /tmp/tmppt__qt4e/cryptography-2.5                                                                                                          
Traceback (most recent call last):
  File "/home/jsirois/.venv/pex/bin/pex", line 10, in <module>
    sys.exit(main())
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/bin/pex.py", line 662, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/bin/pex.py", line 592, in build_pex
    for resolved_dist in resolveds:
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 565, in resolve_multi
    use_manylinux=use_manylinux):
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 501, in resolve
    return resolver.resolve(resolvables_from_iterable(requirements, builder, interpreter=interpreter))
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 299, in resolve
    dist = self.build(package, resolvable.options)
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 378, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 257, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage('file:///home/jsirois/.pex/build/cryptography-2.5.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)
@jsirois jsirois added the bug label Jan 31, 2019
@jsirois jsirois self-assigned this Jan 31, 2019
@jsirois jsirois mentioned this issue Jan 31, 2019
3 tasks
@jsirois
Copy link
Member Author

jsirois commented Jan 31, 2019

I spiked on this yesterday and ammended the venording code to omit conditional vendored imports of the form:

if "__PEX_UNVENDORED__" in __import__("os").environ:
  import pkg_resources
else:
  import pex.third_party.pkg_resources as pkg_resources

This allows exposing the vendored dists in un-vendored mode to run installs with by:

  1. Chrooting the vendored code - we already do this but now need to expose the chroot paths for 2 below.
  2. Adding the chrooted code to the PYTHONPATH of the installer process.
  3. Invoking the installer process with __PEX_UNVENDORED__=1

This will also remove one of the hardest to grok bits of pex where it concocts a substitute setup.py that sets up sys.path and then hands off to the underlying setup.py.

I'll cleanup that spike and prep a PR.

jsirois added a commit to jsirois/pex that referenced this issue Feb 6, 2019
Now that pex vendors setuptools and wheel, we build / install sdists with
potentially fragile import semantics for these two distributions. In
particular, we trip up against this when trying to build cryptography.

Robustify installs by de-vendoring setuptools and wheel when used in an
install context. In order to support de-vendoring make the method of
import conditional on an environment variable instead of attempting to
re-write vendored imports at runtime.

Fixes pex-tool#661
jsirois added a commit to jsirois/pex that referenced this issue Feb 7, 2019
Now that pex vendors setuptools and wheel, we build / install sdists with
potentially fragile import semantics for these two distributions. In
particular, we trip up against this when trying to build cryptography.

Robustify installs by de-vendoring setuptools and wheel when used in an
install context. In order to support de-vendoring make the method of
import conditional on an environment variable instead of attempting to
re-write vendored imports at runtime.

Fixes pex-tool#661
jsirois added a commit that referenced this issue Feb 7, 2019
Now that pex vendors setuptools and wheel, we build / install sdists with
potentially fragile import semantics for these two distributions. In
particular, we trip up against this when trying to build cryptography.

Robustify installs by de-vendoring setuptools and wheel when used in an
install context. In order to support de-vendoring make the method of
import conditional on an environment variable instead of attempting to
re-write vendored imports at runtime.

Fixes #661
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant