Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade to pex 1.6.6. #7186

Merged
merged 1 commit into from
Apr 16, 2019
Merged

Upgrade to pex 1.6.6. #7186

merged 1 commit into from
Apr 16, 2019

Conversation

jsirois
Copy link
Contributor

@jsirois jsirois commented Jan 30, 2019

This allows us to get rid of resolving setuptools and wheel in our
interpreter cache and generally adapts us to more modern hermetic pex
distributions.

In addition pants now:

  1. Takes over old-style namespace package injection when building pexes.
    Pex used to do this, but really only for Pants use cases. The code
    was stripped from Pex and moved into PexBuilderWrapper.
  2. Exposes the new Pex --emit-warnings option to python_binary.
    By default, Pex warns about various unseemly characteristics of built
    pexes discovered when bootstrapping their run-time. Pants now plumbs
    both a global PythonSetup option and a local python_binary option to
    control the pex warning behavior of pexes it builds.

Fixes #6927

@Eric-Arellano
Copy link
Contributor

All these code changes look reasonable and I don't see an immediate reason why most CI shards would be failing.

Something that does looks fishy to me though is that the Py2 blacklist shards are failing by complaining the command to build the wheel for Cryptography with Python 3 fails: https://travis-ci.org/pantsbuild/pants/jobs/486183205#L531. Note that we know this parent process is being run with Python 2 because it renders a unicode literal u''.

I doubt this is relevant, but one potential thing to sanity check is issues with OpenSSL leading to cryptography issues with Py3. See how we resolve this with OSX shards: https://github.com/pantsbuild/pants/blob/master/.travis.yml#L173.

--

Beyond CI issues, I'm able to run this locally as expected. It also fixes the Python 3.7 DeprecationWarning!

Let me know if you'd like me to test anything specific locally.

@jsirois
Copy link
Contributor Author

jsirois commented Jan 30, 2019

This appears to be deeper than CI env issues. Straight-up pex 1.6.1 fails the same way on my machine.

For pex under python3.7:

$ pex -vvvvvvvvv cryptography==2.5
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7f2fe54f2fd0>
pex: 
Modifying given platform of 'current':
Using the current platform of Platform(platform='linux_x86_64', impl='cp', version='37', abi='cp37m')
Under current interpreter PythonInterpreter('/usr/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 2), {})

To match given interpreter PythonInterpreter('/usr/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 2), {}).

Calculated platform: Platform(platform='linux_x86_64', impl='cp', version='37', abi='cp37m')
pex: R: tags for Platform(platform='linux_x86_64', impl='cp', version='37', abi='cp37m') x PythonInterpreter('/usr/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 2), {}) -> [('cp37', 'cp37m', 'manylinux1_x86_64'), ('cp37', 'cp37m', 'linux_x86_64'), ('cp37', 'abi3', 'manylinux1_x86_64'), ('cp37', 'abi3', 'linux_x86_64'), ('cp37', 'none', 'manylinux1_x86_64'), ('cp37', 'none', 'linux_x86_64'), ('py3', 'none', 'manylinux1_x86_64'), ('py3', 'none', 'linux_x86_64'), ('cp37', 'none', 'any'), ('cp3', 'none', 'any'), ('py37', 'none', 'any'), ('py3', 'none', 'any')]
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7f2fe5497ac8>
pex: crawling link i=0 link=Link('file:///home/jsirois/.pex/build') follow_links=False
pex: crawling link i=0 link=Link('https://pypi.org/simple/cryptography/') follow_links=False
**** Failed to install cryptography-2.5 (caused by: NonZeroExit("received exit code 1 during execution of `['/usr/bin/python3.7', '-sE', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpbq0630tn']` while trying to execute `['/usr/bin/python3.7', '-sE', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpbq0630tn']`")
):
stdout:

stderr:
Traceback (most recent call last):
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-j1m6g1u8/pycparser-2.19/setup.py", line 65, in <module>
    # If there's no vectors locally that probably means we are in a tarball and
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 142, in setup
    _install_setup_requires(attrs)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 135, in _install_setup_requires
    dist.parse_config_files(ignore_option_errors=True)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 564, in parse_config_files
    _Distribution.parse_config_files(self, filenames=filenames)
  File "/usr/lib/python3.7/distutils/dist.py", line 397, in parse_config_files
    filenames = self.find_config_files()
  File "/usr/lib/python3.7/distutils/dist.py", line 352, in find_config_files
    sys_dir = os.path.dirname(sys.modules['distutils'].__file__)
KeyError: 'distutils'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 14, in <module>
  File "<string>", line 324, in <module>
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 142, in setup
    _install_setup_requires(attrs)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 137, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 586, in fetch_build_eggs
    replace_conflicting=True,
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 780, in resolve
    replace_conflicting=replace_conflicting
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 1063, in best_match
    return self.obtain(req, installer)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 1075, in obtain
    return installer(requirement)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 653, in fetch_build_egg
    return cmd.easy_install(req)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 679, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 705, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 890, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 1158, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 1144, in run_setup
    run_setup(setup_script, args)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 253, in run_setup
    raise
  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/usr/lib/python3.7/contextlib.py", line 130, in __exit__
    self.gen.throw(type, value, traceback)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/_vendor/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-j1m6g1u8/pycparser-2.19/setup.py", line 65, in <module>
    # If there's no vectors locally that probably means we are in a tarball and
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 142, in setup
    _install_setup_requires(attrs)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 135, in _install_setup_requires
    dist.parse_config_files(ignore_option_errors=True)
  File "/tmp/tmp9djbzpty/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 564, in parse_config_files
    _Distribution.parse_config_files(self, filenames=filenames)
  File "/usr/lib/python3.7/distutils/dist.py", line 397, in parse_config_files
    filenames = self.find_config_files()
  File "/usr/lib/python3.7/distutils/dist.py", line 352, in find_config_files
    sys_dir = os.path.dirname(sys.modules['distutils'].__file__)
KeyError: 'distutils'


pex: Failed to install package at /tmp/tmppt__qt4e/cryptography-2.5: Failed to install /tmp/tmppt__qt4e/cryptography-2.5                                                                                                          
Traceback (most recent call last):
  File "/home/jsirois/.venv/pex/bin/pex", line 10, in <module>
    sys.exit(main())
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/bin/pex.py", line 662, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/bin/pex.py", line 592, in build_pex
    for resolved_dist in resolveds:
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 565, in resolve_multi
    use_manylinux=use_manylinux):
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 501, in resolve
    return resolver.resolve(resolvables_from_iterable(requirements, builder, interpreter=interpreter))
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 299, in resolve
    dist = self.build(package, resolvable.options)
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 378, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 257, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage('file:///home/jsirois/.pex/build/cryptography-2.5.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

And same for pex 1.6.1 under python2.7:

$ pex -vvvvvvvvv --python=python2.7 cryptography==2.5
pex: creating PythonIdentity from id string: cp cp27mu 27 2 7 15
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7fdcc6a4ad68>
pex: 
Modifying given platform of 'current':
Using the current platform of Platform(platform='linux_x86_64', impl='cp', version='37', abi='cp37m')
Under current interpreter PythonInterpreter('/usr/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 2), {})

To match given interpreter PythonInterpreter('/usr/bin/python2.7', PythonIdentity('cp', 'cp27mu', '27', 2, 7, 15), {}).

Calculated platform: Platform(platform='linux_x86_64', impl='cp', version='27', abi='cp27mu')
pex: R: tags for Platform(platform='linux_x86_64', impl='cp', version='27', abi='cp27mu') x PythonInterpreter('/usr/bin/python2.7', PythonIdentity('cp', 'cp27mu', '27', 2, 7, 15), {}) -> [('cp27', 'cp27mu', 'manylinux1_x86_64'), ('cp27', 'cp27mu', 'linux_x86_64'), ('cp27', 'abi3', 'manylinux1_x86_64'), ('cp27', 'abi3', 'linux_x86_64'), ('cp27', 'none', 'manylinux1_x86_64'), ('cp27', 'none', 'linux_x86_64'), ('py2', 'none', 'manylinux1_x86_64'), ('py2', 'none', 'linux_x86_64'), ('cp27', 'none', 'any'), ('cp2', 'none', 'any'), ('py27', 'none', 'any'), ('py2', 'none', 'any')]
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7fdcc6a53dd8>
pex: crawling link i=0 link=Link('file:///home/jsirois/.pex/build') follow_links=False
pex: crawling link i=0 link=Link('https://pypi.org/simple/cryptography/') follow_links=False
**** Failed to install cryptography-2.5 (caused by: NonZeroExit("received exit code 1 during execution of `['/usr/bin/python2.7', '-sE', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpu_74776o']` while trying to execute `['/usr/bin/python2.7', '-sE', '-', 'bdist_wheel', '--dist-dir=/tmp/tmpu_74776o']`")
):
stdout:

stderr:
/usr/lib/python2.7/distutils/dist.py:378: RuntimeWarning: Parent module 'distutils' not found while handling absolute import
  from ConfigParser import ConfigParser
Traceback (most recent call last):
  File "<stdin>", line 14, in <module>
  File "<string>", line 324, in <module>
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 142, in setup
    _install_setup_requires(attrs)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 137, in _install_setup_requires
    dist.fetch_build_eggs(dist.setup_requires)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 586, in fetch_build_eggs
    replace_conflicting=True,
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 780, in resolve
    replace_conflicting=replace_conflicting
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 1063, in best_match
    return self.obtain(req, installer)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/pkg_resources/__init__.py", line 1075, in obtain
    return installer(requirement)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 653, in fetch_build_egg
    return cmd.easy_install(req)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 679, in easy_install
    return self.install_item(spec, dist.location, tmpdir, deps)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 705, in install_item
    dists = self.install_eggs(spec, download, tmpdir)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 890, in install_eggs
    return self.build_and_install(setup_script, setup_base)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 1158, in build_and_install
    self.run_setup(setup_script, setup_base, args)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/command/easy_install.py", line 1144, in run_setup
    run_setup(setup_script, args)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 253, in run_setup
    raise
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/usr/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 166, in save_modules
    saved_exc.resume()
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 141, in resume
    six.reraise(type, exc, self._tb)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 154, in save_modules
    yield saved
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 195, in setup_context
    yield
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 250, in run_setup
    _execfile(setup_script, ns)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/sandbox.py", line 45, in _execfile
    exec(code, globals, locals)
  File "/tmp/easy_install-kTad_i/pycparser-2.19/setup.py", line 65, in <module>
    # If there's no vectors locally that probably means we are in a tarball and
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 142, in setup
    _install_setup_requires(attrs)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/__init__.py", line 135, in _install_setup_requires
    dist.parse_config_files(ignore_option_errors=True)
  File "/tmp/tmpy87j0wj3/pex/vendor/_vendored/setuptools/setuptools/dist.py", line 564, in parse_config_files
    _Distribution.parse_config_files(self, filenames=filenames)
  File "/usr/lib/python2.7/distutils/dist.py", line 381, in parse_config_files
    filenames = self.find_config_files()
  File "/usr/lib/python2.7/distutils/dist.py", line 348, in find_config_files
    sys_dir = os.path.dirname(sys.modules['distutils'].__file__)
KeyError: 'distutils'


pex: Failed to install package at /tmp/tmp6_r9p571/cryptography-2.5: Failed to install /tmp/tmp6_r9p571/cryptography-2.5                                                                                                          
Traceback (most recent call last):
  File "/home/jsirois/.venv/pex/bin/pex", line 10, in <module>
    sys.exit(main())
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/bin/pex.py", line 662, in main
    pex_builder = build_pex(reqs, options, resolver_options_builder)
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/bin/pex.py", line 592, in build_pex
    for resolved_dist in resolveds:
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 565, in resolve_multi
    use_manylinux=use_manylinux):
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 501, in resolve
    return resolver.resolve(resolvables_from_iterable(requirements, builder, interpreter=interpreter))
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 299, in resolve
    dist = self.build(package, resolvable.options)
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 378, in build
    dist = super(CachingResolver, self).build(package, options)
  File "/home/jsirois/.venv/pex/lib/python3.7/site-packages/pex/resolver.py", line 257, in build
    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
pex.resolver.Untranslateable: Package SourcePackage('file:///home/jsirois/.pex/build/cryptography-2.5.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)

Works though prior to pex 1.6:

$ pex "pex>=1.5,<1.6" -cpex -opex1.5.pex
$ ./pex1.5.pex -vvvvvvvvv cryptography==2.5
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7fcd6026d6a0>
pex: 
Modifying given platform of 'current':
Using the current platform of Platform(platform='linux_x86_64', impl='cp', version='37', abi='cp37m')
Under current interpreter PythonInterpreter('/usr/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 2), {('setuptools', '40.7.1'): '/home/jsirois/.pex/install/setuptools-40.7.1-py2.py3-none-any.whl.2c215d2e757b0b49e0a2952a71815487fca1da0e/setuptools-40.7.1-py2.py3-none-any.whl', ('wheel', '0.31.1'): '/home/jsirois/.pex/install/wheel-0.31.1-py2.py3-none-any.whl.9162405b4c6d47ff554018389d29281b6e4854a6/wheel-0.31.1-py2.py3-none-any.whl', ('pex', '1.5.3'): '/home/jsirois/.pex/install/pex-1.5.3-py2.py3-none-any.whl.477f97d1281b87ebf535f8e21edba62745df644a/pex-1.5.3-py2.py3-none-any.whl'})

To match given interpreter PythonInterpreter('/usr/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 2), {('setuptools', '40.7.1'): '/home/jsirois/.pex/install/setuptools-40.7.1-py2.py3-none-any.whl.2c215d2e757b0b49e0a2952a71815487fca1da0e/setuptools-40.7.1-py2.py3-none-any.whl', ('wheel', '0.31.1'): '/home/jsirois/.pex/install/wheel-0.31.1-py2.py3-none-any.whl.9162405b4c6d47ff554018389d29281b6e4854a6/wheel-0.31.1-py2.py3-none-any.whl', ('pex', '1.5.3'): '/home/jsirois/.pex/install/pex-1.5.3-py2.py3-none-any.whl.477f97d1281b87ebf535f8e21edba62745df644a/pex-1.5.3-py2.py3-none-any.whl'}).

Calculated platform: Platform(platform='linux_x86_64', impl='cp', version='37', abi='cp37m')
pex: R: tags for Platform(platform='linux_x86_64', impl='cp', version='37', abi='cp37m') x PythonInterpreter('/usr/bin/python3.7', PythonIdentity('cp', 'cp37m', '37', 3, 7, 2), {('setuptools', '40.7.1'): '/home/jsirois/.pex/install/setuptools-40.7.1-py2.py3-none-any.whl.2c215d2e757b0b49e0a2952a71815487fca1da0e/setuptools-40.7.1-py2.py3-none-any.whl', ('wheel', '0.31.1'): '/home/jsirois/.pex/install/wheel-0.31.1-py2.py3-none-any.whl.9162405b4c6d47ff554018389d29281b6e4854a6/wheel-0.31.1-py2.py3-none-any.whl', ('pex', '1.5.3'): '/home/jsirois/.pex/install/pex-1.5.3-py2.py3-none-any.whl.477f97d1281b87ebf535f8e21edba62745df644a/pex-1.5.3-py2.py3-none-any.whl'}) -> [('cp37', 'cp37m', 'manylinux1_x86_64'), ('cp37', 'cp37m', 'linux_x86_64'), ('cp37', 'abi3', 'manylinux1_x86_64'), ('cp37', 'abi3', 'linux_x86_64'), ('cp37', 'none', 'manylinux1_x86_64'), ('cp37', 'none', 'linux_x86_64'), ('py3', 'none', 'manylinux1_x86_64'), ('py3', 'none', 'linux_x86_64'), ('cp37', 'none', 'any'), ('cp3', 'none', 'any'), ('py37', 'none', 'any'), ('py3', 'none', 'any')]
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7fcd6026dbe0>
pex: crawling link i=0 link=Link('file:///home/jsirois/.pex/build') follow_links=False
pex: crawling link i=0 link=Link('https://pypi.org/simple/cryptography/') follow_links=False
pex: Warning, using a UrllibContext which is known to be flaky.                                                                                                                     
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7fcd5ecbab70>
pex: crawling link i=0 link=Link('https://pypi.org/simple/six/') follow_links=False
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7fcd5ecbab70>
pex: crawling link i=0 link=Link('https://pypi.org/simple/asn1crypto/') follow_links=False
pex: Warning, using a UrllibContext which is known to be flaky.
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7fcd5ecbab70>
pex: crawling link i=0 link=Link('https://pypi.org/simple/cffi/') follow_links=False
pex: Warning, using a UrllibContext which is known to be flaky.                                                                                                                                                                                                                         
pex: Please build pex with the requests module for more reliable downloads.
pex: Constructed UrllibContext context <pex.http.UrllibContext object at 0x7fcd5eb6bc50>
pex: crawling link i=0 link=Link('https://pypi.org/simple/pycparser/') follow_links=False
  cryptography==2.5 -> cryptography 2.5tions :: Translating /tmp/tmpc9uup96s/pycparser-2.19.tar.gz into distribution :: Packaging pycparser :: Installing /tmp/tmpohriwdc6      
  six>=1.4.1 -> six 1.12.0
  asn1crypto>=0.21.0 -> asn1crypto 0.24.0
  cffi!=1.11.3,>=1.8 -> cffi 1.11.5
  pycparser -> pycparser 2.19
pex: Building pex: 24856.7ms                                                                                                                                              
pex:   Resolving interpreters: 0.0ms
pex:   Setting up interpreter /usr/bin/python3.7: 1.7ms
pex:   Resolving distributions: 24853.8ms
pex:     Fetching file:///home/jsirois/.pex/build/cryptography-2.5.tar.gz: 6.0ms
pex:       Fetching file:///home/jsirois/.pex/build/cryptography-2.5.tar.gz: 5.7ms
pex:     Translating /tmp/tmpc0gr_toj/cryptography-2.5.tar.gz into distribution: 18254.6ms
pex:       Packaging cryptography: 18208.2ms
pex:         Installing /tmp/tmp6jvon8d5: 18206.1ms
pex:     Fetching file:///home/jsirois/.pex/build/six-1.12.0-py2.py3-none-any.whl: 1.4ms
pex:       Fetching file:///home/jsirois/.pex/build/six-1.12.0-py2.py3-none-any.whl: 1.2ms
pex:     Translating /tmp/tmpm_79mev6/six-1.12.0-py2.py3-none-any.whl into distribution: 0.7ms
pex:     Fetching file:///home/jsirois/.pex/build/asn1crypto-0.24.0-py2.py3-none-any.whl: 1.8ms
pex:       Fetching file:///home/jsirois/.pex/build/asn1crypto-0.24.0-py2.py3-none-any.whl: 1.5ms
pex:     Translating /tmp/tmps80p258h/asn1crypto-0.24.0-py2.py3-none-any.whl into distribution: 0.9ms
pex:     Fetching https://files.pythonhosted.org/packages/51/7b/d1014289d0578c3522b2798b9cb87c65e5b36798bd3ae68a75fa1fe09e78/cffi-1.11.5-cp37-cp37m-manylinux1_x86_64.whl#sha256=a6a5cb8809091ec9ac03edde9304b3ad82ad4466333432b16d78ef40e0cce0d5: 726.1ms
pex:     Fetching file:///home/jsirois/.pex/build/cffi-1.11.5-cp37-cp37m-manylinux1_x86_64.whl: 0.8ms
pex:       Fetching file:///home/jsirois/.pex/build/cffi-1.11.5-cp37-cp37m-manylinux1_x86_64.whl: 0.6ms
pex:     Translating /tmp/tmpiw3sgoig/cffi-1.11.5-cp37-cp37m-manylinux1_x86_64.whl into distribution: 0.9ms
pex:     Fetching file:///home/jsirois/.pex/build/pycparser-2.19.tar.gz: 2.5ms
pex:       Fetching file:///home/jsirois/.pex/build/pycparser-2.19.tar.gz: 2.1ms
pex:     Translating /tmp/tmpc9uup96s/pycparser-2.19.tar.gz into distribution: 411.1ms
pex:       Packaging pycparser: 385.9ms
pex:         Installing /tmp/tmpohriwdc6: 384.5ms
Running PEX file at /tmp/tmpvhe7_row with args []
pex: PEX.run invoking /usr/bin/python3.7 /tmp/tmpvhe7_row
Python 3.7.2 (default, Jan 10 2019, 23:51:51) 
[GCC 8.2.1 20181127] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> 
now exiting InteractiveConsole...

The issue then must have to do with the improved isolation in pex 1.6+; either the new more constrained python -sE invocation of setup.pys or else the new isolation of pkg_resources inside pex from the sys.path.

Copy link
Contributor

@benjyw benjyw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't comment on what might be causing CI issues, but the code looks great, and I love the simplification of interpreter cache.

@jsirois
Copy link
Contributor Author

jsirois commented Jan 31, 2019

OK - The cryptography thing was a doozy. I'll need to fix this over in pex-tool/pex#661, release 1.6.2 and then return to this PR.

@jsirois jsirois force-pushed the issues/6927 branch 2 times, most recently from b348e97 to b350621 Compare February 14, 2019 20:32
@jsirois jsirois changed the title Upgrade to pex 1.6.1. Upgrade to pex 1.6.2. Feb 14, 2019
@jsirois
Copy link
Contributor Author

jsirois commented Feb 15, 2019

Both production code and a few tests assert that the stderr of pex execution is empty. Since pex now warns (to stderr) when setuptools is used but not declared as a dependency, we have some red. More fixes coming...

@jsirois jsirois force-pushed the issues/6927 branch 3 times, most recently from 5c4614e to 0237fa9 Compare February 22, 2019 03:06
@jsirois
Copy link
Contributor Author

jsirois commented Feb 25, 2019

Although I'd argue some to all of the stderr sensitivity in these tests is unwarranted, I've filed pex-tool/pex#677 to add a knob for pex runtime warning suppression. I'll do a small release with that and consume here.

Eric-Arellano added a commit that referenced this pull request Mar 7, 2019
…erpreter constraints requested (#7285)

### Problem
When running `./pants binary`, we do not consider the global interpreter constraints in passing them to the Pex builder.

This resulted in a bug where the interpreter used to build a PEX in CI differed from the Python used to run the PEX. See pex-tool/pex#676. To resolve this issue, we need some way to specify the Pex builder must use the specific Python version 2.7.13 (UCS4). This blocks 
#7235.

### Solution
Modify `PexBuilderWrapper` to use `compatibility_or_constrains()`. This will first try to get the compatibility constraints from the target itself, and then resort to using the PythonSetup subsystem's constraints (for the global instance, resolved from `pants.ini`, `PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS`, and `--interpreter-constraints`).

### Result
Users of the function `add_interpreter_constraints_from()` will now add the global `--interpreter-constraints` to the Pex builder if the targets do not themselves specify a constraint. At the moment, this only impacts `./pants binary`, as `python_binary_create.py` is the only file which uses this function.

Note this is still not a great solution. It would be better to kill `add_interpreter_constraint()` and `add_interpreter_constraints_from()` to instead automatically set the interpreter constraints from the targets' graph. This PR does not make that change to avoid scope creep.

#### Skipped tests
Due to pex-tool/pex#655, we must now skip several tests that now fail. PEX does not correctly OR interpreter constraints, so these tests complain that they cannot find an appropriate interpreter to satisfy `['CPython>=3.6,<4', 'CPython>=2.7,<3']`.

Because this PR blocks #7235, which blocks #7197, we skip these tests until the upstream fix pex-tool/pex#678 is merged into PEX and Pants upgrades its PEX to the new version #7186.
@jsirois jsirois changed the title Upgrade to pex 1.6.2. Upgrade to pex 1.6.3. Mar 22, 2019
@jsirois
Copy link
Contributor Author

jsirois commented Mar 26, 2019

Ran into pex-tool/pex#684 here - the warnings fix broke lambdex :/

@jsirois jsirois mentioned this pull request Mar 26, 2019
3 tasks
@jsirois jsirois force-pushed the issues/6927 branch 2 times, most recently from c20a09d to 1635569 Compare March 29, 2019 01:55
@jsirois jsirois changed the title Upgrade to pex 1.6.3. Upgrade to pex 1.6.5. Mar 30, 2019
Copy link
Member

@stuhood stuhood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks John!

build-support/bin/release.sh Show resolved Hide resolved
@jsirois
Copy link
Contributor Author

jsirois commented Apr 3, 2019

OK... the python 3.6 hangs are actually infinite pex re-exec loops encountered when executing pants.pex using an interpreter (our pyenv 3.6 shim) that is not on the PATH: pex-tool/pex#700

I'll cut a release with a fix for that and consume pex 1.6.6.

@jsirois jsirois changed the title Upgrade to pex 1.6.5. Upgrade to pex 1.6.6. Apr 3, 2019
@jsirois jsirois force-pushed the issues/6927 branch 2 times, most recently from d08fa8c to 23dbe44 Compare April 5, 2019 03:24
@jsirois
Copy link
Contributor Author

jsirois commented Apr 5, 2019

Using interpreter constraints to attempt to steer interpreter selection is causing alot of pain. We know the exact interpreter to use and should be able to tell pants this (you can tell pex this). Filed #7508 to make this better going forward though I doubt I'll hold up this PR for it.

This allows us to get rid of resolving setuptools and wheel in our
interpreter cache.

Fixes pantsbuild#6927
@jsirois
Copy link
Contributor Author

jsirois commented Apr 14, 2019

Rebased after breaking out independent changes. A CI burn is in progress and once green, I'll circle back and flesh out the PR description a bit to describe both major fixes / features brought by the pex upgrade that Pants will leverage / benefit from as well as describing the major updates to the Pants codebase in this PR.

Copy link
Contributor

@Eric-Arellano Eric-Arellano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great to see this green! Thanks for all your work to land this, John.

Will review the PR description once that's ready.

Copy link
Member

@stuhood stuhood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just nits. Fire away!

src/python/pants/bin/BUILD Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Upgrade to pex 1.6.x
4 participants