Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Multi-interpreter feature doesn't work perfectly #415

Closed
lyw07 opened this issue Sep 7, 2017 · 2 comments · Fixed by #788
Closed

Multi-interpreter feature doesn't work perfectly #415

lyw07 opened this issue Sep 7, 2017 · 2 comments · Fixed by #788
Assignees

Comments

@lyw07
Copy link

lyw07 commented Sep 7, 2017

Summary

I'm trying to create a multi-interpreter pex on ubuntu for cryptography that works on mac. cryptography does not contain enum34 and ipaddress in Python3 but does in Python2. However, it seems that pex couldn't figure out that, so it prints an error saying "enum34 and ipaddress are missing".

Validation

For Python 2.7 cryptography:

$ ls -l python2.7_crypto/
total 1152
-rwxr-xr-x 1 lingyiwang staff 348680 Sep 7 15:31 _cffi_backend.so
drwxr-xr-x 53 lingyiwang staff 1802 Sep 7 15:31 asn1crypto
drwxr-xr-x 9 lingyiwang staff 306 Sep 7 15:31 asn1crypto-0.22.0.dist-info
drwxr-xr-x 35 lingyiwang staff 1190 Sep 7 15:31 cffi
drwxr-xr-x 10 lingyiwang staff 340 Sep 7 15:31 cffi-1.10.0.dist-info
drwxr-xr-x 14 lingyiwang staff 476 Sep 7 15:31 cryptography
drwxr-xr-x 9 lingyiwang staff 306 Sep 7 15:31 cryptography-2.0.3.dist-info
drwxr-xr-x 6 lingyiwang staff 204 Sep 7 15:31 enum
drwxr-xr-x 9 lingyiwang staff 306 Sep 7 15:31 enum34-1.1.6.dist-info
drwxr-xr-x 18 lingyiwang staff 612 Sep 7 15:31 idna
drwxr-xr-x 9 lingyiwang staff 306 Sep 7 15:31 idna-2.6.dist-info
drwxr-xr-x 9 lingyiwang staff 306 Sep 7 15:31 ipaddress-1.0.18.dist-info
-rw-r--r-- 1 lingyiwang staff 80156 Sep 7 15:31 ipaddress.py
-rw-r--r-- 1 lingyiwang staff 84286 Sep 7 15:31 ipaddress.pyc
drwxr-xr-x 26 lingyiwang staff 884 Sep 7 15:31 pycparser
drwxr-xr-x 9 lingyiwang staff 306 Sep 7 15:31 pycparser-2.18.dist-info
drwxr-xr-x 9 lingyiwang staff 306 Sep 7 15:31 six-1.10.0.dist-info
-rw-r--r-- 1 lingyiwang staff 30098 Sep 7 15:31 six.py
-rw-r--r-- 1 lingyiwang staff 33653 Sep 7 15:31 six.pyc

For Python 3.6 cryptography

ls -l python3.6_crypto/
total 760
drwxr-xr-x 3 lingyiwang staff 102 Sep 7 15:33 pycache
-rwxr-xr-x 1 lingyiwang staff 353096 Sep 7 15:33 _cffi_backend.cpython-36m-darwin.so
drwxr-xr-x 29 lingyiwang staff 986 Sep 7 15:33 asn1crypto
drwxr-xr-x 9 lingyiwang staff 306 Sep 7 15:33 asn1crypto-0.22.0.dist-info
drwxr-xr-x 21 lingyiwang staff 714 Sep 7 15:33 cffi
drwxr-xr-x 10 lingyiwang staff 340 Sep 7 15:33 cffi-1.10.0.dist-info
drwxr-xr-x 10 lingyiwang staff 340 Sep 7 15:33 cryptography
drwxr-xr-x 9 lingyiwang staff 306 Sep 7 15:33 cryptography-2.0.3.dist-info
drwxr-xr-x 11 lingyiwang staff 374 Sep 7 15:33 idna
drwxr-xr-x 9 lingyiwang staff 306 Sep 7 15:33 idna-2.6.dist-info
drwxr-xr-x 25 lingyiwang staff 850 Sep 7 15:33 pycparser
drwxr-xr-x 9 lingyiwang staff 306 Sep 7 15:33 pycparser-2.18.dist-info
drwxr-xr-x 9 lingyiwang staff 306 Sep 7 15:33 six-1.10.0.dist-info
-rw-r--r-- 1 lingyiwang staff 30098 Sep 7 15:33 six.py

Reproduction

Pex version: 1.2.11

  • Get the pex generated on ubuntu using the command:
    pex --disable-cache cryptography --platform=macosx-10.6-intel --python=python2.7 --python=python3.6 -o test.pex

  • Run test.pex on mac using the command:
    python3.6 test.pex

    This will print the error:

    Failed to execute PEX file, missing compatible dependencies for:
    ipaddress
    enum34

    Note that if we run test.pex using Python 2.7, everything works fine. Also, if this pex file is generated on mac instead of ubuntu, it works with Python 2.7 and Python 3.6 perfectly.

Thank you so much for your help!
(and sorry that I can't come up with a better title to describe my problem)

@jsirois
Copy link
Member

jsirois commented Jan 30, 2019

A year and a half later - agreed! This is still an issue and was recently re-discovered by Pants in the investigation of pantsbuild/pants#7158

To put a finer point on things, for this historical case, we have:

  1. https://pypi.org/project/ipaddress/1.0.18/#files
    • ipaddress-1.0.18-py2-none-any.whl
    • ipaddress-1.0.18.tar.gz
  2. https://pypi.org/project/enum34/1.1.6/#files
    • enum34-1.1.6-py2-none-any.whl
    • enum34-1.1.6-py3-none-any.whl
    • enum34-1.1.6.tar.gz
    • enum34-1.1.6.zip

When resolving for multiple interpreters the buggy logic is here:
https://github.com/pantsbuild/pex/blob/302f98844487d894252d3dc3f4d30940fbcbd9e1/pex/resolver.py#L553-L568
https://github.com/pantsbuild/pex/blob/8fd5e1b989224046cd0fb038cfebc11b4c10a8c9/pex/resolver.py#L277-L303

For one, a single sdist package will not be built more than once; ie not for each interpreter needed.

@jsirois
Copy link
Member

jsirois commented Jan 30, 2019

I'll work this issue under #658 which is a bit more finely pointed and close both when / if that issue resolves the problem noted here in the original description.

@jsirois jsirois mentioned this issue Feb 11, 2019
2 tasks
@jsirois jsirois mentioned this issue Mar 10, 2019
3 tasks
@jsirois jsirois self-assigned this Nov 14, 2019
jsirois added a commit that referenced this issue Nov 14, 2019
This changes Pex to vendor pip and defer all resolution and building to it.

As part of this change some ambiguous and / or broken features were changed and APIs adjusted:
+ The egg distribution format is no longer supported.
+ The deprecated `--interpreter-cache-dir` CLI option was removed.
+ The `--cache-ttl` CLI option and `cache_ttl` resolver API argument were removed.
+ The resolver API replaced `fetchers` with a list of `indexes` and a list of `find_links` repos.
+ The resolver API removed (http) `context` which is now automatically handled.
+ The resolver API removed `precedence` which is now pip default precedence - wheels when available and not ruled out via the `--no-wheel` CLI option or `use_wheel=False` API argument.
+ The `--platform` CLI option and `platform` resolver API argument now must be full platform strings that include platform, implementation, version and abi; e.g.: `--platform=macosx-10.13-x86_64-cp-36-m`.
+ The `--manylinux` CLI option and `use_manylinux` resolver API argument were removed. Instead, to resolve manylinux wheels for a foreign platform, specify the manylinux platform to use with an explicit `--platform` CLI flag or `platform` resolver API argument; e.g.: `--platform=manylinux2010-x86_64-cp-36-m`.

Fixes #781

Additionally:
Fixes #771
Fixes #763
Fixes #761
Fixes #735 
Fixes #694 
Fixes #660
Fixes #658 
Fixes #642
Fixes #641
Fixes #628
Fixes #620
Fixes #614
Fixes #611
Fixes #608
Fixes #439
Fixes #415
Fixes #387
Fixes #315
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants