Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

pex does not resolve abi3 wheels correctly #539

Closed
zsimic opened this issue Aug 16, 2018 · 7 comments
Closed

pex does not resolve abi3 wheels correctly #539

zsimic opened this issue Aug 16, 2018 · 7 comments
Assignees
Labels

Comments

@zsimic
Copy link

zsimic commented Aug 16, 2018

pex doesn't seem to resolve abi3 wheels correctly. I have a project that depends on cryptography, which seems to have started publishing abi3 wheels on March 20 2018 (v2.2)

At first I thought this was missing wheels from cryptography (see pyca/cryptography#4404), but actually the python 3.4 abi3 wheel should cover python 3.6 for example (which I'm using).

I pex package my project like so:

pip --cache-dir dist wheel --wheel-dir dist .
pex --no-pypi --cache-dir dist --repo dist -cmyproject -omyproject.pex myproject

If I pin cryptography to 2.1.4 (last version that published non-abi3 wheels), pex packaging above works.

If I try latest cryptography version, pip wheel correctly finds cryptography-2.3.1-cp34-abi3-macosx_10_6_intel.whl, however pex complains with Could not satisfy all requirements for cryptography

My environment:

  • OSX 10.13.6
  • python 3.6.6
  • pex 1.4.5
  • pip 18.0
@clintharrison
Copy link
Contributor

From what I can tell, when pex is doing dependency resolution, pep425tags.get_supported() is always called with a version argument. In this case, that'd be '36'; and the code does not reach the branch where previous minor versions are appended.

It's also somewhat unfortunate in that the abi3 tag will not be added if pex is running under py2 resolving for py3 distributions, since this code assumes it can use imp.get_suffixes() from the active interpreter.

@kwlzn et al., are there any concerns with further diverging from the pep425tags from upstream pip? I think modifying get_supported() to always expand back the version range would be the simplest solution.

@jsjeannotte
Copy link

Any update on the issue?

@piger
Copy link

piger commented Feb 7, 2019

Indeed by inverting the if here https://github.com/pantsbuild/pex/blob/fe405f9965dd1698095dde8241c07f8ef2a41517/pex/pep425tags.py#L262 (i.e. if version is not None) the versions list gets populated with all 3.x versions (['37', '36', '35', '34', '33', '32', '31', '30']), which allows pex to find the correct wheel for cryptography 2.5.

pex and pip end up downloading the same wheels:

pex:     Fetching file:///Users/user/.pex/build/cryptography-2.5-cp34-abi3-macosx_10_6_intel.whl: 9.0ms
pex:       Fetching file:///Users/user/.pex/build/cryptography-2.5-cp34-abi3-macosx_10_6_intel.whl: 8.1ms
pex:     Translating /private/var/folders/1v/cdy4c5112fn_vc18qvzblhxc0000gp/T/tmpfw7w1r38/cryptography-2.5-cp34-abi3-macosx_10_6_intel.whl into distribution: 8.0ms
pex:     Fetching file:///Users/user/.pex/build/asn1crypto-0.24.0-py2.py3-none-any.whl: 8.2ms
pex:       Fetching file:///Users/user/.pex/build/asn1crypto-0.24.0-py2.py3-none-any.whl: 1.3ms
pex:     Translating /private/var/folders/1v/cdy4c5112fn_vc18qvzblhxc0000gp/T/tmp4btiyiqd/asn1crypto-0.24.0-py2.py3-none-any.whl into distribution: 1.8ms
pex:     Fetching file:///Users/user/.pex/build/cffi-1.11.5-cp37-cp37m-macosx_10_9_x86_64.whl: 1.7ms
pex:       Fetching file:///Users/user/.pex/build/cffi-1.11.5-cp37-cp37m-macosx_10_9_x86_64.whl: 0.9ms
pex:     Translating /private/var/folders/1v/cdy4c5112fn_vc18qvzblhxc0000gp/T/tmpd17j7mhy/cffi-1.11.5-cp37-cp37m-macosx_10_9_x86_64.whl into distribution: 1.6ms
pex:     Fetching file:///Users/user/.pex/build/six-1.12.0-py2.py3-none-any.whl: 1.9ms
pex:       Fetching file:///Users/user/.pex/build/six-1.12.0-py2.py3-none-any.whl: 0.9ms
pex:     Translating /private/var/folders/1v/cdy4c5112fn_vc18qvzblhxc0000gp/T/tmp4ha57vdq/six-1.12.0-py2.py3-none-any.whl into distribution: 2.6ms
$ pip install -U cryptography
Collecting cryptography
  Using cached https://files.pythonhosted.org/packages/d7/9e/12bb10fd009b0146935c169cc0e1e86221eacf8dc207990d54b783c47a7d/cryptography-2.5-cp34-abi3-macosx_10_6_intel.whl
Collecting asn1crypto>=0.21.0 (from cryptography)
  Using cached https://files.pythonhosted.org/packages/ea/cd/35485615f45f30a510576f1a56d1e0a7ad7bd8ab5ed7cdc600ef7cd06222/asn1crypto-0.24.0-py2.py3-none-any.whl
Collecting cffi!=1.11.3,>=1.8 (from cryptography)
  Using cached https://files.pythonhosted.org/packages/0b/ba/32835c9965d8a0090723e1d0b47373365525c4bd08c807b5efdc9fecbc99/cffi-1.11.5-cp37-cp37m-macosx_10_9_x86_64.whl
Collecting six>=1.4.1 (from cryptography)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl

Tested on:

  • OS X: 10.13.6
  • pex: 1.6.1 (master branch)
  • pip 19.0.1
  • Python 3.7.1

@jsirois jsirois self-assigned this Feb 7, 2019
@jsirois
Copy link
Member

jsirois commented Feb 7, 2019

Thanks for debugging a bit @piger. Although your fix works - it clearly goes against what the conditional is trying to do. I think the answer is to dig and see who / why passes version. I'll dig in here since I just finished dealing with pex 1.6.x errors building cryptography in #661 / #666 so this is fresh on my mind.

@jsirois jsirois mentioned this issue Feb 7, 2019
3 tasks
@jsirois
Copy link
Member

jsirois commented Feb 7, 2019

Note to self: making sure #492 continues to be fixed is important here.

@piger
Copy link

piger commented Feb 8, 2019

@jsirois I know, it was mostly to get a sense of what was going on and try to resume the conversation :) Thanks a lot for working on the other issue!

@piger
Copy link

piger commented Feb 15, 2019

I'm digging a bit more into how pex and pip behavior differs when it comes to install a abi3 wheel like "cryptography", in particular in those code sections that are shared between the 2 projects. I'm doing my tests on Python 3.7.1.

The wheel file that I'm trying to install is cryptography-2.5-cp34-abi3-macosx_10_6_intel.whl (note the cp34 tag).

It seems that pip does not specify any python_version when it tries to lookup (resolve?) the wheel for a package, meaning that the "supported tags" generated by pip include older versions of python, e.g.

  • ('cp34', 'abi3', 'macosx_10_11_x86_64')
  • ('cp37', 'abi3', 'macosx_10_1_universal')
  • ('py3', 'none', 'any')
  • ('cp32', 'abi3', 'macosx_10_8_universal')

while the "supported tags" generated by pex only includes the current interpreter version, e.g. ('cp37', 'abi3', 'macosx_10_1_universal') and similar. pex gets the supported tags for a given requirement by instantiating a Platform instance.

To have pex behave more like pip in this scenario, it could resolve dependencies limiting them to a python version only when a --python-version command line option is specified.

Does this make any sense? Apologies in advance if it doesn't, but this bug is blocking a py3 porting that I'm doing, enough to trigger my will to try to help :)

@jsirois jsirois mentioned this issue Mar 10, 2019
3 tasks
@Eric-Arellano Eric-Arellano self-assigned this Mar 21, 2019
@jsirois jsirois mentioned this issue Mar 26, 2019
1 task
jsirois added a commit to jsirois/pex that referenced this issue Mar 28, 2019
Previously, we'd pass a single version to `get_supported` when we knew
one which would lead to a single tag being generated for the abi3 case
instead of one per previous minor version.

Fixes pex-tool#539
jsirois added a commit to jsirois/pex that referenced this issue Mar 29, 2019
Previously, we'd pass a single version to `get_supported` when we knew
one which would lead to a single tag being generated for the abi3 case
instead of one per previous minor version.

Fixes pex-tool#539
jsirois added a commit to jsirois/pex that referenced this issue Mar 29, 2019
Previously, we'd pass a single version to `get_supported` when we knew
one which would lead to a single tag being generated for the abi3 case
instead of one per previous minor version.

Fixes pex-tool#539
jsirois added a commit that referenced this issue Mar 30, 2019
Previously, we'd pass a single version to `get_supported` when we knew
one which would lead to a single tag being generated for the abi3 case
instead of one per previous minor version.

As part of this change, we need a more modern version of the
pep425tags.py we had copied from pip that allows passing multiple
versions to `get_supported`. Ideally, we'd use the version embedded in
our vendored setuptools code and this is tracked by #696. Extra code
had been added to pep425tags to implement `get_supported_for_any_abi`
and handle manylinux. This is now inlined in platforms.py, which was
the only user,  the door to using undisturbed copies in #696.

Fixes #539
@Eric-Arellano Eric-Arellano removed their assignment Dec 27, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

7 participants