Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to build an OSX pex from prebuilt pants wheels #7383

Closed
stuhood opened this issue Mar 15, 2019 · 8 comments
Closed

Unable to build an OSX pex from prebuilt pants wheels #7383

stuhood opened this issue Mar 15, 2019 · 8 comments
Labels
bug

Comments

@stuhood
Copy link
Member

stuhood commented Mar 15, 2019

Running (on master on an OSX 10.14 machine):

$ rm -Rf dist ; ./build-support/bin/release.sh -p

...results in a pex that cannot be consumed on 10.14:

$ ./dist/pants.*.pex
Failed to execute PEX file, missing macosx_10_14_x86_64-cp-27-cp27m compatible dependencies for:
thriftpy2
cryptography
psutil
pyopenssl
subprocess32
pywatchman
pantsbuild.pants
faulthandler
py-zipkin
requests

Bisecting on whether that command succeeds (and skipping commits that didn't get wheels built for them) results in:

There are only 'skip'ped commits left to test.
The first bad commit could be any of:
	9873b1412b7e343142353b57357dae09895386d2
	7036c5055a4ec95a07507ea093b960491caaa338
	7b2225e0e0e004db6d99abf14329c6bccac65bf8
	63d430e6da1817f03999d985bcaad24a2292b6e0
We cannot bisect more!

Of those, #7235 (7b2225e) is the most likely to be related.
@stuhood stuhood added the bug label Mar 15, 2019
@stuhood
Copy link
Member Author

stuhood commented Mar 15, 2019

I've diffed the resulting 3rdparty wheels from working and non-working commits, and the diff indicates that we didn't lose any wheels... only gained them.

So I'm going to explore the possibility that something in ./build-support/bin/release.sh -p changed.

@Eric-Arellano
Copy link
Contributor

Yes, #7235 is almost certainly the culprit here.

My hypothesis would be that we should be enumerating the expected ABIs here: https://github.com/pantsbuild/pants/blob/master/build-support/bin/release.sh#L594. I don't have time to look closely, but note I did not modify that function at all and it appears likely we should be fetching wheels differently to keep in mind that we have (now) six wheels built.

I don't know what the expected behavior is. Is our Pex supposed to ship every single potential dependency needed, i.e. cp27m and cp27mu wheels for wheels like psutil, all in the same Pex? And then the user's system picks out which wheels it needs to work from there?

@stuhood
Copy link
Member Author

stuhood commented Mar 15, 2019 •
edited
Loading

So, the input wheels are ~identical, and the pex invoke is ~identical:

before (working):
	/var/folders/91/bym4mtpx1p503xmvpp8fbdl40000gp/T/build_pex.XXXXX.x8dOECB6/pex27 \
	  --no-build \
	  --no-pypi \
	  --disable-cache \
	  -f /Users/stuhood/src/pants/dist/deploy/wheels/pantsbuild.pants/e65723a346e91cf6f6e1fec2cc1f025acef6352c/1.15.0.dev2+e65723a3 \
	  -f /Users/stuhood/src/pants/dist/deploy/wheels/3rdparty/e65723a346e91cf6f6e1fec2cc1f025acef6352c/1.15.0.dev2+e65723a3 \
	  -o /Users/stuhood/src/pants/dist/pants.1.15.0.dev2+e65723a3.pex \
	  --script=pants \
	  --platform=linux_x86_64 \
	  --platform=macosx_10.11_x86_64 \
	  pantsbuild.pants==1.15.0.dev2+e65723a3

after (broken):
	/var/folders/91/bym4mtpx1p503xmvpp8fbdl40000gp/T/build_pex.XXXXX.C7UeS0gb/pex27 \
	  --no-build \
	  --no-pypi \
	  --disable-cache \
	  -f /Users/stuhood/src/pants/dist/deploy/wheels/pantsbuild.pants/4b3b7b7ffdff414ebedc8d9569deec7cbac29db3/1.15.0.dev3+4b3b7b7f \
	  -f /Users/stuhood/src/pants/dist/deploy/wheels/3rdparty/4b3b7b7ffdff414ebedc8d9569deec7cbac29db3/1.15.0.dev3+4b3b7b7f \
	  -o /Users/stuhood/src/pants/dist/pants.1.15.0.dev3+4b3b7b7f.pex \
	  --script=pants \
	  --platform=linux_x86_64 \
	  --platform=macosx_10.11_x86_64 \
	  pantsbuild.pants==1.15.0.dev3+4b3b7b7f

... mysterious.

@stuhood
Copy link
Member Author

stuhood commented Mar 15, 2019

Will investigate using PEX_VERBOSE=9 for the pex-build invoke and the pex invoke.

@Eric-Arellano
Copy link
Contributor

@stuhood are you certain the wheels are identical at /Users/stuhood/src/pants/dist/deploy/wheels/3rdparty/4b3b7b7ffdff414ebedc8d9569deec7cbac29db3/1.15.0.dev3+4b3b7b7f? And if so, that might be the issue, that I don't think they should be identical! Before should have less wheels than after.

I would expect for platform specific wheels that we should have 6 wheels for each platform specific dependency like pantsbuild.pants: cp36-abi3, cp27-cpm, and cp27-cpmu. If this is not the case, then we would not be distributing every wheel we need for the Pex to work on any system.

@stuhood
Copy link
Member Author

stuhood commented Mar 15, 2019

@stuhood are you certain the wheels are identical at /Users/stuhood/src/pants/dist/deploy/wheels/3rdparty/4b3b7b7ffdff414ebedc8d9569deec7cbac29db3/1.15.0.dev3+4b3b7b7f? And if so, that might be the issue, that I don't think they should be identical! Before should have less wheels than after.

Not identical, but see: https://gist.github.com/stuhood/54a399dcf711a02396bc5bfc9be505ac

@stuhood
Copy link
Member Author

stuhood commented Mar 15, 2019

PEX_VERBOSE for the unloadable pex claims:

pex: Failed to resolve a requirement: The 'cryptography>=1.9' distribution was not found and is required by pyopenssl
pex: Failed to resolve a requirement: The 'cryptography>=1.9' distribution was not found and is required by the application
pex: Failed to resolve a requirement: The 'subprocess32==3.2.7; python_version < "3"' distribution was not found and is required by the application
pex: Failed to resolve a requirement: The 'pantsbuild.pants==1.15.0.dev3+c835267b' distribution was not found and is required by the application
pex: Failed to resolve a requirement: The 'pywatchman==1.4.1' distribution was not found and is required by the application
pex: Failed to resolve a requirement: The 'faulthandler==2.6; python_version < "3"' distribution was not found and is required by the application
pex: Failed to resolve a requirement: The 'cryptography>=1.3.4; extra == "security"' distribution was not found and is required by requests
pex: Unresolved requirements:
pex:   - cryptography
pex:   - pyopenssl
pex:   - subprocess32
pex:   - pywatchman
pex:   - pantsbuild.pants
pex:   - faulthandler
pex:   - requests
pex: Distributions contained within this pex:
pex:   - pex-1.5.3-py2.py3-none-any.whl
pex:   - requests-2.21.0-py2.py3-none-any.whl
pex:   - subprocess32-3.2.7-cp27-cp27m-linux_x86_64.whl
pex:   - idna-2.8-py2.py3-none-any.whl
pex:   - pystache-0.5.3-cp27-none-any.whl
pex:   - ipaddress-1.0.22-py2.py3-none-any.whl
pex:   - ansicolors-1.0.2-py2-none-any.whl
pex:   - future-0.17.1-py2-none-any.whl
pex:   - pantsbuild.pants-1.15.0.dev3+c835267b-cp27-cp27mu-linux_x86_64.whl
pex:   - www_authenticate-0.9.2-py2-none-any.whl
pex:   - chardet-3.0.4-py2.py3-none-any.whl
pex:   - pyOpenSSL-17.3.0-py2.py3-none-any.whl
pex:   - pyparsing-2.3.1-py2.py3-none-any.whl
pex:   - pywatchman-1.4.1-cp27-cp27mu-macosx_10_11_x86_64.whl
pex:   - contextlib2-0.5.5-py2.py3-none-any.whl
pex:   - docutils-0.12-cp27-none-any.whl
pex:   - twitter.common.confluence-0.3.10-py2-none-any.whl
pex:   - Pygments-2.3.1-py2.py3-none-any.whl
pex:   - pycparser-2.19-py2.py3-none-any.whl
pex:   - twitter.common.options-0.3.10-py2-none-any.whl
pex:   - configparser-3.5.0-py2-none-any.whl
pex:   - monotonic-1.5-py2.py3-none-any.whl
pex:   - urllib3-1.24.1-py2.py3-none-any.whl
pex:   - enum34-1.1.6-py2-none-any.whl
pex:   - Markdown-2.1.1-py2-none-any.whl
pex:   - cffi-1.11.1-cp27-cp27m-macosx_10_6_intel.whl
pex:   - faulthandler-2.6-cp27-cp27mu-macosx_10_11_x86_64.whl
pex:   - pywatchman-1.4.1-cp27-cp27mu-linux_x86_64.whl
pex:   - psutil-5.4.8-cp27-cp27mu-linux_x86_64.whl
pex:   - pantsbuild.pants-1.15.0.dev3+c835267b-cp27-cp27mu-macosx_10_11_x86_64.whl
pex:   - subprocess32-3.2.7-cp27-cp27mu-macosx_10_11_x86_64.whl
pex:   - py_zipkin-0.17.0-py2.py3-none-any.whl
pex:   - fasteners-0.14.1-py2.py3-none-any.whl
pex:   - six-1.12.0-py2.py3-none-any.whl
pex:   - asn1crypto-0.24.0-py2.py3-none-any.whl
pex:   - cryptography-2.6.1-cp27-cp27mu-macosx_10_11_x86_64.whl
pex:   - setproctitle-1.1.10-cp27-cp27m-macosx_10_11_x86_64.whl
pex:   - setproctitle-1.1.10-cp27-cp27m-linux_x86_64.whl
pex:   - twitter.common.lang-0.3.10-py2-none-any.whl
pex:   - cffi-1.11.1-cp27-cp27mu-manylinux1_x86_64.whl
pex:   - ply-3.11-py2.py3-none-any.whl
pex:   - twitter.common.log-0.3.10-py2-none-any.whl
pex:   - faulthandler-2.6-cp27-cp27mu-linux_x86_64.whl
pex:   - wheel-0.31.1-py2.py3-none-any.whl
pex:   - cryptography-2.6.1-cp27-cp27mu-manylinux1_x86_64.whl
pex:   - thriftpy2-0.4.2-cp27-cp27m-macosx_10_11_x86_64.whl
pex:   - setuptools-40.4.3-py2.py3-none-any.whl
pex:   - twitter.common.dirutil-0.3.10-py2-none-any.whl
pex:   - scandir-1.2-cp27-cp27mu-linux_x86_64.whl
pex:   - certifi-2019.3.9-py2.py3-none-any.whl
pex:   - twitter.common.collections-0.3.10-py2-none-any.whl
pex:   - thriftpy2-0.4.2-cp27-cp27m-linux_x86_64.whl
pex:   - scandir-1.2-cp27-cp27m-macosx_10_11_x86_64.whl
pex:   - asttokens-1.1.13-py2.py3-none-any.whl
pex:   - psutil-5.4.8-cp27-cp27m-macosx_10_11_x86_64.whl
pex:   - packaging-16.8-py2.py3-none-any.whl
pex:   - pathspec-0.5.9-py2-none-any.whl

...and inspecting the content of the good/bad pexes, I see the first significant difference:

  • good pex contains: cryptography-2.6.1-cp27-cp27m-macosx_10_6_intel.whl
  • bad pex contains: cryptography-2.6.1-cp27-cp27mu-macosx_10_11_x86_64.whl

The 10_6 vs 10_11 bit doesn't feel relevant, but the mu vs m bit does. I'll see whether requesting more platforms in the pex helps.

@stuhood
Copy link
Member Author

stuhood commented Mar 15, 2019

Ok, the following diff appears to resolve the issue on my machine:

diff --git a/build-support/bin/release.sh b/build-support/bin/release.sh
index ec50f98..60537ae 100755
--- a/build-support/bin/release.sh
+++ b/build-support/bin/release.sh
@@ -539,7 +539,7 @@ function build_pex() {
   local mode="$1"
 
   local linux_platform="linux_x86_64"
-  local osx_platform="macosx_10.11_x86_64"
+  local osx_platform="macosx_10.11_x86_64-cp-27-cp27m"
 
   case "${mode}" in
     build)

So I'll look into including both m and mu platforms in the pex.

@stuhood stuhood mentioned this issue Mar 15, 2019
Merged
stuhood pushed a commit that referenced this issue Mar 15, 2019
When building a pex in release.sh, build it for all supported ABIs. (#…
919db8d 
…7393)

### Problem

After #7235, we began building wheels for multiple ABIs and platforms, but when building a pex, we were allowing `pex` to choose which ABI to use in the output `pex`, which could result in an unusable output pex: see #7383.  

### Solution

Explicitly include all ABIs when constructing a pex with `-p`.

### Result

Released and nightly pexes will be larger, but much more compatible. Fixes #7383.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@stuhood @Eric-Arellano