Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix 'current' platform handling. #6104

Merged

Conversation

jsirois
Copy link
Contributor

@jsirois jsirois commented Jul 12, 2018

Previously the platform passed for 'current' was partial, leading to
ambiguous resolves. Further, partial platforms in user's BUILD files
also would lead to ambiguous resolves. For example, a multi-platform
python_binary with something like
platforms=['current', 'linux-x86_64', 'macosx_10.11_x86_64'].

Expand and and then fixup platforms as required by replacing
get_local_platform with expand_and_maybe_adjust_platform.

@jsirois
Copy link
Contributor Author

jsirois commented Jul 12, 2018

5 Legit looking errors in tests/python/pants_test/backend/python/tasks/test_python_distribution_integration.py, all 5 of the form:

...
                     E   	16:55:41 00:07   [binary]
                     E   	16:55:41 00:07     [binary-jvm-prep-command]
                     E   	16:55:41 00:07       [jvm_prep_command]
                     E   	16:55:41 00:07     [binary-prep-command]
                     E   	16:55:41 00:07     [py]
                     E   	                   Invalidated 1 target.
                     E   	                   created pex ../../../../../var/folders/bb/n7t3rs157850byt_jfdcq9k80000gn/T/tmpUEmFhK/main.pex
                     E   	16:55:42 00:08     [py-wheels]
                     E   	                   created wheel ../../../../../var/folders/bb/n7t3rs157850byt_jfdcq9k80000gn/T/tmpUEmFhK/fasthello-1.0.0-cp27-cp27m-macosx_10_12_x86_64.whl
                     E   	16:55:42 00:08     [jvm]
                     E   	16:55:42 00:08     [dup]
                     E   	16:55:42 00:08     [cpplib]
                     E   	16:55:42 00:08       [cpp-library]
                     E   	16:55:42 00:08     [cpp]
                     E   	16:55:42 00:08       [cpp-binary]
                     E   	16:55:42 00:08     [go]
                     E   	16:55:42 00:08   [run]
                     E   	16:55:42 00:08     [py]
                     E   	                   Invalidated 3 targets.
                     E   	16:55:42 00:08       [run]
                     E   	
                     E   	FAILURE: /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 example.python_distribution.hello.fasthello.main  ... exited non-zero (1)
                     E   	
                     E   	
                     E   	               Waiting for background workers to finish.
                     E   	16:55:43 00:09   [complete]
                     E   	               FAILURE
                     E   stderr:
                     E   	Failed to execute PEX file, missing macosx_10_12_x86_64-cp-27-cp27m compatible dependencies for:
                     E   	fasthello
                      PythonDistributionIntegrationTest.test_pants_run 
...

@jsirois
Copy link
Contributor Author

jsirois commented Jul 12, 2018

A similar issue in tests/python/pants_test/backend/python/tasks/test_pytest_run_integration.py:

tests/python/pants_test/backend/python/tasks/test_pytest_run_integration.py::PytestRunIntegrationTest::test_pants_test_interpreter_selection_with_pexrc <- pyprep/sources/95e7d418de90040eb18b826381cad395b8bf7450/pants_test/backend/python/tasks/test_pytest_run_integration.py
...
                     E   	17:15:57 00:07   [test]
                     E   	17:15:57 00:07     [test-jvm-prep-command]
                     E   	17:15:57 00:07       [jvm_prep_command]
                     E   	17:15:57 00:07     [test-prep-command]
                     E   	17:15:57 00:07     [test]
                     E   	17:15:57 00:07     [pytest-prep]
                     E   	                   Invalidated 3 targets.
                     E   	17:16:00 00:10     [pytest]
                     E   	                   Invalidated 1 target.
                     E   	                   scrubbed PYTHONPATH=/home/travis/build/pantsbuild/pants/src/python: from py.test environment
                     E   	17:16:00 00:10       [run]
                     E   	                     Failed to execute PEX file, missing linux_x86_64-cp-34-cp34m compatible dependencies for:
                     E   	                     coverage
                     E   	                     more-itertools
                     E   	                     pytest
                     E   	                     pytest-cov
                     E   	                     
                     E   	                   testprojects/src/python/interpreter_selection/python_3_selection_testing:test_py3.....   SUCCESSm
                     E   	FAILURE
                     E   	
                     E   	
                     E   	               Waiting for background workers to finish.
                     E   	17:16:01 00:11   [complete]
                     E   	               FAILURE
...

And a slightly different looking error as well:

tests/python/pants_test/backend/python/tasks/test_python_run_integration.py::PythonRunIntegrationTest::test_pex_resolver_blacklist_integration <- pyprep/sources/95e7d418de90040eb18b826381cad395b8bf7450/pants_test/backend/python/tasks/test_python_run_integration.py
...
                     E   	17:16:13 00:01   [pyprep]
                     E   	17:16:13 00:01     [interpreter]
                     E   	                   Invalidated 1 target.
                     E   	17:16:15 00:03     [build-local-dists]
                     E   	17:16:15 00:03     [requirements]
                     E   	                   Invalidated 1 target.
                     E   	               Waiting for background workers to finish.
                     E   	17:16:20 00:08   [complete]
                     E   	               FAILURE
                     E   stderr:
                     E   	INFO] For async removal, run `./pants clean-all --async`
                     E   	
                     E   	Exception caught: (<class 'pex.resolver.Untranslateable'>)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 72, in <module>
                     E   	    main()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 68, in main
                     E   	    PantsLoader.run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 64, in run
                     E   	    cls.load_and_execute(entrypoint)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 57, in load_and_execute
                     E   	    entrypoint_main()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_exe.py", line 38, in main
                     E   	    PantsRunner(exiter, start_time=start_time).run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_runner.py", line 52, in run
                     E   	    return runner.run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/local_pants_runner.py", line 45, in run
                     E   	    self._run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/local_pants_runner.py", line 93, in _run
                     E   	    goal_runner_result = goal_runner.run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/goal_runner.py", line 221, in run
                     E   	    result = self._execute_engine()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/goal_runner.py", line 209, in _execute_engine
                     E   	    result = engine.execute(self._context, self._goals)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/engine/legacy_engine.py", line 25, in execute
                     E   	    self.attempt(context, goals)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/engine/round_engine.py", line 232, in attempt
                     E   	    goal_executor.attempt(explain)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/engine/round_engine.py", line 48, in attempt
                     E   	    task.execute()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/resolve_requirements.py", line 29, in execute
                     E   	    pex = self.resolve_requirements(interpreter, self.context.targets(has_python_requirements))
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/resolve_requirements_task_base.py", line 80, in resolve_requirements
                     E   	    dump_requirement_libs(builder, interpreter, req_libs, self.context.log, platforms=maybe_platforms)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/pex_build_util.py", line 95, in dump_requirement_libs
                     E   	    dump_requirements(builder, interpreter, reqs, log, platforms=platforms)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/pex_build_util.py", line 119, in dump_requirements
                     E   	    distributions = resolve_multi(interpreter, deduped_reqs, platforms, find_links)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/pex_build_util.py", line 163, in resolve_multi
                     E   	    use_manylinux=python_setup.use_manylinux)
                     E   	  File "/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/pex/resolver.py", line 453, in resolve
                     E   	    return resolver.resolve(resolvables_from_iterable(requirements, builder))
                     E   	  File "/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/pex/resolver.py", line 248, in resolve
                     E   	    dist = self.build(package, resolvable.options)
                     E   	  File "/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/pex/resolver.py", line 323, in build
                     E   	    dist = super(CachingResolver, self).build(package, options)
                     E   	  File "/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/pex/resolver.py", line 206, in build
                     E   	    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
                     E   	
                     E   	Exception message: Package SourcePackage(u'file:///home/travis/.cache/pants/python_cache/requirements/CPython-3.4.3/pandocfilters-1.4.2.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)m
...

@Eric-Arellano
Copy link
Contributor

@jsirois I've been getting that first error ever since I started working on this project two weeks ago when running ci.sh on my Mac.

@jsirois
Copy link
Contributor Author

jsirois commented Jul 12, 2018

Thanks for the independent datapoint @Eric-Arellano. I run linux so this is especially useful to me.

That says we definitely have more problems than just in resolve-multi which is the only bit this PR fixes.

@CMLivingston
Copy link
Contributor

@jsirois I have a good recollection of that platform-based python_distribution test and the case it is covering. I can look into that.

@jsirois
Copy link
Contributor Author

jsirois commented Jul 12, 2018

Thanks @CMLivingston

@CMLivingston
Copy link
Contributor

CMLivingston commented Jul 12, 2018

I cannot repro the osx shard failure locally (https://travis-ci.org/pantsbuild/pants/jobs/403189175). Would anyone else be able to take a crack at the tests and report any failures?
./pants test tests/python/pants_test/backend/python/tasks:: -- -k test_python_distribution_integration

From what I can see in the output log, it looks like the wheel is named properly and the os version aligns with mine, so I can't see why this would fail.

@jsirois
Copy link
Contributor Author

jsirois commented Jul 13, 2018

OK, This hacky fix debug output (see: >>> Modifying given platform of 'current':) should also summarize the problem for at least the test_pants_test_interpreter_selection_with_pexrc error above:

                    18:50:55 00:11     [pytest-prep]
                                        Invalidated 3 targets.
                             >>> Modifying given platform of 'current':
                                 Resolves as Platform(platform='linux_x86_64', impl='cp', version='27', abi='cp27mu')
                                 Under current interpreter u"PythonInterpreter('/home/jsirois/dev/pantsbuild/jsirois-pants/build-support/pants_dev_deps.venv/bin/python2.7', PythonIdentity('CPython', 2, 7, 15))"
                                 
                                 To match given interpreter u"PythonInterpreter('/usr/bin/python3.6', PythonIdentity('CPython', 3, 6, 6))".
                                 
                                 Calculated platform: Platform(platform='linux_x86_64', impl='cp', version='36', abi='cp36m')
                     
                     18:51:06 00:22     [pytest]
                                        Invalidated 1 target.
                                        scrubbed PYTHONPATH=/home/jsirois/dev/pantsbuild/jsirois-pants/src/python: from py.test environment
                     18:51:06 00:22       [run]
                                          ============================= test session starts ==============================
                                          platform linux -- Python 3.6.6, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
                                          rootdir: /home/jsirois/dev/pantsbuild/jsirois-pants/.pants.d/tmp/tmpoJAkuo.pants.d, inifile: /home/jsirois/dev/pantsbuild/jsirois-pants/.pants.d/tmp/tmpoJAkuo.pants.d/test/pytest-prep/CPython-3.6.6/ef25b5846bb2b47d95d4248cab9384a52f73af8b/pytest.ini
                                          plugins: timeout-1.2.1, cov-2.4.0
                                          collected 1 item
                                          
                                          testprojects/src/python/interpreter_selection/python_3_selection_testing/test_py3.py . [100%]
                                          
                                           generated xml file: /home/jsirois/dev/pantsbuild/jsirois-pants/.pants.d/tmp/tmpoJAkuo.pants.d/test/pytest/testprojects.src.python.interpreter_selection.python_3_selection_testing.test_py3/junitxml/TEST-testprojects.src.python.interpreter_selection.python_3_selection_testing.test_py3.xml 
                                          =========================== 1 passed in 0.04 seconds ===========================
                                          
                                        testprojects/src/python/interpreter_selection/python_3_selection_testing:test_py3.....   SUCCESS

@jsirois
Copy link
Contributor Author

jsirois commented Jul 13, 2018

This is more work related to workarounds for pex-tool/pex#511. Most of the logic introduced here - or all of it - should move into pex.resolver.resolve.

@jsirois
Copy link
Contributor Author

jsirois commented Jul 13, 2018

OK @CMLivingston all linux test failures are now fixed so I'll dig into test_python_distribution_integration failing on OSX.

@jsirois
Copy link
Contributor Author

jsirois commented Jul 13, 2018

Noting that in the OSX failures, ./pants, via which and the PATH selects /usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/bin/python2.7 as the runtime interpreter and the test failures all show /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 - selected no doubt by the min here. So two different local interpreters are in-play.

The fix above is designed to handle that case in the resolve_multi path. I suspect the python distribution code represents another - unfixed path.

@@ -227,7 +227,8 @@ def _create_dist(self, dist_tgt, dist_target_dir, interpreter,

setup_runner = SetupPyRunner.for_bdist_wheel(
dist_target_dir,
is_platform_specific=is_platform_specific)
is_platform_specific=is_platform_specific,
interpreter=interpreter)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK @CMLivingston and @cosmicexplorer - other issues elsewhere aside - this code never stood a fair chance of working since the selected interpreter was not passed.

@cosmicexplorer
Copy link
Contributor

cosmicexplorer commented Jul 13, 2018

The interpreter selection bit you just described seems utterly fantastic because

stderr:
                     E   	Failed to execute PEX file, missing macosx_10_12_x86_64-cp-27-cp27m compatible dependencies for:
                     E   	fasthello

is what i spent an entire migraine on trying to understand in the osx shard in #6022 last week (which you and Kris were there for, which I appreciate immensely) — not trying to whine, just this seems like it would directly address that maybe, so that’s great.

@jsirois jsirois force-pushed the python/resolve/fix-current-platform-handling branch from 9bd378a to ab976db Compare July 13, 2018 09:54
@jsirois
Copy link
Contributor Author

jsirois commented Jul 13, 2018

Noting current issue on OSX:

With the correct interpreter being used for run/test/repl/... pexes, which is the system 2.7.10 in this case (as opposed to brew 2.7.14), the accepted platforms calculation looks wrong - no macosx_10_*_x86_64, just macosx_10_*_intel:

pex: E: tags for 'macosx-10.12-intel' x PythonInterpreter('/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python', PythonIdentity('CPython', 2, 7, 10), {('matplotlib', '1.3.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-corelocation', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-coretext', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-syncservices', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('six', '1.4.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-fsevents', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-cocoa', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-preferencepanes', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('zope.interface', '4.1.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyparsing', '2.0.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-collaboration', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pytz', '2013.7'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-pubsub', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-scriptingbridge', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-servicemanagement', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('numpy', '1.8.0rc1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-installerplugins', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('setuptools', '18.5'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-opendirectory', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-applescriptkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-automator', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-addressbook', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-searchkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-applescriptobjc', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-qtkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-latentsemanticmapping', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('scipy', '0.13.0b1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('modulegraph', '0.10.4'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('altgraph', '0.10.2'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-instantmessage', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('xattr', '0.6.4'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-eventkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-social', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-cfnetwork', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-quartz', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-exceptionhandling', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-launchservices', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-coredata', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('python', '2.7.10'): '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', ('bdist-mpkg', '0.5.0'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-message', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-inputmethodkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('py2app', '0.7.3'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-dictionaryservices', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-accounts', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-screensaver', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyopenssl', '0.13.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('bonjour-py', '0.3'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-core', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('wsgiref', '0.1.2'): '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', ('pyobjc-framework-webkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('python-dateutil', '1.5'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-systemconfiguration', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('macholib', '1.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python'}) -> [('cp27', 'cp27m', 'macosx_10_8_universal'), ('cp27', 'cp27m', 'macosx_10_12_universal'), ('cp27', 'cp27mu', 'macosx_10_7_intel'), ('cp27', 'cp27mu', 'macosx_10_4_intel'), ('cp27', 'cp27d', 'macosx_10_9_intel'), ('cp27', 'cp27d', 'macosx_10_12_universal'), ('cp27', 'cp27dm', 'macosx_10_8_intel'), ('cp27', 'cp27dmu', 'macosx_10_0_universal'), ('cp27', 'cp27d', 'macosx_10_11_intel'), ('cp27', 'cp27dmu', 'macosx_10_10_intel'), ('cp27', 'cp27dm', 'macosx_10_8_universal'), ('cp27', 'cp27dm', 'macosx_10_11_universal'), ('cp27', 'cp27m', 'macosx_10_4_universal'), ('cp27', 'cp27mu', 'macosx_10_9_intel'), ('cp27', 'cp27dm', 'macosx_10_0_universal'), ('py2', 'none', 'macosx_10_0_universal'), ('cp27', 'cp27dmu', 'macosx_10_12_intel'), ('cp27', 'cp27m', 'macosx_10_5_universal'), ('cp2', 'none', 'any'), ('cp27', 'cp27m', 'macosx_10_8_intel'), ('py2', 'none', 'macosx_10_11_universal'), ('cp27', 'cp27m', 'macosx_10_10_intel'), ('cp27', 'cp27u', 'macosx_10_9_intel'), ('py2', 'none', 'macosx_10_6_intel'), ('cp27', 'none', 'macosx_10_10_intel'), ('py2', 'none', 'any'), ('cp27', 'none', 'macosx_10_5_intel'), ('cp27', 'none', 'macosx_10_8_intel'), ('cp27', 'cp27mu', 'macosx_10_0_universal'), ('cp27', 'cp27u', 'macosx_10_9_universal'), ('cp27', 'cp27mu', 'macosx_10_11_universal'), ('cp27', 'none', 'macosx_10_12_intel'), ('py27', 'none', 'any'), ('cp27', 'cp27dm', 'macosx_10_4_intel'), ('cp27', 'cp27mu', 'macosx_10_6_intel'), ('cp27', 'none', 'macosx_10_6_intel'), ('cp27', 'cp27mu', 'macosx_10_9_universal'), ('cp27', 'cp27mu', 'macosx_10_5_intel'), ('cp27', 'cp27dmu', 'macosx_10_11_intel'), ('cp27', 'cp27dmu', 'macosx_10_5_universal'), ('cp27', 'cp27dmu', 'macosx_10_5_intel'), ('cp27', 'cp27u', 'macosx_10_3_universal'), ('cp27', 'cp27dmu', 'macosx_10_4_universal'), ('cp27', 'cp27dmu', 'macosx_10_6_intel'), ('cp27', 'cp27dm', 'macosx_10_7_intel'), ('cp27', 'none', 'macosx_10_12_universal'), ('cp27', 'cp27u', 'macosx_10_4_intel'), ('cp27', 'none', 'macosx_10_11_intel'), ('cp27', 'cp27u', 'macosx_10_0_universal'), ('cp27', 'cp27m', 'macosx_10_12_intel'), ('cp27', 'none', 'macosx_10_4_universal'), ('cp27', 'cp27dmu', 'macosx_10_11_universal'), ('cp27', 'cp27dmu', 'macosx_10_3_universal'), ('cp27', 'none', 'macosx_10_5_universal'), ('cp27', 'cp27dmu', 'macosx_10_12_universal'), ('py2', 'none', 'macosx_10_9_universal'), ('cp27', 'cp27dm', 'macosx_10_9_universal'), ('cp27', 'cp27u', 'macosx_10_11_universal'), ('cp27', 'cp27dm', 'macosx_10_1_universal'), ('cp27', 'cp27m', 'macosx_10_2_universal'), ('cp27', 'cp27d', 'macosx_10_8_intel'), ('cp27', 'cp27u', 'macosx_10_1_universal'), ('cp27', 'none', 'macosx_10_10_universal'), ('cp27', 'cp27d', 'macosx_10_5_intel'), ('cp27', 'cp27m', 'macosx_10_10_universal'), ('cp27', 'cp27mu', 'macosx_10_7_universal'), ('cp27', 'cp27mu', 'macosx_10_11_intel'), ('cp27', 'cp27dmu', 'macosx_10_6_universal'), ('cp27', 'none', 'macosx_10_2_universal'), ('cp27', 'cp27u', 'macosx_10_7_intel'), ('cp27', 'none', 'macosx_10_3_universal'), ('cp27', 'cp27d', 'macosx_10_4_intel'), ('cp27', 'cp27dm', 'macosx_10_9_intel'), ('py2', 'none', 'macosx_10_9_intel'), ('cp27', 'cp27m', 'macosx_10_7_intel'), ('cp27', 'cp27m', 'macosx_10_3_universal'), ('cp27', 'cp27m', 'macosx_10_5_intel'), ('cp27', 'cp27m', 'macosx_10_6_universal'), ('cp27', 'none', 'macosx_10_6_universal'), ('py2', 'none', 'macosx_10_7_universal'), ('py2', 'none', 'macosx_10_11_intel'), ('cp27', 'cp27m', 'macosx_10_6_intel'), ('cp27', 'cp27dmu', 'macosx_10_2_universal'), ('cp27', 'cp27m', 'macosx_10_1_universal'), ('cp27', 'cp27dmu', 'macosx_10_8_intel'), ('cp27', 'cp27d', 'macosx_10_11_universal'), ('cp27', 'cp27mu', 'macosx_10_12_intel'), ('cp27', 'cp27u', 'macosx_10_5_intel'), ('cp27', 'cp27u', 'macosx_10_8_universal'), ('cp27', 'cp27dm', 'macosx_10_6_universal'), ('cp27', 'none', 'macosx_10_9_universal'), ('cp27', 'cp27d', 'macosx_10_6_intel'), ('cp27', 'cp27mu', 'macosx_10_10_intel'), ('cp27', 'cp27u', 'macosx_10_7_universal'), ('cp27', 'cp27dmu', 'macosx_10_7_universal'), ('py2', 'none', 'macosx_10_12_intel'), ('cp27', 'cp27dmu', 'macosx_10_9_universal'), ('py2', 'none', 'macosx_10_8_universal'), ('py2', 'none', 'macosx_10_12_universal'), ('cp27', 'cp27mu', 'macosx_10_5_universal'), ('cp27', 'cp27d', 'macosx_10_3_universal'), ('cp27', 'cp27d', 'macosx_10_1_universal'), ('cp27', 'cp27m', 'macosx_10_7_universal'), ('py2', 'none', 'macosx_10_10_intel'), ('cp27', 'cp27mu', 'macosx_10_4_universal'), ('cp27', 'cp27dm', 'macosx_10_12_universal'), ('cp27', 'cp27d', 'macosx_10_10_universal'), ('cp27', 'cp27u', 'macosx_10_11_intel'), ('cp27', 'cp27dmu', 'macosx_10_10_universal'), ('cp27', 'cp27m', 'macosx_10_0_universal'), ('cp27', 'cp27mu', 'macosx_10_8_universal'), ('cp27', 'cp27mu', 'macosx_10_12_universal'), ('cp27', 'cp27d', 'macosx_10_2_universal'), ('cp27', 'cp27d', 'macosx_10_7_intel'), ('cp27', 'cp27dm', 'macosx_10_6_intel'), ('cp27', 'cp27m', 'macosx_10_4_intel'), ('py2', 'none', 'macosx_10_4_universal'), ('cp27', 'none', 'macosx_10_4_intel'), ('py2', 'none', 'macosx_10_5_intel'), ('cp27', 'cp27d', 'macosx_10_6_universal'), ('cp27', 'cp27dm', 'macosx_10_5_universal'), ('cp27', 'cp27dm', 'macosx_10_4_universal'), ('cp27', 'none', 'macosx_10_1_universal'), ('cp27', 'cp27dmu', 'macosx_10_4_intel'), ('cp27', 'cp27u', 'macosx_10_2_universal'), ('cp27', 'cp27mu', 'macosx_10_6_universal'), ('cp27', 'cp27mu', 'macosx_10_2_universal'), ('cp27', 'cp27d', 'macosx_10_9_universal'), ('cp27', 'cp27dm', 'macosx_10_10_universal'), ('cp27', 'cp27dmu', 'macosx_10_1_universal'), ('cp27', 'cp27mu', 'macosx_10_8_intel'), ('cp27', 'cp27u', 'macosx_10_10_intel'), ('cp27', 'cp27dmu', 'macosx_10_9_intel'), ('cp27', 'none', 'macosx_10_8_universal'), ('py2', 'none', 'macosx_10_6_universal'), ('cp27', 'cp27dm', 'macosx_10_12_intel'), ('cp27', 'cp27d', 'macosx_10_0_universal'), ('cp27', 'none', 'any'), ('cp27', 'cp27m', 'macosx_10_9_universal'), ('cp27', 'cp27d', 'macosx_10_7_universal'), ('cp27', 'none', 'macosx_10_9_intel'), ('py2', 'none', 'macosx_10_8_intel'), ('cp27', 'cp27dm', 'macosx_10_2_universal'), ('cp27', 'cp27m', 'macosx_10_11_universal'), ('cp27', 'cp27u', 'macosx_10_6_universal'), ('cp27', 'cp27dm', 'macosx_10_10_intel'), ('cp27', 'cp27dmu', 'macosx_10_8_universal'), ('cp27', 'cp27d', 'macosx_10_12_intel'), ('cp27', 'cp27u', 'macosx_10_12_intel'), ('cp27', 'none', 'macosx_10_7_intel'), ('cp27', 'cp27u', 'macosx_10_6_intel'), ('py2', 'none', 'macosx_10_3_universal'), ('py2', 'none', 'macosx_10_5_universal'), ('py2', 'none', 'macosx_10_2_universal'), ('cp27', 'cp27dmu', 'macosx_10_7_intel'), ('cp27', 'cp27d', 'macosx_10_10_intel'), ('cp27', 'cp27u', 'macosx_10_4_universal'), ('cp27', 'cp27mu', 'macosx_10_3_universal'), ('py2', 'none', 'macosx_10_1_universal'), ('py2', 'none', 'macosx_10_10_universal'), ('cp27', 'cp27dm', 'macosx_10_5_intel'), ('cp27', 'cp27d', 'macosx_10_5_universal'), ('cp27', 'cp27u', 'macosx_10_5_universal'), ('cp27', 'cp27dm', 'macosx_10_11_intel'), ('cp27', 'cp27dm', 'macosx_10_7_universal'), ('cp27', 'cp27u', 'macosx_10_12_universal'), ('cp27', 'cp27d', 'macosx_10_4_universal'), ('cp27', 'cp27u', 'macosx_10_10_universal'), ('cp27', 'cp27m', 'macosx_10_9_intel'), ('py2', 'none', 'macosx_10_7_intel'), ('py2', 'none', 'macosx_10_4_intel'), ('cp27', 'cp27d', 'macosx_10_8_universal'), ('cp27', 'cp27u', 'macosx_10_8_intel'), ('cp27', 'none', 'macosx_10_0_universal'), ('cp27', 'cp27mu', 'macosx_10_1_universal'), ('cp27', 'cp27mu', 'macosx_10_10_universal'), ('cp27', 'cp27m', 'macosx_10_11_intel'), ('cp27', 'none', 'macosx_10_11_universal'), ('cp27', 'none', 'macosx_10_7_universal'), ('cp27', 'cp27dm', 'macosx_10_3_universal')]

@jsirois jsirois force-pushed the python/resolve/fix-current-platform-handling branch 2 times, most recently from 6196ae2 to 503b193 Compare July 13, 2018 13:05
@Eric-Arellano
Copy link
Contributor

Awesome! I'm running ci.sh on mac - will let you know what happens.

@Eric-Arellano
Copy link
Contributor

6 tests fail now instead of I think it was 11 earlier, that's exciting! https://hastebin.com/oxinuwepol.rb

One of the failures is the same as CI's unit test failure, TestBuildLocalPythonDistributions.test_python_create_platform_specific_distribution

@jsirois
Copy link
Contributor Author

jsirois commented Jul 13, 2018

@Eric-Arellano note well that the OSX fix for Travis was ultimately getting off an OSX VM with Apple's supplied 2.7.10 which returns a broken -intel build platform. If you too have that, this may not provide you relief locally yet.

Previously the platform passed for 'current' was partial, leading to
ambiguous resolves. Further, partial platforms in user's BUILD files
also would lead to ambiguous resolves. For example, a multi-platform
`python_binary` with something like
`platforms=['current', 'linux-x86_64', 'macosx_10.11_x86_64']`.

Expand and and then fixup platforms as required by replacing
`get_local_platform` with `expand_and_maybe_adjust_platform`.
+ Fix an unused `interpreter` parameter - forward to `SetupPyRunner`.
+ Kill declaring `--universal` on behalf of the package author. There
  is not enough info to make the decision if the code being dist'd, if
  pure python, is 2/3 compatible.

To compensate for the removal of `--universal`, give an example of how
to declare your python_dist as `--universal` when you know it is as the
BUILD `python_dist` target author.
@jsirois jsirois force-pushed the python/resolve/fix-current-platform-handling branch from 833d40b to fe2907e Compare July 13, 2018 16:12
@jsirois
Copy link
Contributor Author

jsirois commented Jul 13, 2018

Thanks @Eric-Arellano - I was able to repro that failure on linux and it should be fixed this round.

Copy link
Member

@kwlzn kwlzn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm!

abi_tag=ident.abi_tag))


def expand_and_maybe_adjust_platform(interpreter, platform):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a TODO to upstream this into pex might be good?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup - this old one pex-tool/pex#511, I'll add a TODO similar to the method above.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

env = copy(kwargs.pop('env', {}))

# Hack around bug in PEX where custom interpreters are not forwarded to PEXEnvironments.
# TODO(John Sirois): Remove when XXX is fixed.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/XXX/a ticket?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup - just finished filing on the pex side. I'll send up a final comment-only diff with issue inks here as soon as this goes green then merge: pex-tool/pex#522

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

.travis.yml Outdated
@@ -476,6 +476,10 @@ matrix:
language: generic
env:
- SHARD="Rust + Platform-specific Tests OSX"
# Specifically avoid the OSX provided 2.7.10 under xcode8.3 since it returns a platform
# of `macosx-*-intel` where the `intel` suffix is bogus but pex has not yet been taught to
# deal with this. Can be removed when this issue is resolved: YYY
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed.

@jsirois jsirois merged commit a1696c8 into pantsbuild:master Jul 13, 2018
@jsirois jsirois deleted the python/resolve/fix-current-platform-handling branch July 13, 2018 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants