Fix 'current' platform handling. #6104

jsirois · 2018-07-12T16:08:48Z

Previously the platform passed for 'current' was partial, leading to
ambiguous resolves. Further, partial platforms in user's BUILD files
also would lead to ambiguous resolves. For example, a multi-platform
python_binary with something like
platforms=['current', 'linux-x86_64', 'macosx_10.11_x86_64'].

Expand and and then fixup platforms as required by replacing
get_local_platform with expand_and_maybe_adjust_platform.

jsirois · 2018-07-12T17:12:56Z

5 Legit looking errors in tests/python/pants_test/backend/python/tasks/test_python_distribution_integration.py, all 5 of the form:

...
                     E   	16:55:41 00:07   [binary]
                     E   	16:55:41 00:07     [binary-jvm-prep-command]
                     E   	16:55:41 00:07       [jvm_prep_command]
                     E   	16:55:41 00:07     [binary-prep-command]
                     E   	16:55:41 00:07     [py]
                     E   	                   Invalidated 1 target.
                     E   	                   created pex ../../../../../var/folders/bb/n7t3rs157850byt_jfdcq9k80000gn/T/tmpUEmFhK/main.pex
                     E   	16:55:42 00:08     [py-wheels]
                     E   	                   created wheel ../../../../../var/folders/bb/n7t3rs157850byt_jfdcq9k80000gn/T/tmpUEmFhK/fasthello-1.0.0-cp27-cp27m-macosx_10_12_x86_64.whl
                     E   	16:55:42 00:08     [jvm]
                     E   	16:55:42 00:08     [dup]
                     E   	16:55:42 00:08     [cpplib]
                     E   	16:55:42 00:08       [cpp-library]
                     E   	16:55:42 00:08     [cpp]
                     E   	16:55:42 00:08       [cpp-binary]
                     E   	16:55:42 00:08     [go]
                     E   	16:55:42 00:08   [run]
                     E   	16:55:42 00:08     [py]
                     E   	                   Invalidated 3 targets.
                     E   	16:55:42 00:08       [run]
                     E   	
                     E   	FAILURE: /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 example.python_distribution.hello.fasthello.main  ... exited non-zero (1)
                     E   	
                     E   	
                     E   	               Waiting for background workers to finish.
                     E   	16:55:43 00:09   [complete]
                     E   	               FAILURE
                     E   stderr:
                     E   	Failed to execute PEX file, missing macosx_10_12_x86_64-cp-27-cp27m compatible dependencies for:
                     E   	fasthello
                      PythonDistributionIntegrationTest.test_pants_run 
...

jsirois · 2018-07-12T17:26:07Z

A similar issue in tests/python/pants_test/backend/python/tasks/test_pytest_run_integration.py:

tests/python/pants_test/backend/python/tasks/test_pytest_run_integration.py::PytestRunIntegrationTest::test_pants_test_interpreter_selection_with_pexrc <- pyprep/sources/95e7d418de90040eb18b826381cad395b8bf7450/pants_test/backend/python/tasks/test_pytest_run_integration.py
...
                     E   	17:15:57 00:07   [test]
                     E   	17:15:57 00:07     [test-jvm-prep-command]
                     E   	17:15:57 00:07       [jvm_prep_command]
                     E   	17:15:57 00:07     [test-prep-command]
                     E   	17:15:57 00:07     [test]
                     E   	17:15:57 00:07     [pytest-prep]
                     E   	                   Invalidated 3 targets.
                     E   	17:16:00 00:10     [pytest]
                     E   	                   Invalidated 1 target.
                     E   	                   scrubbed PYTHONPATH=/home/travis/build/pantsbuild/pants/src/python: from py.test environment
                     E   	17:16:00 00:10       [run]
                     E   	                     Failed to execute PEX file, missing linux_x86_64-cp-34-cp34m compatible dependencies for:
                     E   	                     coverage
                     E   	                     more-itertools
                     E   	                     pytest
                     E   	                     pytest-cov
                     E   	                     
                     E   	                   testprojects/src/python/interpreter_selection/python_3_selection_testing:test_py3.....   SUCCESSm
                     E   	FAILURE
                     E   	
                     E   	
                     E   	               Waiting for background workers to finish.
                     E   	17:16:01 00:11   [complete]
                     E   	               FAILURE
...

And a slightly different looking error as well:

tests/python/pants_test/backend/python/tasks/test_python_run_integration.py::PythonRunIntegrationTest::test_pex_resolver_blacklist_integration <- pyprep/sources/95e7d418de90040eb18b826381cad395b8bf7450/pants_test/backend/python/tasks/test_python_run_integration.py
...
                     E   	17:16:13 00:01   [pyprep]
                     E   	17:16:13 00:01     [interpreter]
                     E   	                   Invalidated 1 target.
                     E   	17:16:15 00:03     [build-local-dists]
                     E   	17:16:15 00:03     [requirements]
                     E   	                   Invalidated 1 target.
                     E   	               Waiting for background workers to finish.
                     E   	17:16:20 00:08   [complete]
                     E   	               FAILURE
                     E   stderr:
                     E   	INFO] For async removal, run `./pants clean-all --async`
                     E   	
                     E   	Exception caught: (<class 'pex.resolver.Untranslateable'>)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 72, in <module>
                     E   	    main()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 68, in main
                     E   	    PantsLoader.run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 64, in run
                     E   	    cls.load_and_execute(entrypoint)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_loader.py", line 57, in load_and_execute
                     E   	    entrypoint_main()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_exe.py", line 38, in main
                     E   	    PantsRunner(exiter, start_time=start_time).run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/pants_runner.py", line 52, in run
                     E   	    return runner.run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/local_pants_runner.py", line 45, in run
                     E   	    self._run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/local_pants_runner.py", line 93, in _run
                     E   	    goal_runner_result = goal_runner.run()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/goal_runner.py", line 221, in run
                     E   	    result = self._execute_engine()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/bin/goal_runner.py", line 209, in _execute_engine
                     E   	    result = engine.execute(self._context, self._goals)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/engine/legacy_engine.py", line 25, in execute
                     E   	    self.attempt(context, goals)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/engine/round_engine.py", line 232, in attempt
                     E   	    goal_executor.attempt(explain)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/engine/round_engine.py", line 48, in attempt
                     E   	    task.execute()
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/resolve_requirements.py", line 29, in execute
                     E   	    pex = self.resolve_requirements(interpreter, self.context.targets(has_python_requirements))
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/resolve_requirements_task_base.py", line 80, in resolve_requirements
                     E   	    dump_requirement_libs(builder, interpreter, req_libs, self.context.log, platforms=maybe_platforms)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/pex_build_util.py", line 95, in dump_requirement_libs
                     E   	    dump_requirements(builder, interpreter, reqs, log, platforms=platforms)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/pex_build_util.py", line 119, in dump_requirements
                     E   	    distributions = resolve_multi(interpreter, deduped_reqs, platforms, find_links)
                     E   	  File "/home/travis/build/pantsbuild/pants/src/python/pants/backend/python/tasks/pex_build_util.py", line 163, in resolve_multi
                     E   	    use_manylinux=python_setup.use_manylinux)
                     E   	  File "/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/pex/resolver.py", line 453, in resolve
                     E   	    return resolver.resolve(resolvables_from_iterable(requirements, builder))
                     E   	  File "/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/pex/resolver.py", line 248, in resolve
                     E   	    dist = self.build(package, resolvable.options)
                     E   	  File "/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/pex/resolver.py", line 323, in build
                     E   	    dist = super(CachingResolver, self).build(package, options)
                     E   	  File "/home/travis/build/pantsbuild/pants/build-support/pants_dev_deps.venv/lib/python2.7/site-packages/pex/resolver.py", line 206, in build
                     E   	    raise Untranslateable('Package %s is not translateable by %s' % (package, translator))
                     E   	
                     E   	Exception message: Package SourcePackage(u'file:///home/travis/.cache/pants/python_cache/requirements/CPython-3.4.3/pandocfilters-1.4.2.tar.gz') is not translateable by ChainedTranslator(WheelTranslator, EggTranslator, SourceTranslator)m
...

Eric-Arellano · 2018-07-12T17:28:22Z

@jsirois I've been getting that first error ever since I started working on this project two weeks ago when running ci.sh on my Mac.

jsirois · 2018-07-12T17:31:21Z

Thanks for the independent datapoint @Eric-Arellano. I run linux so this is especially useful to me.

That says we definitely have more problems than just in resolve-multi which is the only bit this PR fixes.

CMLivingston · 2018-07-12T17:40:31Z

@jsirois I have a good recollection of that platform-based python_distribution test and the case it is covering. I can look into that.

jsirois · 2018-07-12T17:41:59Z

Thanks @CMLivingston

CMLivingston · 2018-07-12T22:22:46Z

I cannot repro the osx shard failure locally (https://travis-ci.org/pantsbuild/pants/jobs/403189175). Would anyone else be able to take a crack at the tests and report any failures?
./pants test tests/python/pants_test/backend/python/tasks:: -- -k test_python_distribution_integration

From what I can see in the output log, it looks like the wheel is named properly and the os version aligns with mine, so I can't see why this would fail.

jsirois · 2018-07-13T00:54:04Z

OK, This hacky fix debug output (see: >>> Modifying given platform of 'current':) should also summarize the problem for at least the test_pants_test_interpreter_selection_with_pexrc error above:

                    18:50:55 00:11     [pytest-prep]
                                        Invalidated 3 targets.
                             >>> Modifying given platform of 'current':
                                 Resolves as Platform(platform='linux_x86_64', impl='cp', version='27', abi='cp27mu')
                                 Under current interpreter u"PythonInterpreter('/home/jsirois/dev/pantsbuild/jsirois-pants/build-support/pants_dev_deps.venv/bin/python2.7', PythonIdentity('CPython', 2, 7, 15))"
                                 
                                 To match given interpreter u"PythonInterpreter('/usr/bin/python3.6', PythonIdentity('CPython', 3, 6, 6))".
                                 
                                 Calculated platform: Platform(platform='linux_x86_64', impl='cp', version='36', abi='cp36m')
                     
                     18:51:06 00:22     [pytest]
                                        Invalidated 1 target.
                                        scrubbed PYTHONPATH=/home/jsirois/dev/pantsbuild/jsirois-pants/src/python: from py.test environment
                     18:51:06 00:22       [run]
                                          ============================= test session starts ==============================
                                          platform linux -- Python 3.6.6, pytest-3.6.3, py-1.5.4, pluggy-0.6.0
                                          rootdir: /home/jsirois/dev/pantsbuild/jsirois-pants/.pants.d/tmp/tmpoJAkuo.pants.d, inifile: /home/jsirois/dev/pantsbuild/jsirois-pants/.pants.d/tmp/tmpoJAkuo.pants.d/test/pytest-prep/CPython-3.6.6/ef25b5846bb2b47d95d4248cab9384a52f73af8b/pytest.ini
                                          plugins: timeout-1.2.1, cov-2.4.0
                                          collected 1 item
                                          
                                          testprojects/src/python/interpreter_selection/python_3_selection_testing/test_py3.py . [100%]
                                          
                                           generated xml file: /home/jsirois/dev/pantsbuild/jsirois-pants/.pants.d/tmp/tmpoJAkuo.pants.d/test/pytest/testprojects.src.python.interpreter_selection.python_3_selection_testing.test_py3/junitxml/TEST-testprojects.src.python.interpreter_selection.python_3_selection_testing.test_py3.xml 
                                          =========================== 1 passed in 0.04 seconds ===========================
                                          
                                        testprojects/src/python/interpreter_selection/python_3_selection_testing:test_py3.....   SUCCESS

jsirois · 2018-07-13T02:53:07Z

This is more work related to workarounds for pex-tool/pex#511. Most of the logic introduced here - or all of it - should move into pex.resolver.resolve.

jsirois · 2018-07-13T04:00:11Z

OK @CMLivingston all linux test failures are now fixed so I'll dig into test_python_distribution_integration failing on OSX.

jsirois · 2018-07-13T05:23:05Z

Noting that in the OSX failures, ./pants, via which and the PATH selects /usr/local/Cellar/python/2.7.14/Frameworks/Python.framework/Versions/2.7/bin/python2.7 as the runtime interpreter and the test failures all show /System/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7 - selected no doubt by the min here. So two different local interpreters are in-play.

The fix above is designed to handle that case in the resolve_multi path. I suspect the python distribution code represents another - unfixed path.

jsirois · 2018-07-13T05:33:48Z

src/python/pants/backend/python/tasks/build_local_python_distributions.py

@@ -227,7 +227,8 @@ def _create_dist(self, dist_tgt, dist_target_dir, interpreter,

    setup_runner = SetupPyRunner.for_bdist_wheel(
      dist_target_dir,
-      is_platform_specific=is_platform_specific)
+      is_platform_specific=is_platform_specific,
+      interpreter=interpreter)


OK @CMLivingston and @cosmicexplorer - other issues elsewhere aside - this code never stood a fair chance of working since the selected interpreter was not passed.

cosmicexplorer · 2018-07-13T09:21:54Z

The interpreter selection bit you just described seems utterly fantastic because

stderr:
                     E   	Failed to execute PEX file, missing macosx_10_12_x86_64-cp-27-cp27m compatible dependencies for:
                     E   	fasthello

is what i spent an entire migraine on trying to understand in the osx shard in #6022 last week (which you and Kris were there for, which I appreciate immensely) — not trying to whine, just this seems like it would directly address that maybe, so that’s great.

jsirois · 2018-07-13T11:10:46Z

Noting current issue on OSX:

With the correct interpreter being used for run/test/repl/... pexes, which is the system 2.7.10 in this case (as opposed to brew 2.7.14), the accepted platforms calculation looks wrong - no macosx_10_*_x86_64, just macosx_10_*_intel:

pex: E: tags for 'macosx-10.12-intel' x PythonInterpreter('/System/Library/Frameworks/Python.framework/Versions/2.7/Resources/Python.app/Contents/MacOS/Python', PythonIdentity('CPython', 2, 7, 10), {('matplotlib', '1.3.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-corelocation', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-coretext', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-syncservices', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('six', '1.4.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-fsevents', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-cocoa', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-preferencepanes', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('zope.interface', '4.1.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyparsing', '2.0.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-collaboration', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pytz', '2013.7'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-pubsub', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-scriptingbridge', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-servicemanagement', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('numpy', '1.8.0rc1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-installerplugins', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('setuptools', '18.5'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-opendirectory', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-applescriptkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-automator', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-addressbook', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-searchkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-applescriptobjc', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-qtkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-latentsemanticmapping', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('scipy', '0.13.0b1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('modulegraph', '0.10.4'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('altgraph', '0.10.2'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-instantmessage', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('xattr', '0.6.4'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-eventkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-social', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-cfnetwork', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-quartz', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-exceptionhandling', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-launchservices', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-coredata', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('python', '2.7.10'): '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/lib-dynload', ('bdist-mpkg', '0.5.0'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-message', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-inputmethodkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('py2app', '0.7.3'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-dictionaryservices', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-accounts', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyobjc-framework-screensaver', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('pyopenssl', '0.13.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('bonjour-py', '0.3'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-core', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('wsgiref', '0.1.2'): '/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7', ('pyobjc-framework-webkit', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('python-dateutil', '1.5'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python', ('pyobjc-framework-systemconfiguration', '2.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/PyObjC', ('macholib', '1.5.1'): '/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python'}) -> [('cp27', 'cp27m', 'macosx_10_8_universal'), ('cp27', 'cp27m', 'macosx_10_12_universal'), ('cp27', 'cp27mu', 'macosx_10_7_intel'), ('cp27', 'cp27mu', 'macosx_10_4_intel'), ('cp27', 'cp27d', 'macosx_10_9_intel'), ('cp27', 'cp27d', 'macosx_10_12_universal'), ('cp27', 'cp27dm', 'macosx_10_8_intel'), ('cp27', 'cp27dmu', 'macosx_10_0_universal'), ('cp27', 'cp27d', 'macosx_10_11_intel'), ('cp27', 'cp27dmu', 'macosx_10_10_intel'), ('cp27', 'cp27dm', 'macosx_10_8_universal'), ('cp27', 'cp27dm', 'macosx_10_11_universal'), ('cp27', 'cp27m', 'macosx_10_4_universal'), ('cp27', 'cp27mu', 'macosx_10_9_intel'), ('cp27', 'cp27dm', 'macosx_10_0_universal'), ('py2', 'none', 'macosx_10_0_universal'), ('cp27', 'cp27dmu', 'macosx_10_12_intel'), ('cp27', 'cp27m', 'macosx_10_5_universal'), ('cp2', 'none', 'any'), ('cp27', 'cp27m', 'macosx_10_8_intel'), ('py2', 'none', 'macosx_10_11_universal'), ('cp27', 'cp27m', 'macosx_10_10_intel'), ('cp27', 'cp27u', 'macosx_10_9_intel'), ('py2', 'none', 'macosx_10_6_intel'), ('cp27', 'none', 'macosx_10_10_intel'), ('py2', 'none', 'any'), ('cp27', 'none', 'macosx_10_5_intel'), ('cp27', 'none', 'macosx_10_8_intel'), ('cp27', 'cp27mu', 'macosx_10_0_universal'), ('cp27', 'cp27u', 'macosx_10_9_universal'), ('cp27', 'cp27mu', 'macosx_10_11_universal'), ('cp27', 'none', 'macosx_10_12_intel'), ('py27', 'none', 'any'), ('cp27', 'cp27dm', 'macosx_10_4_intel'), ('cp27', 'cp27mu', 'macosx_10_6_intel'), ('cp27', 'none', 'macosx_10_6_intel'), ('cp27', 'cp27mu', 'macosx_10_9_universal'), ('cp27', 'cp27mu', 'macosx_10_5_intel'), ('cp27', 'cp27dmu', 'macosx_10_11_intel'), ('cp27', 'cp27dmu', 'macosx_10_5_universal'), ('cp27', 'cp27dmu', 'macosx_10_5_intel'), ('cp27', 'cp27u', 'macosx_10_3_universal'), ('cp27', 'cp27dmu', 'macosx_10_4_universal'), ('cp27', 'cp27dmu', 'macosx_10_6_intel'), ('cp27', 'cp27dm', 'macosx_10_7_intel'), ('cp27', 'none', 'macosx_10_12_universal'), ('cp27', 'cp27u', 'macosx_10_4_intel'), ('cp27', 'none', 'macosx_10_11_intel'), ('cp27', 'cp27u', 'macosx_10_0_universal'), ('cp27', 'cp27m', 'macosx_10_12_intel'), ('cp27', 'none', 'macosx_10_4_universal'), ('cp27', 'cp27dmu', 'macosx_10_11_universal'), ('cp27', 'cp27dmu', 'macosx_10_3_universal'), ('cp27', 'none', 'macosx_10_5_universal'), ('cp27', 'cp27dmu', 'macosx_10_12_universal'), ('py2', 'none', 'macosx_10_9_universal'), ('cp27', 'cp27dm', 'macosx_10_9_universal'), ('cp27', 'cp27u', 'macosx_10_11_universal'), ('cp27', 'cp27dm', 'macosx_10_1_universal'), ('cp27', 'cp27m', 'macosx_10_2_universal'), ('cp27', 'cp27d', 'macosx_10_8_intel'), ('cp27', 'cp27u', 'macosx_10_1_universal'), ('cp27', 'none', 'macosx_10_10_universal'), ('cp27', 'cp27d', 'macosx_10_5_intel'), ('cp27', 'cp27m', 'macosx_10_10_universal'), ('cp27', 'cp27mu', 'macosx_10_7_universal'), ('cp27', 'cp27mu', 'macosx_10_11_intel'), ('cp27', 'cp27dmu', 'macosx_10_6_universal'), ('cp27', 'none', 'macosx_10_2_universal'), ('cp27', 'cp27u', 'macosx_10_7_intel'), ('cp27', 'none', 'macosx_10_3_universal'), ('cp27', 'cp27d', 'macosx_10_4_intel'), ('cp27', 'cp27dm', 'macosx_10_9_intel'), ('py2', 'none', 'macosx_10_9_intel'), ('cp27', 'cp27m', 'macosx_10_7_intel'), ('cp27', 'cp27m', 'macosx_10_3_universal'), ('cp27', 'cp27m', 'macosx_10_5_intel'), ('cp27', 'cp27m', 'macosx_10_6_universal'), ('cp27', 'none', 'macosx_10_6_universal'), ('py2', 'none', 'macosx_10_7_universal'), ('py2', 'none', 'macosx_10_11_intel'), ('cp27', 'cp27m', 'macosx_10_6_intel'), ('cp27', 'cp27dmu', 'macosx_10_2_universal'), ('cp27', 'cp27m', 'macosx_10_1_universal'), ('cp27', 'cp27dmu', 'macosx_10_8_intel'), ('cp27', 'cp27d', 'macosx_10_11_universal'), ('cp27', 'cp27mu', 'macosx_10_12_intel'), ('cp27', 'cp27u', 'macosx_10_5_intel'), ('cp27', 'cp27u', 'macosx_10_8_universal'), ('cp27', 'cp27dm', 'macosx_10_6_universal'), ('cp27', 'none', 'macosx_10_9_universal'), ('cp27', 'cp27d', 'macosx_10_6_intel'), ('cp27', 'cp27mu', 'macosx_10_10_intel'), ('cp27', 'cp27u', 'macosx_10_7_universal'), ('cp27', 'cp27dmu', 'macosx_10_7_universal'), ('py2', 'none', 'macosx_10_12_intel'), ('cp27', 'cp27dmu', 'macosx_10_9_universal'), ('py2', 'none', 'macosx_10_8_universal'), ('py2', 'none', 'macosx_10_12_universal'), ('cp27', 'cp27mu', 'macosx_10_5_universal'), ('cp27', 'cp27d', 'macosx_10_3_universal'), ('cp27', 'cp27d', 'macosx_10_1_universal'), ('cp27', 'cp27m', 'macosx_10_7_universal'), ('py2', 'none', 'macosx_10_10_intel'), ('cp27', 'cp27mu', 'macosx_10_4_universal'), ('cp27', 'cp27dm', 'macosx_10_12_universal'), ('cp27', 'cp27d', 'macosx_10_10_universal'), ('cp27', 'cp27u', 'macosx_10_11_intel'), ('cp27', 'cp27dmu', 'macosx_10_10_universal'), ('cp27', 'cp27m', 'macosx_10_0_universal'), ('cp27', 'cp27mu', 'macosx_10_8_universal'), ('cp27', 'cp27mu', 'macosx_10_12_universal'), ('cp27', 'cp27d', 'macosx_10_2_universal'), ('cp27', 'cp27d', 'macosx_10_7_intel'), ('cp27', 'cp27dm', 'macosx_10_6_intel'), ('cp27', 'cp27m', 'macosx_10_4_intel'), ('py2', 'none', 'macosx_10_4_universal'), ('cp27', 'none', 'macosx_10_4_intel'), ('py2', 'none', 'macosx_10_5_intel'), ('cp27', 'cp27d', 'macosx_10_6_universal'), ('cp27', 'cp27dm', 'macosx_10_5_universal'), ('cp27', 'cp27dm', 'macosx_10_4_universal'), ('cp27', 'none', 'macosx_10_1_universal'), ('cp27', 'cp27dmu', 'macosx_10_4_intel'), ('cp27', 'cp27u', 'macosx_10_2_universal'), ('cp27', 'cp27mu', 'macosx_10_6_universal'), ('cp27', 'cp27mu', 'macosx_10_2_universal'), ('cp27', 'cp27d', 'macosx_10_9_universal'), ('cp27', 'cp27dm', 'macosx_10_10_universal'), ('cp27', 'cp27dmu', 'macosx_10_1_universal'), ('cp27', 'cp27mu', 'macosx_10_8_intel'), ('cp27', 'cp27u', 'macosx_10_10_intel'), ('cp27', 'cp27dmu', 'macosx_10_9_intel'), ('cp27', 'none', 'macosx_10_8_universal'), ('py2', 'none', 'macosx_10_6_universal'), ('cp27', 'cp27dm', 'macosx_10_12_intel'), ('cp27', 'cp27d', 'macosx_10_0_universal'), ('cp27', 'none', 'any'), ('cp27', 'cp27m', 'macosx_10_9_universal'), ('cp27', 'cp27d', 'macosx_10_7_universal'), ('cp27', 'none', 'macosx_10_9_intel'), ('py2', 'none', 'macosx_10_8_intel'), ('cp27', 'cp27dm', 'macosx_10_2_universal'), ('cp27', 'cp27m', 'macosx_10_11_universal'), ('cp27', 'cp27u', 'macosx_10_6_universal'), ('cp27', 'cp27dm', 'macosx_10_10_intel'), ('cp27', 'cp27dmu', 'macosx_10_8_universal'), ('cp27', 'cp27d', 'macosx_10_12_intel'), ('cp27', 'cp27u', 'macosx_10_12_intel'), ('cp27', 'none', 'macosx_10_7_intel'), ('cp27', 'cp27u', 'macosx_10_6_intel'), ('py2', 'none', 'macosx_10_3_universal'), ('py2', 'none', 'macosx_10_5_universal'), ('py2', 'none', 'macosx_10_2_universal'), ('cp27', 'cp27dmu', 'macosx_10_7_intel'), ('cp27', 'cp27d', 'macosx_10_10_intel'), ('cp27', 'cp27u', 'macosx_10_4_universal'), ('cp27', 'cp27mu', 'macosx_10_3_universal'), ('py2', 'none', 'macosx_10_1_universal'), ('py2', 'none', 'macosx_10_10_universal'), ('cp27', 'cp27dm', 'macosx_10_5_intel'), ('cp27', 'cp27d', 'macosx_10_5_universal'), ('cp27', 'cp27u', 'macosx_10_5_universal'), ('cp27', 'cp27dm', 'macosx_10_11_intel'), ('cp27', 'cp27dm', 'macosx_10_7_universal'), ('cp27', 'cp27u', 'macosx_10_12_universal'), ('cp27', 'cp27d', 'macosx_10_4_universal'), ('cp27', 'cp27u', 'macosx_10_10_universal'), ('cp27', 'cp27m', 'macosx_10_9_intel'), ('py2', 'none', 'macosx_10_7_intel'), ('py2', 'none', 'macosx_10_4_intel'), ('cp27', 'cp27d', 'macosx_10_8_universal'), ('cp27', 'cp27u', 'macosx_10_8_intel'), ('cp27', 'none', 'macosx_10_0_universal'), ('cp27', 'cp27mu', 'macosx_10_1_universal'), ('cp27', 'cp27mu', 'macosx_10_10_universal'), ('cp27', 'cp27m', 'macosx_10_11_intel'), ('cp27', 'none', 'macosx_10_11_universal'), ('cp27', 'none', 'macosx_10_7_universal'), ('cp27', 'cp27dm', 'macosx_10_3_universal')]

Eric-Arellano · 2018-07-13T13:49:08Z

Awesome! I'm running ci.sh on mac - will let you know what happens.

Eric-Arellano · 2018-07-13T14:44:59Z

6 tests fail now instead of I think it was 11 earlier, that's exciting! https://hastebin.com/oxinuwepol.rb

One of the failures is the same as CI's unit test failure, TestBuildLocalPythonDistributions.test_python_create_platform_specific_distribution

jsirois · 2018-07-13T14:46:40Z

@Eric-Arellano note well that the OSX fix for Travis was ultimately getting off an OSX VM with Apple's supplied 2.7.10 which returns a broken -intel build platform. If you too have that, this may not provide you relief locally yet.

Previously the platform passed for 'current' was partial, leading to ambiguous resolves. Further, partial platforms in user's BUILD files also would lead to ambiguous resolves. For example, a multi-platform `python_binary` with something like `platforms=['current', 'linux-x86_64', 'macosx_10.11_x86_64']`. Expand and and then fixup platforms as required by replacing `get_local_platform` with `expand_and_maybe_adjust_platform`.

+ Fix an unused `interpreter` parameter - forward to `SetupPyRunner`. + Kill declaring `--universal` on behalf of the package author. There is not enough info to make the decision if the code being dist'd, if pure python, is 2/3 compatible. To compensate for the removal of `--universal`, give an example of how to declare your python_dist as `--universal` when you know it is as the BUILD `python_dist` target author.

jsirois · 2018-07-13T16:37:16Z

Thanks @Eric-Arellano - I was able to repro that failure on linux and it should be fixed this round.

kwlzn

lgtm!

kwlzn · 2018-07-13T17:40:34Z

src/python/pants/backend/python/pex_util.py

+                  abi_tag=ident.abi_tag))
+
+
+def expand_and_maybe_adjust_platform(interpreter, platform):


a TODO to upstream this into pex might be good?

Yup - this old one pex-tool/pex#511, I'll add a TODO similar to the method above.

kwlzn · 2018-07-13T17:43:08Z

src/python/pants/backend/python/tasks/wrapped_pex.py

+    env = copy(kwargs.pop('env', {}))
+
+    # Hack around bug in PEX where custom interpreters are not forwarded to PEXEnvironments.
+    # TODO(John Sirois): Remove when XXX is fixed.


s/XXX/a ticket?

Yup - just finished filing on the pex side. I'll send up a final comment-only diff with issue inks here as soon as this goes green then merge: pex-tool/pex#522

jsirois · 2018-07-13T18:01:23Z

.travis.yml

@@ -476,6 +476,10 @@ matrix:
      language: generic
      env:
        - SHARD="Rust + Platform-specific Tests OSX"
+        # Specifically avoid the OSX provided 2.7.10 under xcode8.3 since it returns a platform
+        # of `macosx-*-intel` where the `intel` suffix is bogus but pex has not yet been taught to
+        # deal with this. Can be removed when this issue is resolved: YYY


YYY -> pex-tool/pex#523

jsirois mentioned this pull request Jul 12, 2018

Use extended platform specification for current platform in resolve_multi. #6099

Closed

stuhood approved these changes Jul 12, 2018

View reviewed changes

kwlzn approved these changes Jul 12, 2018

View reviewed changes

jsirois commented Jul 13, 2018

View reviewed changes

jsirois force-pushed the python/resolve/fix-current-platform-handling branch from 9bd378a to ab976db Compare July 13, 2018 09:54

jsirois force-pushed the python/resolve/fix-current-platform-handling branch 2 times, most recently from 6196ae2 to 503b193 Compare July 13, 2018 13:05

jsirois added 4 commits July 13, 2018 10:11

Hack around PEX not forwarding custom interpreters to PEXEnvironment.

2c69db0

Avoid the troublesome Apple 2.7.10 python for now.

fe2907e

jsirois force-pushed the python/resolve/fix-current-platform-handling branch from 833d40b to fe2907e Compare July 13, 2018 16:12

jsirois mentioned this pull request Jul 13, 2018

PEX fails to pass a custom interpreter down to PEXEnvironments it _activates pex-tool/pex#522

Closed

kwlzn approved these changes Jul 13, 2018

View reviewed changes

jsirois mentioned this pull request Jul 13, 2018

OSX's python 2.7.10 interpreter reports a bad local platform, bdists fail to resolve pex-tool/pex#523

Closed

jsirois commented Jul 13, 2018

View reviewed changes

Finish up TODOs with issues.

fae5bd9

jsirois merged commit a1696c8 into pantsbuild:master Jul 13, 2018

jsirois deleted the python/resolve/fix-current-platform-handling branch July 13, 2018 18:15

jsirois mentioned this pull request Jul 13, 2018

Rust/Cargo CI perf improvements #6119

Closed

cosmicexplorer mentioned this pull request Jul 15, 2018

add the target fingerprint to the version of each local dist so that we don't use the first cached one #6022

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix 'current' platform handling. #6104

Fix 'current' platform handling. #6104

jsirois commented Jul 12, 2018 •

edited

Loading

jsirois commented Jul 12, 2018

jsirois commented Jul 12, 2018

Eric-Arellano commented Jul 12, 2018

jsirois commented Jul 12, 2018

CMLivingston commented Jul 12, 2018

jsirois commented Jul 12, 2018

CMLivingston commented Jul 12, 2018 •

edited

Loading

jsirois commented Jul 13, 2018

jsirois commented Jul 13, 2018

jsirois commented Jul 13, 2018

jsirois commented Jul 13, 2018 •

edited

Loading

jsirois Jul 13, 2018

cosmicexplorer commented Jul 13, 2018 •

edited

Loading

jsirois commented Jul 13, 2018

Eric-Arellano commented Jul 13, 2018

Eric-Arellano commented Jul 13, 2018

jsirois commented Jul 13, 2018

jsirois commented Jul 13, 2018

kwlzn left a comment

kwlzn Jul 13, 2018

jsirois Jul 13, 2018

jsirois Jul 13, 2018

kwlzn Jul 13, 2018

jsirois Jul 13, 2018

jsirois Jul 13, 2018

jsirois Jul 13, 2018

jsirois Jul 13, 2018

		abi_tag=ident.abi_tag))


		def expand_and_maybe_adjust_platform(interpreter, platform):

Fix 'current' platform handling. #6104

Fix 'current' platform handling. #6104

Conversation

jsirois commented Jul 12, 2018 • edited Loading

jsirois commented Jul 12, 2018

jsirois commented Jul 12, 2018

Eric-Arellano commented Jul 12, 2018

jsirois commented Jul 12, 2018

CMLivingston commented Jul 12, 2018

jsirois commented Jul 12, 2018

CMLivingston commented Jul 12, 2018 • edited Loading

jsirois commented Jul 13, 2018

jsirois commented Jul 13, 2018

jsirois commented Jul 13, 2018

jsirois commented Jul 13, 2018 • edited Loading

Choose a reason for hiding this comment

cosmicexplorer commented Jul 13, 2018 • edited Loading

jsirois commented Jul 13, 2018

Eric-Arellano commented Jul 13, 2018

Eric-Arellano commented Jul 13, 2018

jsirois commented Jul 13, 2018

jsirois commented Jul 13, 2018

kwlzn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsirois commented Jul 12, 2018 •

edited

Loading

CMLivingston commented Jul 12, 2018 •

edited

Loading

jsirois commented Jul 13, 2018 •

edited

Loading

cosmicexplorer commented Jul 13, 2018 •

edited

Loading