Introduce Clang, GCC, and Binutils to form a native toolchain

This is the first instance in Pants of selectively instantiating a subsystem based on the platform Pants is being invoked on, because gnu binutils (or at least, ld) doesn't support OSX. The XCode cli tools will be relied on instead for linking (more work needed here on effective error messages). Create by squashing 119 commits, most of which had been merged into master: many of these commits were merged with the first python_dist PR, or the PR that introduced the "LLVM" subsystem (now named "Clang"). Squashed commit of the following: commit 9ae7284 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Wed Mar 21 15:03:38 2018 -0700 add some more comments to explain what i'm thinking commit 8d48d0a Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Wed Mar 21 14:52:07 2018 -0700 some mild refactoring and more documentation commit 29e9622 Merge: ba727b7 a5410b6 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Wed Mar 21 12:52:28 2018 -0700 Merge branch 'master' into dmcclanahan/python-dist-c++-sources commit ba727b7 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Tue Mar 20 02:10:40 2018 -0700 gcc is linux specific again (but only for a little bit) commit aaae18c Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Fri Mar 16 15:26:48 2018 -0700 correct gcc to non-platform specific, but remove from osx for now commit dfe7340 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Fri Mar 16 14:38:39 2018 -0700 add platform-specific native toolchain and use to build python dists commit f06c83f Merge: f22dfca f9ee6fd Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Fri Mar 16 12:20:11 2018 -0700 Merge branch 'master' into dmcclanahan/python-dist-c++-sources commit f22dfca Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Wed Mar 7 16:27:59 2018 -0800 add fixme commit 7d76e70 Merge: a35256d d77483e Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Fri Mar 2 15:47:08 2018 -0800 Merge branch 'master' into dmcclanahan/python-dist-c++-sources commit a35256d Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Thu Mar 1 11:51:29 2018 -0800 update name of `compiler` package to `clang` commit 529324e Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Thu Feb 22 12:07:52 2018 -0800 fix lint errors commit 7763e5d Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Thu Feb 22 11:46:58 2018 -0800 rename LLVM -> Compiler commit 91e1353 Merge: 6a68876 1ed1fdb Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Thu Feb 22 11:28:16 2018 -0800 Merge branch 'dmcclanahan/python-dist-c++-sources' of github.com:cosmicexplorer/pants into dmcclanahan/python-dist-c++-sources commit 6a68876 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Wed Feb 21 11:45:30 2018 -0800 refactor out unnecessary interpreter wrapper to use a contextmanager commit 1ed1fdb Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Wed Feb 21 11:45:30 2018 -0800 refactor out unnecessary interpreter wrapper to use a contextmanager commit 41a1a9a Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Tue Feb 20 14:17:23 2018 -0800 revert unnecessary python_dist changes commit 1cfd60a Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Tue Feb 20 14:12:44 2018 -0800 make final simplifications commit a4ebbbd Merge: 7783678 71a33d6 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Tue Feb 20 13:57:19 2018 -0800 Merge branch 'master' into dmcclanahan/python-dist-c++-sources commit 7783678 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Tue Feb 20 13:45:38 2018 -0800 remove more implementation artifacts commit 88851f1 Merge: c7c90b2 c014e8d Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Tue Feb 20 10:44:14 2018 -0800 Merge branch 'master' into dmcclanahan/python-dist-c++-sources commit c7c90b2 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Fri Feb 16 07:20:37 2018 -0800 refactor unnecessary complexity commit 35899ee Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Thu Feb 15 05:47:49 2018 -0800 try to understand what makes the breaking change commit d9fc2c4 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Wed Feb 14 13:55:02 2018 -0800 remove comments in sandboxed_interpreter.py commit ceae8bb Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Wed Feb 14 13:51:46 2018 -0800 slim down the pull request quite a bit commit 48d7a93 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Wed Feb 14 13:27:13 2018 -0800 rewrite BinaryTool a little and remove native toolchain subsystem commit cfe5c86 Merge: b1d39ec f55260a Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Wed Feb 14 08:34:39 2018 -0800 Merge branch 'master' of github.com:pantsbuild/pants into dmcclanahan/python-dist-c++-sources commit b1d39ec Merge: 55dbd7e 7cdea9a Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Tue Feb 13 11:54:50 2018 -0800 Merge branch 'master' into dmcclanahan/python-dist-c++-sources commit 55dbd7e Author: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Mon Feb 12 19:08:59 2018 -0800 suddenly, with very few changes, everything "just works"! commit 608769e Merge: badd80f 569f14c Author: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Mon Feb 12 19:05:19 2018 -0800 Merge branch 'master' into dmcclanahan/python-dist-c++-sources commit badd80f Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Mon Feb 5 16:09:46 2018 -0800 add llvm distribution support, not just clang commit 0ac2bbe Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Fri Feb 2 16:33:37 2018 -0800 remove source copying contextmanager and set clang arch for setup.py commit 6c52529 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Fri Feb 2 12:16:55 2018 -0800 try cpp module sources and comment out future work commit f33677b Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Fri Feb 2 11:39:58 2018 -0800 add some more context, leave everything in an inconsistent state commit 5a6ad48 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Thu Feb 1 12:07:56 2018 -0800 plumb in a native toolchain subsystem commit 5ed218e Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Thu Feb 1 11:11:10 2018 -0800 use a contextmanager to copy source files and edit PYTHONPATH commit 3155138 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Wed Jan 31 20:47:16 2018 -0800 now we can declare cpp_sources in python_dist targets! commit 2591288 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Wed Jan 31 03:17:57 2018 -0800 cut off work for now commit 629ff46 Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com> Date: Wed Jan 31 01:02:38 2018 -0800 clean up PythonDistribution and add c_sources field commit 9e3d2ca Author: Chris Livingston <clivingston@twitter.com> Date: Tue Jan 30 14:36:45 2018 -0800 Remove mod to travis yml commit 1a04e80 Author: Chris Livingston <clivingston@twitter.com> Date: Tue Jan 30 13:13:37 2018 -0800 Rebase with master commit b4d367c Author: Chris Livingston <clivingston@twitter.com> Date: Wed Jan 24 17:59:30 2018 -0800 Rename superhello to fasthello commit 59164c1 Author: Chris Livingston <clivingston@twitter.com> Date: Wed Jan 24 15:26:03 2018 -0800 Add remove command to travis.yml to remove problematic file from failing CI target commit 8ac832b Author: Chris Livingston <clivingston@twitter.com> Date: Wed Jan 24 13:00:12 2018 -0800 Add clean-all statements to integration tests to gauge flakiness commit a787717 Author: Chris Livingston <clivingston@twitter.com> Date: Wed Jan 24 10:39:22 2018 -0800 Slightly modify test assertion for conflicting deps test commit 0b82067 Author: Chris Livingston <clivingston@twitter.com> Date: Tue Jan 23 15:02:47 2018 -0800 Resolve merge conflicts from removal of tasks2 commit 7f757c0 Author: Chris Livingston <clivingston@twitter.com> Date: Mon Jan 22 12:28:18 2018 -0800 Disallow dependencies on a python_dist target commit 3024a34 Author: Chris Livingston <clivingston@twitter.com> Date: Fri Jan 19 11:17:45 2018 -0800 Add xfail test to testprojects tests commit eb8598d Author: Chris Livingston <clivingston@twitter.com> Date: Thu Jan 18 17:26:24 2018 -0800 Fix lint commit 654c20e Author: Chris Livingston <clivingston@twitter.com> Date: Thu Jan 18 14:13:42 2018 -0800 Remove unused import commit 22b72f0 Author: Chris Livingston <clivingston@twitter.com> Date: Thu Jan 18 11:39:49 2018 -0800 Remove add_labels from PythonDistribution object commit a4150d0 Author: Chris Livingston <clivingston@twitter.com> Date: Wed Jan 17 19:17:59 2018 -0800 Remove duplicate functions to enforce DRY commit 178bcbd Author: Chris Livingston <clivingston@twitter.com> Date: Wed Jan 17 14:21:47 2018 -0800 Fix issues with CI and failing testprojects target commit cae065c Author: Chris Livingston <clivingston@twitter.com> Date: Mon Jan 15 23:44:32 2018 -0800 Add integration tests for targets that conflict with transitive deps listed in the install_requires field of a python distribution's setup.py commit ffae43d Author: Chris Livingston <clivingston@twitter.com> Date: Mon Jan 15 22:36:45 2018 -0800 Remove crufty files commit c6edc08 Author: Chris Livingston <clivingston@twitter.com> Date: Fri Jan 12 17:23:46 2018 -0800 remove unneeded dependency test commit 2925530 Author: Chris Livingston <clivingston@twitter.com> Date: Fri Jan 12 17:21:33 2018 -0800 Edge case impl for same setup.py package name/version as a binary dep commit fe8dd55 Author: Chris Livingston <clivingston@twitter.com> Date: Thu Jan 11 22:36:45 2018 -0800 Simplify a few lines, add check for ambiguous python dists, and fix copyright date commit aabf88f Author: Chris Livingston <clivingston@twitter.com> Date: Thu Jan 11 18:35:34 2018 -0800 Update TODO github issue link commit a6377e4 Author: Chris Livingston <clivingston@twitter.com> Date: Thu Jan 11 16:36:12 2018 -0800 Remove tests that break testprojects integration testing commit d41e7ee Author: Chris Livingston <clivingston@twitter.com> Date: Wed Jan 10 16:29:25 2018 -0800 Remove unnecessary targets from BUILD file in superhello_testproject commit 4fd1567 Author: Chris Livingston <clivingston@twitter.com> Date: Wed Jan 10 14:44:23 2018 -0800 Cleanup integration test and move superhello test project to examples/tests due to test breakage when placed in testprojects commit e8f6b98 Author: Chris Livingston <clivingston@twitter.com> Date: Wed Jan 10 11:59:32 2018 -0800 Fix multiple binary target case and add integration test commit 9e0b2d6 Author: Chris Livingston <clivingston@twitter.com> Date: Tue Jan 9 15:57:03 2018 -0800 Add TODO with github link for package conflict case in python dist backend commit a2116c4 Author: Chris Livingston <clivingston@twitter.com> Date: Tue Jan 9 15:20:25 2018 -0800 Remove cruft commit 03bc4c0 Author: Chris Livingston <clivingston@twitter.com> Date: Mon Jan 8 16:15:21 2018 -0800 Remove unneccessary checks for invalid targets and streamline method signatures commit 4e9bb6b Author: Chris Livingston <clivingston@twitter.com> Date: Fri Jan 5 18:24:44 2018 -0800 Fix install directory clobbering and setup.py positioning commit f1ed6da Author: Chris Livingston <clivingston@twitter.com> Date: Fri Jan 5 18:23:02 2018 -0800 Fix install directory clobbering and setup.py positioning commit be2583a Author: Lionel Vital <lvital@twitter.com> Date: Fri Jan 5 17:48:46 2018 -0800 Addresses a few changes commit 022b1e7 Author: Chris Livingston <clivingston@twitter.com> Date: Thu Dec 21 16:50:35 2017 -0800 Add rjiang suggestion for counting setup.py files commit e2bf445 Author: Chris Livingston <clivingston@twitter.com> Date: Thu Dec 21 15:16:35 2017 -0800 Clean up comments, docstrings, and fix broken testprojects integration tests commit 3e1739f Author: Chris Livingston <clivingston@twitter.com> Date: Thu Dec 21 12:42:23 2017 -0800 Fix merge conflict commit 365e708 Author: Chris Livingston <clivingston@twitter.com> Date: Thu Dec 21 12:30:08 2017 -0800 Style fixes and cruftslaying per rjiang's comments commit 5f4e07c Author: Chris Livingston <clivingston@twitter.com> Date: Thu Dec 21 12:10:02 2017 -0800 Add detection of multiple setup.py files and throw an error. commit 056e267 Author: Chris Livingston <clivingston@twitter.com> Date: Wed Dec 20 17:50:17 2017 -0800 Add integration testing and simple unit test for python create distributions task. Also cleanup code to DRY by creating util helper method and streamline invalid python dist target detection. commit 401a10d Author: Chris Livingston <clivingston@twitter.com> Date: Tue Dec 19 18:12:14 2017 -0800 Working goals using invalidated blocks commit 53a1355 Author: Chris Livingston <clivingston@twitter.com> Date: Tue Dec 19 16:40:56 2017 -0800 Solid working state based off of vt.results dir caching commit dacd672 Author: Chris Livingston <clivingston@twitter.com> Date: Tue Dec 19 15:57:37 2017 -0800 Working caching under vt.results dir. Moving to tests. commit edf110d Author: Lionel Vital <lvital@twitter.com> Date: Tue Dec 19 14:58:12 2017 -0800 More cleanup + move stuff in task execution under invalidated commit 8c7b658 Author: Lionel Vital <lvital@twitter.com> Date: Fri Dec 15 15:43:56 2017 -0800 Easy nits and whitespace issues commit 31d3a91 Author: Lionel Vital <lvital@twitter.com> Date: Fri Dec 15 14:14:19 2017 -0800 Another whitespace error commit 177c379 Author: Lionel Vital <lvital@twitter.com> Date: Fri Dec 15 12:32:06 2017 -0800 Whitespace lint fixes and unused import commit 7b8e8be Author: Chris Livingston <clivingston@twitter.com> Date: Thu Dec 14 17:43:54 2017 -0800 Remove unused imports created from refactor commit 51e9c64 Author: Chris Livingston <clivingston@twitter.com> Date: Thu Dec 14 14:44:11 2017 -0800 Remove unused imports and add guard from consuming python dist products commit 59ff560 Author: Chris Livingston <clivingston@twitter.com> Date: Wed Dec 13 16:52:16 2017 -0800 Add guard statement for case where pants test run does not require data from PythonCreateDistributions task. commit fb1c903 Author: Chris Livingston <clivingston@twitter.com> Date: Wed Dec 13 14:27:04 2017 -0800 Fix minor BUILD file error to pass CI commit 23f6999 Author: Chris Livingston <clivingston@twitter.com> Date: Tue Dec 12 16:37:39 2017 -0800 Working distribution create task and integration for pants run/binary/test commit 314ecfb Author: Chris Livingston <clivingston@twitter.com> Date: Tue Dec 12 16:05:10 2017 -0800 pants test example commit a859f46 Author: Chris Livingston <clivingston@twitter.com> Date: Mon Dec 11 18:07:08 2017 -0800 Remove cruft commit 6881a72 Author: Chris Livingston <clivingston@twitter.com> Date: Mon Dec 11 18:01:16 2017 -0800 Revert pants run mods and fix bug in pex build util commit 1c981f7 Author: Lionel Vital <lvital@twitter.com> Date: Mon Dec 11 17:36:58 2017 -0800 Run/test commit 8436dad Author: Lionel Vital <lvital@twitter.com> Date: Mon Dec 11 17:14:20 2017 -0800 More cleanup of docs commit b6fa385 Author: Lionel Vital <lvital@twitter.com> Date: Mon Dec 11 16:48:33 2017 -0800 Minor cleanup + rename alias to be standard commit 4b64f2d Author: Chris Livingston <clivingston@twitter.com> Date: Fri Dec 8 16:08:53 2017 -0800 Further iteration on python distribution task, now functioning for python binary create commit 76aeda2 Author: Chris Livingston <clivingston@twitter.com> Date: Fri Dec 8 15:20:34 2017 -0800 Working distribution creation task commit 7823d2a Author: Chris Livingston <clivingston@twitter.com> Date: Thu Dec 7 17:37:24 2017 -0800 Progress on pex_build_util commit e4c0556 Author: Chris Livingston <clivingston@twitter.com> Date: Thu Dec 7 10:02:26 2017 -0800 Install Python Dist create task commit 4b65461 Author: Chris Livingston <clivingston@twitter.com> Date: Tue Dec 5 17:51:57 2017 -0800 Debug session 1 commit c340abf Author: Chris Livingston <clivingston@twitter.com> Date: Tue Dec 5 17:22:32 2017 -0800 Create python distribution task and refactor example commit 9bb228f Author: Lionel Vital <lvital@twitter.com> Date: Mon Dec 4 16:05:01 2017 -0800 PythonDistribution -> inherit Target instead of PythonTarget commit 7297096 Author: Chris Livingston <clivingston@twitter.com> Date: Fri Dec 1 12:41:17 2017 -0800 First pass at kwlzn change suggestions commit e2f84cc Author: Chris Livingston <clivingston@twitter.com> Date: Wed Nov 29 10:38:58 2017 -0800 Add python_distribution to backend/targets build file commit 194452b Author: Chris Livingston <clivingston@twitter.com> Date: Tue Nov 28 16:50:14 2017 -0800 pants run task implementation - first pass commit 6585142 Author: Lionel Vital <lvital@twitter.com> Date: Tue Nov 28 16:11:15 2017 -0800 First stab at run task commit 0ae435e Author: Lionel Vital <lvital@twitter.com> Date: Tue Nov 28 15:37:57 2017 -0800 Remove excess __init__.py files commit 1940c53 Author: Chris Livingston <clivingston@twitter.com> Date: Tue Nov 28 15:22:47 2017 -0800 Clean up misc style issues and delete unnecessary file. commit 9cd32a4 Author: Lionel Vital <lvital@twitter.com> Date: Tue Nov 28 12:59:37 2017 -0800 Clean up + add some docs commit c8a954e Author: Chris Livingston <clivingston@twitter.com> Date: Tue Nov 28 12:16:26 2017 -0800 Add cleanup of egg-info in pex build util commit 83be228 Author: Chris Livingston <clivingston@twitter.com> Date: Fri Nov 24 10:32:26 2017 -0800 Add tensorflow dependency example to superhello; this puts ./pants binary on main:main in a good state for a workable demo commit 74bfb8c Author: Chris Livingston <clivingston@twitter.com> Date: Tue Nov 21 16:45:55 2017 -0800 Finalized initial approach at packaging wheels w/ c sources into a pex commit 4e2b1b9 Author: Lionel Vital <lvital@twitter.com> Date: Tue Nov 21 15:02:05 2017 -0800 Minor cleanup commit dd99194 Author: Chris Livingston <clivingston@twitter.com> Date: Tue Nov 21 14:43:19 2017 -0800 Working superhello project equivalent to backends/tensorflow commit 5837f30 Author: Lionel Vital <lvital@twitter.com> Date: Tue Nov 21 14:27:00 2017 -0800 Modify example to use superhello commit 4f9bc98 Author: Chris Livingston <clivingston@twitter.com> Date: Tue Nov 21 13:51:52 2017 -0800 Working python dist example commit cd32058 Author: Chris Livingston <clivingston@twitter.com> Date: Tue Nov 21 12:22:23 2017 -0800 Changes to python binary creation + add new target definition commit 3aa0fe4 Author: Chris Livingston <clivingston@twitter.com> Date: Tue Nov 21 12:19:55 2017 -0800 Changes to hello2 package commit 46eab74 Author: Chris Livingston <clivingston@twitter.com> Date: Tue Nov 21 12:19:22 2017 -0800 Working package example for hello2 commit ca86bf6 Author: Chris Livingston <clivingston@twitter.com> Date: Wed Nov 15 13:01:29 2017 -0800 First pass at distribution task and target commit 3ec11c2 Author: Lionel Vital <lvital@twitter.com> Date: Thu Nov 9 23:27:59 2017 -0800 Python_distribution example
pantsbuild · Mar 22, 2018 · 3c3c7d2 · 3c3c7d2
1 parent a5410b6
commit 3c3c7d2
Show file tree

Hide file tree

Showing 14 changed files with 257 additions and 15 deletions.
diff --git a/build-support/docker/centos6/Dockerfile b/build-support/docker/centos6/Dockerfile
@@ -16,6 +16,12 @@ RUN yum install -y \
   libffi \
   libffi-devel \
   openssl-devel
+# LLVM release tarballs are xz-compressed.
+RUN yum install -y xz
+# GCC requires these to compile.
+RUN yum install -y \
+  m4 \
+  wget
 
 # By default, execute in an environment with python27 enabled.
 ENTRYPOINT ["/usr/bin/scl", "enable", "python27", "devtoolset-7",  "--"]
diff --git a/src/python/pants/backend/native/subsystems/BUILD b/src/python/pants/backend/native/subsystems/BUILD
@@ -4,5 +4,9 @@
 python_library(
   dependencies=[
     'src/python/pants/binaries:binary_util',
+    'src/python/pants/binaries:execution_environment_mixin',
+    'src/python/pants/subsystem',
+    'src/python/pants/util:memo',
+    'src/python/pants/util:osutil',
   ],
 )
diff --git a/src/python/pants/backend/native/subsystems/clang.py b/src/python/pants/backend/native/subsystems/clang.py
@@ -0,0 +1,20 @@
+# coding=utf-8
+# Copyright 2018 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
+                        unicode_literals, with_statement)
+
+import os
+
+from pants.binaries.binary_tool import NativeTool
+from pants.binaries.execution_environment_mixin import ExecutionPathEnvironment
+
+
+class Clang(NativeTool, ExecutionPathEnvironment):
+  options_scope = 'clang'
+  default_version = '6.0.0'
+  archive_type = 'tgz'
+
+  def get_additional_paths(self):
+    return [os.path.join(self.select(), 'bin')]
diff --git a/...n/pants/backend/native/subsystems/llvm.py → ...on/pants/backend/native/subsystems/gcc.py b/...n/pants/backend/native/subsystems/llvm.py → ...on/pants/backend/native/subsystems/gcc.py
@@ -5,10 +5,16 @@
 from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
                         unicode_literals, with_statement)
 
+import os
+
 from pants.binaries.binary_tool import NativeTool
+from pants.binaries.execution_environment_mixin import ExecutionPathEnvironment
 
 
-class LLVM(NativeTool):
-  options_scope = 'llvm'
-  default_version = '5.0.1'
+class GCC(NativeTool, ExecutionPathEnvironment):
+  options_scope = 'gcc'
+  default_version = '7.3.0'
   archive_type = 'tgz'
+
+  def get_additional_paths(self):
+    return [os.path.join(self.select(), 'bin')]
diff --git a/src/python/pants/backend/native/subsystems/native_toolchain.py b/src/python/pants/backend/native/subsystems/native_toolchain.py
@@ -0,0 +1,47 @@
+# coding=utf-8
+# Copyright 2018 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
+                        unicode_literals, with_statement)
+
+from contextlib import contextmanager
+
+from pants.backend.native.subsystems.clang import Clang
+from pants.backend.native.subsystems.gcc import GCC
+from pants.backend.native.subsystems.platform_specific.linux.binutils import Binutils
+from pants.binaries.execution_environment_mixin import ExecutionEnvironmentMixin
+from pants.subsystem.subsystem import Subsystem
+from pants.util.memo import memoized_property
+from pants.util.osutil import get_os_name, normalize_os_name
+
+
+class NativeToolchain(Subsystem, ExecutionEnvironmentMixin):
+
+  options_scope = 'native-toolchain'
+
+  PLATFORM_SPECIFIC_TOOLCHAINS = {
+    # TODO(cosmicexplorer): 'darwin' should have everything here, but there's no
+    # open-source linker for OSX...yet.
+    'darwin': [GCC, Clang],
+    'linux': [GCC, Binutils, Clang],
+  }
+
+  @classmethod
+  def _get_platform_specific_toolchains(cls):
+    normed_os_name = normalize_os_name(get_os_name())
+    return cls.PLATFORM_SPECIFIC_TOOLCHAINS[normed_os_name]
+
+  @classmethod
+  def subsystem_dependencies(cls):
+    prev = super(NativeToolchain, cls).subsystem_dependencies()
+    cur_platform_subsystems = cls._get_platform_specific_toolchains()
+    return prev + tuple(sub.scoped(cls) for sub in cur_platform_subsystems)
+
+  @memoized_property
+  def _toolchain_instances(self):
+    cur_platform_subsystems = self._get_platform_specific_toolchains()
+    return [sub.scoped_instance(self) for sub in cur_platform_subsystems]
+
+  def modify_environment(self, env):
+    return self.apply_successive_env_modifications(env, self._toolchain_instances)
diff --git a/src/python/pants/backend/native/subsystems/platform_specific/__init__.py b/src/python/pants/backend/native/subsystems/platform_specific/__init__.py
diff --git a/src/python/pants/backend/native/subsystems/platform_specific/linux/BUILD b/src/python/pants/backend/native/subsystems/platform_specific/linux/BUILD
@@ -0,0 +1,9 @@
+# Copyright 2018 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+python_library(
+  dependencies=[
+    'src/python/pants/binaries:binary_util',
+    'src/python/pants/binaries:execution_environment_mixin',
+  ],
+)
diff --git a/src/python/pants/backend/native/subsystems/platform_specific/linux/__init__.py b/src/python/pants/backend/native/subsystems/platform_specific/linux/__init__.py
diff --git a/src/python/pants/backend/native/subsystems/platform_specific/linux/binutils.py b/src/python/pants/backend/native/subsystems/platform_specific/linux/binutils.py
@@ -0,0 +1,20 @@
+# coding=utf-8
+# Copyright 2018 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
+                        unicode_literals, with_statement)
+
+import os
+
+from pants.binaries.binary_tool import NativeTool
+from pants.binaries.execution_environment_mixin import ExecutionPathEnvironment
+
+
+class Binutils(NativeTool, ExecutionPathEnvironment):
+  options_scope = 'binutils'
+  default_version = '2.30'
+  archive_type = 'tgz'
+
+  def get_additional_paths(self):
+    return [os.path.join(self.select(), 'bin')]
diff --git a/src/python/pants/backend/python/subsystems/BUILD b/src/python/pants/backend/python/subsystems/BUILD
@@ -4,7 +4,10 @@
 python_library(
   dependencies = [
     '3rdparty/python:setuptools',
+    'src/python/pants/binaries:execution_environment_mixin',
+    'src/python/pants/backend/native',
     'src/python/pants/option',
     'src/python/pants/subsystem',
+    'src/python/pants/util:memo',
   ],
 )
diff --git a/src/python/pants/backend/python/subsystems/python_dist_build_environment.py b/src/python/pants/backend/python/subsystems/python_dist_build_environment.py
@@ -0,0 +1,37 @@
+# coding=utf-8
+# Copyright 2018 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
+                        unicode_literals, with_statement)
+
+from pants.backend.native.subsystems.native_toolchain import NativeToolchain
+from pants.binaries.execution_environment_mixin import ExecutionEnvironmentMixin
+from pants.subsystem.subsystem import Subsystem
+from pants.util.memo import memoized_property
+
+
+class PythonDistBuildEnvironment(Subsystem, ExecutionEnvironmentMixin):
+
+  options_scope = 'python-dist-build-environment'
+
+  @classmethod
+  def subsystem_dependencies(cls):
+    return super(PythonDistBuildEnvironment, cls).subsystem_dependencies() + (NativeToolchain.scoped(cls),)
+
+  @memoized_property
+  def _native_toolchain(self):
+    return NativeToolchain.scoped_instance(self)
+
+  def modify_environment(self, env):
+    env = self._native_toolchain.modify_environment(env)
+    # If we're going to be wrapping setup.py-based projects, we really should be
+    # doing it through a subclass of a distutils UnixCCompiler (in Lib/distutils
+    # in the CPython source) instead of hoping setup.py knows what to do. The
+    # default UnixCCompiler from distutils will build a 32/64-bit "fat binary"
+    # unless you use their undocumented ARCHFLAGS env var, and there may be more
+    # dragons later on.
+    env['CC'] = 'gcc'
+    env['CXX'] = 'g++'
+    env['ARCHFLAGS'] = '-arch x86_64'
+    return env
diff --git a/src/python/pants/backend/python/tasks/build_local_python_distributions.py b/src/python/pants/backend/python/tasks/build_local_python_distributions.py
@@ -8,10 +8,12 @@
 import glob
 import os
 import shutil
+from contextlib import contextmanager
 
 from pex.interpreter import PythonInterpreter
 
 from pants.backend.python.python_requirement import PythonRequirement
+from pants.backend.python.subsystems.python_dist_build_environment import PythonDistBuildEnvironment
 from pants.backend.python.targets.python_requirement_library import PythonRequirementLibrary
 from pants.backend.python.tasks.pex_build_util import is_local_python_dist
 from pants.backend.python.tasks.setup_py import SetupPyRunner
@@ -20,7 +22,9 @@
 from pants.base.fingerprint_strategy import DefaultFingerprintStrategy
 from pants.build_graph.address import Address
 from pants.task.task import Task
+from pants.util.contextutil import environment_as
 from pants.util.dirutil import safe_mkdir
+from pants.util.memo import memoized_property
 
 
 class BuildLocalPythonDistributions(Task):
@@ -39,53 +43,73 @@ def product_types(cls):
   def prepare(cls, options, round_manager):
     round_manager.require_data(PythonInterpreter)
 
+  @classmethod
+  def implementation_version(cls):
+    return super(BuildLocalPythonDistributions, cls).implementation_version() + [('BuildLocalPythonDistributions', 1)]
+
+  @classmethod
+  def subsystem_dependencies(cls):
+    return super(BuildLocalPythonDistributions, cls).subsystem_dependencies() + (PythonDistBuildEnvironment.scoped(cls),)
+
+  # TODO: seriously consider subclassing UnixCCompiler as well as the build_ext
+  # command from distutils to control the compiler and linker invocations
+  # transparently instead of hoping distutils does the right thing.
+  @memoized_property
+  def _python_dist_build_environment(self):
+    return PythonDistBuildEnvironment.scoped_instance(self)
+
   @property
   def cache_target_dirs(self):
     return True
 
   def execute(self):
     dist_targets = self.context.targets(is_local_python_dist)
-    build_graph = self.context.build_graph
 
     if dist_targets:
       with self.invalidated(dist_targets,
                             fingerprint_strategy=DefaultFingerprintStrategy(),
                             invalidate_dependents=True) as invalidation_check:
+        interpreter = self.context.products.get_data(PythonInterpreter)
+
         for vt in invalidation_check.invalid_vts:
           if vt.target.dependencies:
             raise TargetDefinitionException(
               vt.target, 'The `dependencies` field is disallowed on `python_dist` targets. '
                          'List any 3rd party requirements in the install_requirements argument '
                          'of your setup function.'
             )
-          self._create_dist(vt.target, vt.results_dir)
+          self._create_dist(vt.target, vt.results_dir, interpreter)
 
         for vt in invalidation_check.all_vts:
           dist = self._get_whl_from_dir(os.path.join(vt.results_dir, 'dist'))
           req_lib_addr = Address.parse('{}__req_lib'.format(vt.target.address.spec))
           self._inject_synthetic_dist_requirements(dist, req_lib_addr)
           # Make any target that depends on the dist depend on the synthetic req_lib,
           # for downstream consumption.
-          for dependent in build_graph.dependents_of(vt.target.address):
-            build_graph.inject_dependency(dependent, req_lib_addr)
-
-  def _create_dist(self, dist_tgt, dist_target_dir):
-    """Create a .whl file for the specified python_distribution target."""
-    interpreter = self.context.products.get_data(PythonInterpreter)
+          for dependent in self.context.build_graph.dependents_of(vt.target.address):
+            self.context.build_graph.inject_dependency(dependent, req_lib_addr)
 
+  def _copy_sources(self, dist_tgt, dist_target_dir):
     # Copy sources and setup.py over to vt results directory for packaging.
     # NB: The directory structure of the destination directory needs to match 1:1
     # with the directory structure that setup.py expects.
-    for src_relative_to_target_base in dist_tgt.sources_relative_to_target_base():
+    all_sources = list(dist_tgt.sources_relative_to_target_base())
+    for src_relative_to_target_base in all_sources:
       src_rel_to_results_dir = os.path.join(dist_target_dir, src_relative_to_target_base)
       safe_mkdir(os.path.dirname(src_rel_to_results_dir))
       abs_src_path = os.path.join(get_buildroot(),
                                   dist_tgt.address.spec_path,
                                   src_relative_to_target_base)
       shutil.copyfile(abs_src_path, src_rel_to_results_dir)
-    # Build a whl using SetupPyRunner and return its absolute path.
-    setup_runner = SetupPyRunner(dist_target_dir, 'bdist_wheel', interpreter=interpreter)
-    setup_runner.run()
+
+  def _create_dist(self, dist_tgt, dist_target_dir, interpreter):
+    """Create a .whl file for the specified python_distribution target."""
+    self.context.log.debug("dist_target_dir: '{}'".format(dist_target_dir))
+    self._copy_sources(dist_tgt, dist_target_dir)
+    with self._python_dist_build_environment.execution_environment():
+      # Build a whl using SetupPyRunner and return its absolute path.
+      setup_runner = SetupPyRunner(dist_target_dir, 'bdist_wheel', interpreter=interpreter)
+      setup_runner.run()
 
   def _inject_synthetic_dist_requirements(self, dist, req_lib_addr):
     """Inject a synthetic requirements library that references a local wheel.

diff --git a/src/python/pants/binaries/BUILD b/src/python/pants/binaries/BUILD
@@ -30,3 +30,11 @@ python_library(
     'src/python/pants/util:memo',
   ],
 )
+
+python_library(
+  name='execution_environment_mixin',
+  sources=['execution_environment_mixin.py'],
+  dependencies=[
+    'src/python/pants/util:contextutil',
+  ],
+)
diff --git a/src/python/pants/binaries/execution_environment_mixin.py b/src/python/pants/binaries/execution_environment_mixin.py
@@ -0,0 +1,58 @@
+# coding=utf-8
+# Copyright 2018 Pants project contributors (see CONTRIBUTORS.md).
+# Licensed under the Apache License, Version 2.0 (see LICENSE).
+
+from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
+                        unicode_literals, with_statement)
+
+import os
+from contextlib import contextmanager
+
+from abc import abstractmethod
+
+from pants.util.contextutil import environment_as
+
+
+def prepend_path(prev_env, path_var, new_entries):
+  prev_path = prev_env.get(path_var, None)
+  if prev_path is None:
+    return ':'.join(new_entries)
+  return ':'.join(new_entries + prev_path.split(':'))
+
+
+class ExecutionEnvironmentMixin(object):
+
+  @classmethod
+  def apply_successive_env_modifications(cls, env, environment_mixin_instances):
+    for env_modifier in environment_mixin_instances:
+      env = env_modifier.modify_environment(env)
+    return env
+
+  # There's no reason this has to be just about the shell "environment", really
+  # just has to be a contextmanager. Implementing this method using a literal
+  # chroot, or VM image, or something might be really interesting to just
+  # completely sidestep the installation problem.
+  @contextmanager
+  def execution_environment(self, prev_env=None):
+    if prev_env is None:
+      prev_env = os.environ
+    env_copy = prev_env.copy()
+    new_env = self.modify_environment(env_copy)
+
+    with environment_as(**new_env):
+      yield
+
+  @abstractmethod
+  def modify_environment(self, env): pass
+
+
+class ExecutionPathEnvironment(ExecutionEnvironmentMixin):
+
+  @abstractmethod
+  def get_additional_paths(self): pass
+
+  def modify_environment(self, env):
+    additional_paths = self.get_additional_paths()
+    new_path = prepend_path(env, 'PATH', additional_paths)
+    env['PATH'] = new_path
+    return env