Skip to content

Commit

Permalink
Introduce Clang, GCC, and Binutils to form a native toolchain
Browse files Browse the repository at this point in the history
This is the first instance in Pants of selectively instantiating a subsystem
based on the platform Pants is being invoked on, because gnu binutils (or at
least, ld) doesn't support OSX. The XCode cli tools will be relied on
instead for linking (more work needed here on effective error messages).

Create by squashing 119 commits, most of which had been merged into master: many
of these commits were merged with the first python_dist PR, or the PR that
introduced the "LLVM" subsystem (now named "Clang").

Squashed commit of the following:

commit 9ae7284
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Mar 21 15:03:38 2018 -0700

    add some more comments to explain what i'm thinking

commit 8d48d0a
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Mar 21 14:52:07 2018 -0700

    some mild refactoring and more documentation

commit 29e9622
Merge: ba727b7 a5410b6
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Mar 21 12:52:28 2018 -0700

    Merge branch 'master' into dmcclanahan/python-dist-c++-sources

commit ba727b7
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Tue Mar 20 02:10:40 2018 -0700

    gcc is linux specific again (but only for a little bit)

commit aaae18c
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Fri Mar 16 15:26:48 2018 -0700

    correct gcc to non-platform specific, but remove from osx for now

commit dfe7340
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Fri Mar 16 14:38:39 2018 -0700

    add platform-specific native toolchain and use to build python dists

commit f06c83f
Merge: f22dfca f9ee6fd
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Fri Mar 16 12:20:11 2018 -0700

    Merge branch 'master' into dmcclanahan/python-dist-c++-sources

commit f22dfca
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Mar 7 16:27:59 2018 -0800

    add fixme

commit 7d76e70
Merge: a35256d d77483e
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Fri Mar 2 15:47:08 2018 -0800

    Merge branch 'master' into dmcclanahan/python-dist-c++-sources

commit a35256d
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Thu Mar 1 11:51:29 2018 -0800

    update name of `compiler` package to `clang`

commit 529324e
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Thu Feb 22 12:07:52 2018 -0800

    fix lint errors

commit 7763e5d
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Thu Feb 22 11:46:58 2018 -0800

    rename LLVM -> Compiler

commit 91e1353
Merge: 6a68876 1ed1fdb
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Thu Feb 22 11:28:16 2018 -0800

    Merge branch 'dmcclanahan/python-dist-c++-sources' of github.com:cosmicexplorer/pants into dmcclanahan/python-dist-c++-sources

commit 6a68876
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Feb 21 11:45:30 2018 -0800

    refactor out unnecessary interpreter wrapper to use a contextmanager

commit 1ed1fdb
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Feb 21 11:45:30 2018 -0800

    refactor out unnecessary interpreter wrapper to use a contextmanager

commit 41a1a9a
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Tue Feb 20 14:17:23 2018 -0800

    revert unnecessary python_dist changes

commit 1cfd60a
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Tue Feb 20 14:12:44 2018 -0800

    make final simplifications

commit a4ebbbd
Merge: 7783678 71a33d6
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Tue Feb 20 13:57:19 2018 -0800

    Merge branch 'master' into dmcclanahan/python-dist-c++-sources

commit 7783678
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Tue Feb 20 13:45:38 2018 -0800

    remove more implementation artifacts

commit 88851f1
Merge: c7c90b2 c014e8d
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Tue Feb 20 10:44:14 2018 -0800

    Merge branch 'master' into dmcclanahan/python-dist-c++-sources

commit c7c90b2
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Fri Feb 16 07:20:37 2018 -0800

    refactor unnecessary complexity

commit 35899ee
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Thu Feb 15 05:47:49 2018 -0800

    try to understand what makes the breaking change

commit d9fc2c4
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Feb 14 13:55:02 2018 -0800

    remove comments in sandboxed_interpreter.py

commit ceae8bb
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Feb 14 13:51:46 2018 -0800

    slim down the pull request quite a bit

commit 48d7a93
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Feb 14 13:27:13 2018 -0800

    rewrite BinaryTool a little and remove native toolchain subsystem

commit cfe5c86
Merge: b1d39ec f55260a
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Feb 14 08:34:39 2018 -0800

    Merge branch 'master' of github.com:pantsbuild/pants into dmcclanahan/python-dist-c++-sources

commit b1d39ec
Merge: 55dbd7e 7cdea9a
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Tue Feb 13 11:54:50 2018 -0800

    Merge branch 'master' into dmcclanahan/python-dist-c++-sources

commit 55dbd7e
Author: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Mon Feb 12 19:08:59 2018 -0800

    suddenly, with very few changes, everything "just works"!

commit 608769e
Merge: badd80f 569f14c
Author: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Mon Feb 12 19:05:19 2018 -0800

    Merge branch 'master' into dmcclanahan/python-dist-c++-sources

commit badd80f
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Mon Feb 5 16:09:46 2018 -0800

    add llvm distribution support, not just clang

commit 0ac2bbe
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Fri Feb 2 16:33:37 2018 -0800

    remove source copying contextmanager and set clang arch for setup.py

commit 6c52529
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Fri Feb 2 12:16:55 2018 -0800

    try cpp module sources and comment out future work

commit f33677b
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Fri Feb 2 11:39:58 2018 -0800

    add some more context, leave everything in an inconsistent state

commit 5a6ad48
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Thu Feb 1 12:07:56 2018 -0800

    plumb in a native toolchain subsystem

commit 5ed218e
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Thu Feb 1 11:11:10 2018 -0800

    use a contextmanager to copy source files and edit PYTHONPATH

commit 3155138
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Jan 31 20:47:16 2018 -0800

    now we can declare cpp_sources in python_dist targets!

commit 2591288
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Jan 31 03:17:57 2018 -0800

    cut off work for now

commit 629ff46
Author: Daniel McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Jan 31 01:02:38 2018 -0800

    clean up PythonDistribution and add c_sources field

commit 9e3d2ca
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Jan 30 14:36:45 2018 -0800

    Remove mod to travis yml

commit 1a04e80
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Jan 30 13:13:37 2018 -0800

    Rebase with master

commit b4d367c
Author: Chris Livingston <clivingston@twitter.com>
Date:   Wed Jan 24 17:59:30 2018 -0800

    Rename superhello to fasthello

commit 59164c1
Author: Chris Livingston <clivingston@twitter.com>
Date:   Wed Jan 24 15:26:03 2018 -0800

    Add remove command to travis.yml to remove problematic file from failing CI target

commit 8ac832b
Author: Chris Livingston <clivingston@twitter.com>
Date:   Wed Jan 24 13:00:12 2018 -0800

    Add clean-all statements to integration tests to gauge flakiness

commit a787717
Author: Chris Livingston <clivingston@twitter.com>
Date:   Wed Jan 24 10:39:22 2018 -0800

    Slightly modify test assertion for conflicting deps test

commit 0b82067
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Jan 23 15:02:47 2018 -0800

    Resolve merge conflicts from removal of tasks2

commit 7f757c0
Author: Chris Livingston <clivingston@twitter.com>
Date:   Mon Jan 22 12:28:18 2018 -0800

    Disallow dependencies on a python_dist target

commit 3024a34
Author: Chris Livingston <clivingston@twitter.com>
Date:   Fri Jan 19 11:17:45 2018 -0800

    Add xfail test to testprojects tests

commit eb8598d
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Jan 18 17:26:24 2018 -0800

    Fix lint

commit 654c20e
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Jan 18 14:13:42 2018 -0800

    Remove unused import

commit 22b72f0
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Jan 18 11:39:49 2018 -0800

    Remove add_labels from PythonDistribution object

commit a4150d0
Author: Chris Livingston <clivingston@twitter.com>
Date:   Wed Jan 17 19:17:59 2018 -0800

    Remove duplicate functions to enforce DRY

commit 178bcbd
Author: Chris Livingston <clivingston@twitter.com>
Date:   Wed Jan 17 14:21:47 2018 -0800

    Fix issues with CI and failing testprojects target

commit cae065c
Author: Chris Livingston <clivingston@twitter.com>
Date:   Mon Jan 15 23:44:32 2018 -0800

    Add integration tests for targets that conflict with transitive deps listed in the install_requires field of a python distribution's setup.py

commit ffae43d
Author: Chris Livingston <clivingston@twitter.com>
Date:   Mon Jan 15 22:36:45 2018 -0800

    Remove crufty files

commit c6edc08
Author: Chris Livingston <clivingston@twitter.com>
Date:   Fri Jan 12 17:23:46 2018 -0800

    remove unneeded dependency test

commit 2925530
Author: Chris Livingston <clivingston@twitter.com>
Date:   Fri Jan 12 17:21:33 2018 -0800

    Edge case impl for same setup.py package name/version as a binary dep

commit fe8dd55
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Jan 11 22:36:45 2018 -0800

    Simplify a few lines, add check for ambiguous python dists, and fix copyright date

commit aabf88f
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Jan 11 18:35:34 2018 -0800

    Update TODO github issue link

commit a6377e4
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Jan 11 16:36:12 2018 -0800

    Remove tests that break testprojects integration testing

commit d41e7ee
Author: Chris Livingston <clivingston@twitter.com>
Date:   Wed Jan 10 16:29:25 2018 -0800

    Remove unnecessary targets from BUILD file in superhello_testproject

commit 4fd1567
Author: Chris Livingston <clivingston@twitter.com>
Date:   Wed Jan 10 14:44:23 2018 -0800

    Cleanup integration test and move superhello test project to examples/tests due to test breakage when placed in testprojects

commit e8f6b98
Author: Chris Livingston <clivingston@twitter.com>
Date:   Wed Jan 10 11:59:32 2018 -0800

    Fix multiple binary target case and add integration test

commit 9e0b2d6
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Jan 9 15:57:03 2018 -0800

    Add TODO with github link for package conflict case in python dist backend

commit a2116c4
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Jan 9 15:20:25 2018 -0800

    Remove cruft

commit 03bc4c0
Author: Chris Livingston <clivingston@twitter.com>
Date:   Mon Jan 8 16:15:21 2018 -0800

    Remove unneccessary checks for invalid targets and streamline method signatures

commit 4e9bb6b
Author: Chris Livingston <clivingston@twitter.com>
Date:   Fri Jan 5 18:24:44 2018 -0800

    Fix install directory clobbering and setup.py positioning

commit f1ed6da
Author: Chris Livingston <clivingston@twitter.com>
Date:   Fri Jan 5 18:23:02 2018 -0800

    Fix install directory clobbering and setup.py positioning

commit be2583a
Author: Lionel Vital <lvital@twitter.com>
Date:   Fri Jan 5 17:48:46 2018 -0800

    Addresses a few changes

commit 022b1e7
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Dec 21 16:50:35 2017 -0800

    Add rjiang suggestion for counting setup.py files

commit e2bf445
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Dec 21 15:16:35 2017 -0800

    Clean up comments, docstrings, and fix broken testprojects integration tests

commit 3e1739f
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Dec 21 12:42:23 2017 -0800

    Fix merge conflict

commit 365e708
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Dec 21 12:30:08 2017 -0800

    Style fixes and cruftslaying per rjiang's comments

commit 5f4e07c
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Dec 21 12:10:02 2017 -0800

    Add detection of multiple setup.py files and throw an error.

commit 056e267
Author: Chris Livingston <clivingston@twitter.com>
Date:   Wed Dec 20 17:50:17 2017 -0800

    Add integration testing and simple unit test for python create distributions task. Also cleanup code to DRY by creating util helper method and streamline invalid python dist target detection.

commit 401a10d
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Dec 19 18:12:14 2017 -0800

    Working goals using invalidated blocks

commit 53a1355
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Dec 19 16:40:56 2017 -0800

    Solid working state based off of vt.results dir caching

commit dacd672
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Dec 19 15:57:37 2017 -0800

    Working caching under vt.results dir. Moving to tests.

commit edf110d
Author: Lionel Vital <lvital@twitter.com>
Date:   Tue Dec 19 14:58:12 2017 -0800

    More cleanup + move stuff in task execution under invalidated

commit 8c7b658
Author: Lionel Vital <lvital@twitter.com>
Date:   Fri Dec 15 15:43:56 2017 -0800

    Easy nits and whitespace issues

commit 31d3a91
Author: Lionel Vital <lvital@twitter.com>
Date:   Fri Dec 15 14:14:19 2017 -0800

    Another whitespace error

commit 177c379
Author: Lionel Vital <lvital@twitter.com>
Date:   Fri Dec 15 12:32:06 2017 -0800

    Whitespace lint fixes and unused import

commit 7b8e8be
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Dec 14 17:43:54 2017 -0800

    Remove unused imports created from refactor

commit 51e9c64
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Dec 14 14:44:11 2017 -0800

    Remove unused imports and add guard from consuming python dist products

commit 59ff560
Author: Chris Livingston <clivingston@twitter.com>
Date:   Wed Dec 13 16:52:16 2017 -0800

    Add guard statement for case where pants test run does not require data from PythonCreateDistributions task.

commit fb1c903
Author: Chris Livingston <clivingston@twitter.com>
Date:   Wed Dec 13 14:27:04 2017 -0800

    Fix minor BUILD file error to pass CI

commit 23f6999
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Dec 12 16:37:39 2017 -0800

    Working distribution create task and integration for pants run/binary/test

commit 314ecfb
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Dec 12 16:05:10 2017 -0800

    pants test example

commit a859f46
Author: Chris Livingston <clivingston@twitter.com>
Date:   Mon Dec 11 18:07:08 2017 -0800

    Remove cruft

commit 6881a72
Author: Chris Livingston <clivingston@twitter.com>
Date:   Mon Dec 11 18:01:16 2017 -0800

    Revert pants run mods and fix bug in pex build util

commit 1c981f7
Author: Lionel Vital <lvital@twitter.com>
Date:   Mon Dec 11 17:36:58 2017 -0800

    Run/test

commit 8436dad
Author: Lionel Vital <lvital@twitter.com>
Date:   Mon Dec 11 17:14:20 2017 -0800

    More cleanup of docs

commit b6fa385
Author: Lionel Vital <lvital@twitter.com>
Date:   Mon Dec 11 16:48:33 2017 -0800

    Minor cleanup + rename alias to be standard

commit 4b64f2d
Author: Chris Livingston <clivingston@twitter.com>
Date:   Fri Dec 8 16:08:53 2017 -0800

    Further iteration on python distribution task, now functioning for python binary create

commit 76aeda2
Author: Chris Livingston <clivingston@twitter.com>
Date:   Fri Dec 8 15:20:34 2017 -0800

    Working distribution creation task

commit 7823d2a
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Dec 7 17:37:24 2017 -0800

    Progress on pex_build_util

commit e4c0556
Author: Chris Livingston <clivingston@twitter.com>
Date:   Thu Dec 7 10:02:26 2017 -0800

    Install Python Dist create task

commit 4b65461
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Dec 5 17:51:57 2017 -0800

    Debug session 1

commit c340abf
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Dec 5 17:22:32 2017 -0800

    Create python distribution task and refactor example

commit 9bb228f
Author: Lionel Vital <lvital@twitter.com>
Date:   Mon Dec 4 16:05:01 2017 -0800

    PythonDistribution -> inherit Target instead of PythonTarget

commit 7297096
Author: Chris Livingston <clivingston@twitter.com>
Date:   Fri Dec 1 12:41:17 2017 -0800

    First pass at kwlzn change suggestions

commit e2f84cc
Author: Chris Livingston <clivingston@twitter.com>
Date:   Wed Nov 29 10:38:58 2017 -0800

    Add python_distribution to backend/targets build file

commit 194452b
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Nov 28 16:50:14 2017 -0800

    pants run task implementation - first pass

commit 6585142
Author: Lionel Vital <lvital@twitter.com>
Date:   Tue Nov 28 16:11:15 2017 -0800

    First stab at run task

commit 0ae435e
Author: Lionel Vital <lvital@twitter.com>
Date:   Tue Nov 28 15:37:57 2017 -0800

    Remove excess __init__.py files

commit 1940c53
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Nov 28 15:22:47 2017 -0800

    Clean up misc style issues and delete unnecessary file.

commit 9cd32a4
Author: Lionel Vital <lvital@twitter.com>
Date:   Tue Nov 28 12:59:37 2017 -0800

    Clean up + add some docs

commit c8a954e
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Nov 28 12:16:26 2017 -0800

    Add cleanup of egg-info in pex build util

commit 83be228
Author: Chris Livingston <clivingston@twitter.com>
Date:   Fri Nov 24 10:32:26 2017 -0800

    Add tensorflow dependency example to superhello; this puts ./pants binary on main:main in a good state for a workable demo

commit 74bfb8c
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Nov 21 16:45:55 2017 -0800

    Finalized initial approach at packaging wheels w/ c sources into a pex

commit 4e2b1b9
Author: Lionel Vital <lvital@twitter.com>
Date:   Tue Nov 21 15:02:05 2017 -0800

    Minor cleanup

commit dd99194
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Nov 21 14:43:19 2017 -0800

    Working superhello project equivalent to backends/tensorflow

commit 5837f30
Author: Lionel Vital <lvital@twitter.com>
Date:   Tue Nov 21 14:27:00 2017 -0800

    Modify example to use superhello

commit 4f9bc98
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Nov 21 13:51:52 2017 -0800

    Working python dist example

commit cd32058
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Nov 21 12:22:23 2017 -0800

    Changes to python binary creation + add new target definition

commit 3aa0fe4
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Nov 21 12:19:55 2017 -0800

    Changes to hello2 package

commit 46eab74
Author: Chris Livingston <clivingston@twitter.com>
Date:   Tue Nov 21 12:19:22 2017 -0800

    Working package example for hello2

commit ca86bf6
Author: Chris Livingston <clivingston@twitter.com>
Date:   Wed Nov 15 13:01:29 2017 -0800

    First pass at distribution task and target

commit 3ec11c2
Author: Lionel Vital <lvital@twitter.com>
Date:   Thu Nov 9 23:27:59 2017 -0800

    Python_distribution example
  • Loading branch information
cosmicexplorer committed Mar 22, 2018
1 parent a5410b6 commit 3c3c7d2
Show file tree
Hide file tree
Showing 14 changed files with 257 additions and 15 deletions.
6 changes: 6 additions & 0 deletions build-support/docker/centos6/Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,12 @@ RUN yum install -y \
libffi \
libffi-devel \
openssl-devel
# LLVM release tarballs are xz-compressed.
RUN yum install -y xz
# GCC requires these to compile.
RUN yum install -y \
m4 \
wget

# By default, execute in an environment with python27 enabled.
ENTRYPOINT ["/usr/bin/scl", "enable", "python27", "devtoolset-7", "--"]
4 changes: 4 additions & 0 deletions src/python/pants/backend/native/subsystems/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -4,5 +4,9 @@
python_library(
dependencies=[
'src/python/pants/binaries:binary_util',
'src/python/pants/binaries:execution_environment_mixin',
'src/python/pants/subsystem',
'src/python/pants/util:memo',
'src/python/pants/util:osutil',
],
)
20 changes: 20 additions & 0 deletions src/python/pants/backend/native/subsystems/clang.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# coding=utf-8
# Copyright 2018 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
unicode_literals, with_statement)

import os

from pants.binaries.binary_tool import NativeTool
from pants.binaries.execution_environment_mixin import ExecutionPathEnvironment


class Clang(NativeTool, ExecutionPathEnvironment):
options_scope = 'clang'
default_version = '6.0.0'
archive_type = 'tgz'

def get_additional_paths(self):
return [os.path.join(self.select(), 'bin')]
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,16 @@
from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
unicode_literals, with_statement)

import os

from pants.binaries.binary_tool import NativeTool
from pants.binaries.execution_environment_mixin import ExecutionPathEnvironment


class LLVM(NativeTool):
options_scope = 'llvm'
default_version = '5.0.1'
class GCC(NativeTool, ExecutionPathEnvironment):
options_scope = 'gcc'
default_version = '7.3.0'
archive_type = 'tgz'

def get_additional_paths(self):
return [os.path.join(self.select(), 'bin')]
47 changes: 47 additions & 0 deletions src/python/pants/backend/native/subsystems/native_toolchain.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
# coding=utf-8
# Copyright 2018 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
unicode_literals, with_statement)

from contextlib import contextmanager

from pants.backend.native.subsystems.clang import Clang
from pants.backend.native.subsystems.gcc import GCC
from pants.backend.native.subsystems.platform_specific.linux.binutils import Binutils
from pants.binaries.execution_environment_mixin import ExecutionEnvironmentMixin
from pants.subsystem.subsystem import Subsystem
from pants.util.memo import memoized_property
from pants.util.osutil import get_os_name, normalize_os_name


class NativeToolchain(Subsystem, ExecutionEnvironmentMixin):

options_scope = 'native-toolchain'

PLATFORM_SPECIFIC_TOOLCHAINS = {
# TODO(cosmicexplorer): 'darwin' should have everything here, but there's no
# open-source linker for OSX...yet.
'darwin': [GCC, Clang],
'linux': [GCC, Binutils, Clang],
}

@classmethod
def _get_platform_specific_toolchains(cls):
normed_os_name = normalize_os_name(get_os_name())
return cls.PLATFORM_SPECIFIC_TOOLCHAINS[normed_os_name]

@classmethod
def subsystem_dependencies(cls):
prev = super(NativeToolchain, cls).subsystem_dependencies()
cur_platform_subsystems = cls._get_platform_specific_toolchains()
return prev + tuple(sub.scoped(cls) for sub in cur_platform_subsystems)

@memoized_property
def _toolchain_instances(self):
cur_platform_subsystems = self._get_platform_specific_toolchains()
return [sub.scoped_instance(self) for sub in cur_platform_subsystems]

def modify_environment(self, env):
return self.apply_successive_env_modifications(env, self._toolchain_instances)
Empty file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Copyright 2018 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

python_library(
dependencies=[
'src/python/pants/binaries:binary_util',
'src/python/pants/binaries:execution_environment_mixin',
],
)
Empty file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# coding=utf-8
# Copyright 2018 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
unicode_literals, with_statement)

import os

from pants.binaries.binary_tool import NativeTool
from pants.binaries.execution_environment_mixin import ExecutionPathEnvironment


class Binutils(NativeTool, ExecutionPathEnvironment):
options_scope = 'binutils'
default_version = '2.30'
archive_type = 'tgz'

def get_additional_paths(self):
return [os.path.join(self.select(), 'bin')]
3 changes: 3 additions & 0 deletions src/python/pants/backend/python/subsystems/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,10 @@
python_library(
dependencies = [
'3rdparty/python:setuptools',
'src/python/pants/binaries:execution_environment_mixin',
'src/python/pants/backend/native',
'src/python/pants/option',
'src/python/pants/subsystem',
'src/python/pants/util:memo',
],
)
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# coding=utf-8
# Copyright 2018 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
unicode_literals, with_statement)

from pants.backend.native.subsystems.native_toolchain import NativeToolchain
from pants.binaries.execution_environment_mixin import ExecutionEnvironmentMixin
from pants.subsystem.subsystem import Subsystem
from pants.util.memo import memoized_property


class PythonDistBuildEnvironment(Subsystem, ExecutionEnvironmentMixin):

options_scope = 'python-dist-build-environment'

@classmethod
def subsystem_dependencies(cls):
return super(PythonDistBuildEnvironment, cls).subsystem_dependencies() + (NativeToolchain.scoped(cls),)

@memoized_property
def _native_toolchain(self):
return NativeToolchain.scoped_instance(self)

def modify_environment(self, env):
env = self._native_toolchain.modify_environment(env)
# If we're going to be wrapping setup.py-based projects, we really should be
# doing it through a subclass of a distutils UnixCCompiler (in Lib/distutils
# in the CPython source) instead of hoping setup.py knows what to do. The
# default UnixCCompiler from distutils will build a 32/64-bit "fat binary"
# unless you use their undocumented ARCHFLAGS env var, and there may be more
# dragons later on.
env['CC'] = 'gcc'
env['CXX'] = 'g++'
env['ARCHFLAGS'] = '-arch x86_64'
return env
Original file line number Diff line number Diff line change
Expand Up @@ -8,10 +8,12 @@
import glob
import os
import shutil
from contextlib import contextmanager

from pex.interpreter import PythonInterpreter

from pants.backend.python.python_requirement import PythonRequirement
from pants.backend.python.subsystems.python_dist_build_environment import PythonDistBuildEnvironment
from pants.backend.python.targets.python_requirement_library import PythonRequirementLibrary
from pants.backend.python.tasks.pex_build_util import is_local_python_dist
from pants.backend.python.tasks.setup_py import SetupPyRunner
Expand All @@ -20,7 +22,9 @@
from pants.base.fingerprint_strategy import DefaultFingerprintStrategy
from pants.build_graph.address import Address
from pants.task.task import Task
from pants.util.contextutil import environment_as
from pants.util.dirutil import safe_mkdir
from pants.util.memo import memoized_property


class BuildLocalPythonDistributions(Task):
Expand All @@ -39,53 +43,73 @@ def product_types(cls):
def prepare(cls, options, round_manager):
round_manager.require_data(PythonInterpreter)

@classmethod
def implementation_version(cls):
return super(BuildLocalPythonDistributions, cls).implementation_version() + [('BuildLocalPythonDistributions', 1)]

@classmethod
def subsystem_dependencies(cls):
return super(BuildLocalPythonDistributions, cls).subsystem_dependencies() + (PythonDistBuildEnvironment.scoped(cls),)

# TODO: seriously consider subclassing UnixCCompiler as well as the build_ext
# command from distutils to control the compiler and linker invocations
# transparently instead of hoping distutils does the right thing.
@memoized_property
def _python_dist_build_environment(self):
return PythonDistBuildEnvironment.scoped_instance(self)

@property
def cache_target_dirs(self):
return True

def execute(self):
dist_targets = self.context.targets(is_local_python_dist)
build_graph = self.context.build_graph

if dist_targets:
with self.invalidated(dist_targets,
fingerprint_strategy=DefaultFingerprintStrategy(),
invalidate_dependents=True) as invalidation_check:
interpreter = self.context.products.get_data(PythonInterpreter)

for vt in invalidation_check.invalid_vts:
if vt.target.dependencies:
raise TargetDefinitionException(
vt.target, 'The `dependencies` field is disallowed on `python_dist` targets. '
'List any 3rd party requirements in the install_requirements argument '
'of your setup function.'
)
self._create_dist(vt.target, vt.results_dir)
self._create_dist(vt.target, vt.results_dir, interpreter)

for vt in invalidation_check.all_vts:
dist = self._get_whl_from_dir(os.path.join(vt.results_dir, 'dist'))
req_lib_addr = Address.parse('{}__req_lib'.format(vt.target.address.spec))
self._inject_synthetic_dist_requirements(dist, req_lib_addr)
# Make any target that depends on the dist depend on the synthetic req_lib,
# for downstream consumption.
for dependent in build_graph.dependents_of(vt.target.address):
build_graph.inject_dependency(dependent, req_lib_addr)

def _create_dist(self, dist_tgt, dist_target_dir):
"""Create a .whl file for the specified python_distribution target."""
interpreter = self.context.products.get_data(PythonInterpreter)
for dependent in self.context.build_graph.dependents_of(vt.target.address):
self.context.build_graph.inject_dependency(dependent, req_lib_addr)

def _copy_sources(self, dist_tgt, dist_target_dir):
# Copy sources and setup.py over to vt results directory for packaging.
# NB: The directory structure of the destination directory needs to match 1:1
# with the directory structure that setup.py expects.
for src_relative_to_target_base in dist_tgt.sources_relative_to_target_base():
all_sources = list(dist_tgt.sources_relative_to_target_base())
for src_relative_to_target_base in all_sources:
src_rel_to_results_dir = os.path.join(dist_target_dir, src_relative_to_target_base)
safe_mkdir(os.path.dirname(src_rel_to_results_dir))
abs_src_path = os.path.join(get_buildroot(),
dist_tgt.address.spec_path,
src_relative_to_target_base)
shutil.copyfile(abs_src_path, src_rel_to_results_dir)
# Build a whl using SetupPyRunner and return its absolute path.
setup_runner = SetupPyRunner(dist_target_dir, 'bdist_wheel', interpreter=interpreter)
setup_runner.run()

def _create_dist(self, dist_tgt, dist_target_dir, interpreter):
"""Create a .whl file for the specified python_distribution target."""
self.context.log.debug("dist_target_dir: '{}'".format(dist_target_dir))
self._copy_sources(dist_tgt, dist_target_dir)
with self._python_dist_build_environment.execution_environment():
# Build a whl using SetupPyRunner and return its absolute path.
setup_runner = SetupPyRunner(dist_target_dir, 'bdist_wheel', interpreter=interpreter)
setup_runner.run()

def _inject_synthetic_dist_requirements(self, dist, req_lib_addr):
"""Inject a synthetic requirements library that references a local wheel.
Expand Down
8 changes: 8 additions & 0 deletions src/python/pants/binaries/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -30,3 +30,11 @@ python_library(
'src/python/pants/util:memo',
],
)

python_library(
name='execution_environment_mixin',
sources=['execution_environment_mixin.py'],
dependencies=[
'src/python/pants/util:contextutil',
],
)
58 changes: 58 additions & 0 deletions src/python/pants/binaries/execution_environment_mixin.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,58 @@
# coding=utf-8
# Copyright 2018 Pants project contributors (see CONTRIBUTORS.md).
# Licensed under the Apache License, Version 2.0 (see LICENSE).

from __future__ import (absolute_import, division, generators, nested_scopes, print_function,
unicode_literals, with_statement)

import os
from contextlib import contextmanager

from abc import abstractmethod

from pants.util.contextutil import environment_as


def prepend_path(prev_env, path_var, new_entries):
prev_path = prev_env.get(path_var, None)
if prev_path is None:
return ':'.join(new_entries)
return ':'.join(new_entries + prev_path.split(':'))


class ExecutionEnvironmentMixin(object):

@classmethod
def apply_successive_env_modifications(cls, env, environment_mixin_instances):
for env_modifier in environment_mixin_instances:
env = env_modifier.modify_environment(env)
return env

# There's no reason this has to be just about the shell "environment", really
# just has to be a contextmanager. Implementing this method using a literal
# chroot, or VM image, or something might be really interesting to just
# completely sidestep the installation problem.
@contextmanager
def execution_environment(self, prev_env=None):
if prev_env is None:
prev_env = os.environ
env_copy = prev_env.copy()
new_env = self.modify_environment(env_copy)

with environment_as(**new_env):
yield

@abstractmethod
def modify_environment(self, env): pass


class ExecutionPathEnvironment(ExecutionEnvironmentMixin):

@abstractmethod
def get_additional_paths(self): pass

def modify_environment(self, env):
additional_paths = self.get_additional_paths()
new_path = prepend_path(env, 'PATH', additional_paths)
env['PATH'] = new_path
return env

0 comments on commit 3c3c7d2

Please sign in to comment.