Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for releasing Python 3 wheels #7197

Merged
merged 31 commits into from
Mar 14, 2019

Conversation

Eric-Arellano
Copy link
Contributor

@Eric-Arellano Eric-Arellano commented Feb 1, 2019

Problem

The final step in #6062 is to add support for releasing Pants and its contrib packages as compatible with Python 2.7 and 3.6 and 3.7. Specifically, universal wheels should be marked as py27.py36.py37 and pantsbuild.pants should be marked as cp27-cp27{m,mu} or as cp36-abi3 because it uses native code (see comment in packages.py for an explanation on abi3).

packages.py and release.sh had several issues preventing building Python 3 wheels. Further, we needed to add shards to build Python 3 wheels in CI.

This resolves the wheels portion of #6450.

Solution

  • Add a -3 CLI flag to release.sh to indicate it should build wheels compatible with Py3. Also add --py3 to packages.py.
  • Update the default Package config to setup support for py27.py36.py37, meaning any Python implementation that is one of those 3 versions.
  • Update the pantsbuild.pants flags to indicate either cp27 or cp36-abi3. The former is only compatible with CPython2.7, and the latter is compatible with any CPython implementation >= 3.6 due to the ABI constraint (see https://docs.python.org/3/c-api/stable.html).
  • Add Python 3 wheels shards to CI.
  • Pull down either a pex27 or pex36 release, rather than alway using pex27.
  • Modernize Python code, such as fixing bad subprocess32 import.
  • Remove the compatibility tags from PantsRequirement. John explained it was wrong to put here, I think because it should be marked in packages.py when we actually build the wheels? This results in also tweaking the corresponding unit tests.

Result

./build-support/bin/release.sh -3n builds Python 3 compatible wheels ready for release.

Note we still cannot release a Python 3 compatible PEX, which is blocked by pex-tool/pex#654.

Although our solution is going to end up releasing a separate wheel for Py2 vs Py3 for pantsbuild.pants, this is a good starting point for the changes we need to make.
It's not needed according to John.
Allow us to mark pantsbuild.pants as Py3 wheel. Default to Py2 wheel. Note this one cannot be a universal wheel.
The backport is only offered for Py2, so will fail when imported with Py3. Adopt the idiom we use in process_handler.py.
Note that release.sh -3n is still not working, but I'm checking this in as the changes all appear correct and act as a good milestone.
Because pantsbuild.pants is not universal, we must have a build wheels shard for Py2 vs Py3 (in addition to OSX vs Linux).
This allows us to ensure that any Python version >= 3.6 works. It also allows us to remove the code in pants_loader.py doing a runtime check.

Note this does not seem to be working yet...even though it all looks like how you do it.
I missed updating one reference when rebasing against py37-venv PR.
Copy link
Member

@stuhood stuhood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Looks good from my perspective. Would like to see a little less churn in release.sh before landing though.

build-support/bin/release.sh Outdated Show resolved Hide resolved
build-support/bin/release.sh Outdated Show resolved Hide resolved
src/python/pants/backend/python/pants_requirement.py Outdated Show resolved Hide resolved
src/python/pants/releases/packages.py Outdated Show resolved Hide resolved
It's because of subparsers. Make this explicit, rather than implicit.
We need to somehow mark in our generated setup.py that we have a native extension with a PY_LIMITED_API. For now, we use the deprecated `ext_modules` while we figure out how to more robustly do this via Extension or support this first class in `PythonArtifact`.
We don't consistently call `./pants3`, so it won't do this automatically for us. Without this, we run subprocesses like `bdist_wheel` with Py2, which is not what we want when using `-3`.
commit efaae09
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 26 23:41:55 2019 -0700

    Add debugging to release.sh for linux ucs2

    It looks like the bootstrap part now works completely as intended! It's consistently using UCS2.

    But the release script is failing for some reason. Turn on debugging to wake up to hopefully some insight tomorrow morning..

commit 4cb6cae
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 26 18:27:22 2019 -0700

    Squashed commit of the following:

    commit 9c754dc
    Merge: 3f30d39 7819724
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Tue Feb 26 17:17:34 2019 -0700

        Merge branch 'master' of github.com:pantsbuild/pants into pex-interpreter-constraints

    commit 3f30d39
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Tue Feb 26 17:12:01 2019 -0700

        Fix issue with compatibility_or_constraints() returning a tuple

        add_interpreter_constraints() expects a str, so we must unpack the tuple when calling it.

    commit ff17f73
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Mon Feb 25 22:46:16 2019 -0700

        Revert "Constrain ci.sh to the exact Python interpreter version"

        This reverts commit 887a8ef.

        This change is necessary to fix the original motivation for this PR, but it does not really belong in this PR anymore. Instead, it should be in the Py2 ABI PR (7235). This PR should be kept more generic, and there is no logical connection to the changes being made with ci.sh, beyond that original motivating problem.

    commit 6b07abd
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Mon Feb 25 21:49:52 2019 -0700

        Remove bad import

        My bad for not catching this before pushing.

    commit 2c6fdb0
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Mon Feb 25 21:37:41 2019 -0700

        Generify solution by using compatibility_or_constraints()

        Instead of applying a bandaid for only `./pants binary`, John proposed fixing the issue with our PexBuilderWrapper itself. So, we use `compatibility_or_constrains()`, which will first try to return the target's compatibility, else will return the Python Setup subystem's value.

        The wrapper still is not ideal and John proposes killing add_interpreter_constraint() and add_interpreter_constraints_from() to instead automatically be setting the interpreter constraints from the targets graph. This PR does not make that change for the scope, but this should be noted.

    commit b71f164
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Mon Feb 25 21:03:46 2019 -0700

        Fix typo

    commit 3bca020
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Mon Feb 25 20:31:14 2019 -0700

        Add global interpreter constraints to Python binary creation

    commit 887a8ef
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Fri Feb 22 21:05:50 2019 -0700

        Constrain ci.sh to the exact Python interpreter version

        Earlier we allowed the patch version to float. We discovered in pantsbuild#7235 with the CI run https://travis-ci.org/pantsbuild/pants/jobs/497208431#L891 that PEX was building with 2.7.10 but running with 2.7.13.

        The fix will require having Pants pass interpreter constraints to Pex. Even with that change though, the CI shard would still have the issue because the constraint would be floating.

        Now, we have the constraint be exact.

commit 0d25bcc
Merge: 373ffee 7819724
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 26 17:41:10 2019 -0700

    Merge branch 'master' of github.com:pantsbuild/pants into py2-wheels-abi-specified

commit 373ffee
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 26 17:39:25 2019 -0700

    Configure PEX_PYTHON on linux UCS2

    It looks like ./pants binary now completely builds the PEX with 2.7.15 / UCS2! But then when trying to run `./pants.pex -V`, it resolves the runtime interpreter to 2.7.13 :/

    I'm not sure how the runtime interpreter selection is supposed to work, but there is an env var PEX_PYTHON that allows passing a path to the value you always want to use. So, we use this for now.

commit 7819724
Author: Alex Schmitt <codealchemy@users.noreply.github.com>
Date:   Tue Feb 26 13:42:36 2019 -0800

    Allow tasks to opt-in to target filtering (pantsbuild#7283)

    Followup to pantsbuild#7275 following [discussion](pantsbuild#7275 (comment)) that the target filter was being applied to tasks that do not support it (e.g. tasks that don't access targets via `get_targets()`)

    This adds a class property to `Task` that allows subclasses to effectively opt-in to the new behavior - and sets that flag to `True` for `fmt` and `lint` tasks.

commit 3bf2d28
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 26 12:11:17 2019 -0700

    Add Pyenv back to Travis path

    Even though we directly pass $PY, later processes expect the 2.7.15 interpreter to be discoverable so Pyenv must be on the path.

commit 2166efe
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 26 09:52:10 2019 -0700

    Set $PY to disambiguate which Py2.7 version to use

    Interpreter constraints don't work, as previously noted.

commit 8dd215d
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 26 09:43:42 2019 -0700

    Allow user to set $PY in ci.sh

    If not set, will resolve to the Python version being used. We should allow the user to set it though in cases like this PR, where we may have to set $PY to a very specific interpreter path.

commit 49fe576
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 26 09:41:35 2019 -0700

    Stop hardcoding interpreter constraints

    Let ci.sh determine them based on the $PY value. While working on the Linux UCS2 shard, it became clear that what really matters is which interpreter $PY (i.e. `python2`) resolves to. Setting the interpreter constraints will not impact what this resolves to nor how we bootstrap Pants. So, we should focus on setting $PY and let the interpreter constraints be resolved accordingly.

commit c3dd843
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 26 08:49:44 2019 -0700

    Improve wording.

commit f810849
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 26 08:43:42 2019 -0700

    Move interpreter constraints to Docker env entry

    Docker does not pull in external env vars. Instead, we must specify this in the Dockerfile.

    This change has added benefit that it moves all of the Py2 logic into the Dockerfile out of .travis.yml, and leaves .travis.yml solely to call the Dockerfile.

commit b9efbf0
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 26 01:03:21 2019 -0700

    Also set PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS for OSX

    Even though it was resolving correctly already, explicit is better than implicit.

commit 8402c1f
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 26 01:01:03 2019 -0700

    Ensure Linux UCS2 always uses Py2.7.15 (UCS2)

    It was not enough to install 2.7.15 and use Pyenv global. The 2.7.13 (UCS2) interpreter was still being recognized.

commit 225f153
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 26 00:53:56 2019 -0700

    Remove bad merge lines

commit 1be9e90
Author: Eric Arellano <ericarellano@me.com>
Date:   Mon Feb 25 22:41:04 2019 -0700

    Squashed commit of the following:

    commit 6b07abd
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Mon Feb 25 21:49:52 2019 -0700

        Remove bad import

        My bad for not catching this before pushing.

    commit 2c6fdb0
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Mon Feb 25 21:37:41 2019 -0700

        Generify solution by using compatibility_or_constraints()

        Instead of applying a bandaid for only `./pants binary`, John proposed fixing the issue with our PexBuilderWrapper itself. So, we use `compatibility_or_constrains()`, which will first try to return the target's compatibility, else will return the Python Setup subystem's value.

        The wrapper still is not ideal and John proposes killing add_interpreter_constraint() and add_interpreter_constraints_from() to instead automatically be setting the interpreter constraints from the targets graph. This PR does not make that change for the scope, but this should be noted.

    commit b71f164
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Mon Feb 25 21:03:46 2019 -0700

        Fix typo

    commit 3bca020
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Mon Feb 25 20:31:14 2019 -0700

        Add global interpreter constraints to Python binary creation

    commit 887a8ef
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Fri Feb 22 21:05:50 2019 -0700

        Constrain ci.sh to the exact Python interpreter version

        Earlier we allowed the patch version to float. We discovered in pantsbuild#7235 with the CI run https://travis-ci.org/pantsbuild/pants/jobs/497208431#L891 that PEX was building with 2.7.10 but running with 2.7.13.

        The fix will require having Pants pass interpreter constraints to Pex. Even with that change though, the CI shard would still have the issue because the constraint would be floating.

        Now, we have the constraint be exact.

commit f530843
Author: Eric Arellano <ericarellano@me.com>
Date:   Mon Feb 25 22:39:29 2019 -0700

    Move unit tests above wheel building shards

    Now that we have 4 wheel building shards (soon 6)—and 2 of them require bootstrapping Pants—we move unit tests above to get more immediate feedback on if the PR is good or not. We still keep them high up relative to others because several major workflows require wheel building output.

commit 223541e
Author: Eric Arellano <ericarellano@me.com>
Date:   Mon Feb 25 22:35:02 2019 -0700

    Fix Linux UCS2 using 2.7.13 with UCS4 instead of UCS2 sometimes

    There were two versions of 2.7.13 installed on the system, so Pants would sometimes choose an unintended version and would be inconsistent.

commit 8428376
Author: Eric Arellano <ericarellano@me.com>
Date:   Mon Feb 25 21:03:22 2019 -0700

    Fix typo from squashed pex-constraints

commit 48ef4dd
Author: Eric Arellano <ericarellano@me.com>
Date:   Mon Feb 25 20:44:37 2019 -0700

    Squashed commit of the following:

    commit 3bca020
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Mon Feb 25 20:31:14 2019 -0700

        Add global interpreter constraints to Python binary creation

    commit 887a8ef
    Author: Eric Arellano <ericarellano@me.com>
    Date:   Fri Feb 22 21:05:50 2019 -0700

        Constrain ci.sh to the exact Python interpreter version

        Earlier we allowed the patch version to float. We discovered in pantsbuild#7235 with the CI run https://travis-ci.org/pantsbuild/pants/jobs/497208431#L891 that PEX was building with 2.7.10 but running with 2.7.13.

        The fix will require having Pants pass interpreter constraints to Pex. Even with that change though, the CI shard would still have the issue because the constraint would be floating.

        Now, we have the constraint be exact.

commit 78a1aa9
Merge: 04c4ee0 26b0179
Author: Eric Arellano <ericarellano@me.com>
Date:   Mon Feb 25 20:44:15 2019 -0700

    Merge branch 'master' of github.com:pantsbuild/pants into py2-wheels-abi-specified

commit 26b0179
Author: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Mon Feb 25 16:23:46 2019 -0800

    try defining algebraic Executables in the native backend to compose more readable toolchains (pantsbuild#6855)

    ### Problem

    As can be seen in `native_toolchain.py` in e.g. pantsbuild#6800, it is often difficult to follow changes to the native backend, especially changes which modify the order of resources such as library and include directories for our linkers and compilers. This is because we have been patching together collections of these resources "by hand", without applying any higher-level structure (explicitly constructing each `path_entries` and `library_dirs` field for every executable, every time, for example). This was done to avoid creating abstractions that might break down due to the rapidly evolving code. We can now take the step of more clearly defining the relationships between the toolchains we construct hierarchically.

    ### Solution

    - Add an `ExtensibleAlgebraic` mixin which allows declaring list fields which can be immutably modified one at a time with `prepend_field` and `append_field`, or all at once with `sequence`.
    - Add a `for_compiler` method to `BaseLinker` to wrap the specific process required to prep our linker for a specific compiler.
    - Apply all of the above in `native_toolchain.py`.

    ### Result

    The compilers and linkers provided by `@rule`s in `native_toolchain.py` are composed with consistent verbs from `ExtensibleAlgebraic`, leading to increased readability.

commit cd4c773
Author: Nora Howard <nh@baroquebobcat.com>
Date:   Mon Feb 25 17:22:08 2019 -0700

    [zinc-compile] fully adopt enum based switches for hermetic/not; test coverage (pantsbuild#7268)

    @cosmicexplorer wrote this as part of pantsbuild#7227. This patch is pulling out just the Zinc changes, with a few differences. I also added a new test for hermetic failures and some additional assertions to ensure that the right message is being communicated on failures, while doing that I discovered that hermetic/non-hermetic appear to produce error messages on different streams.

commit c095f3b
Author: Alex Schmitt <codealchemy@users.noreply.github.com>
Date:   Mon Feb 25 10:49:57 2019 -0800

    Update TargetFiltering args for applying criteria (pantsbuild#7280)

    Update the class to take the criteria in the constructor, and helper methods take the targets against which to apply said criteria.

    From suggestion https://github.com/pantsbuild/pants/pull/7275\#discussion_r259554586

commit a87a01b
Author: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Fri Feb 22 18:53:52 2019 -0800

    don't do a pants run on osx (pantsbuild#7278)

    ### Problem

    Fixes pantsbuild#7247, catching a case that was otherwise missed.

    ### Solution

    - Don't do a `./pants run` on osx using the gnu toolchain in testing, as it doesn't work yet.

    ### Result

    As noted in pantsbuild#7249, it's strange that that PR passes but the nightly job fails -- it may be nondeterministic.

commit a86639e
Author: Alex Schmitt <codealchemy@users.noreply.github.com>
Date:   Fri Feb 22 16:52:32 2019 -0800

    Add filtering subsystem to permit skipping targets by tags (pantsbuild#7275)

    This subsystem is responsible for handling options meant to exclude targets from specific tasks

    The application of the logic itself is contained in the TargetFiltering class - which currently only handles excluding targets with provided tags and can be expanded upon for additional filtering options.

commit b34d66f
Author: John Sirois <john.sirois@gmail.com>
Date:   Fri Feb 22 16:46:32 2019 -0800

    Prepare the 1.15.0.dev1 release. (pantsbuild#7277)

commit 8069653
Author: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Fri Feb 22 11:41:58 2019 -0800

    cache python tools in ~/.cache/pants (pantsbuild#7236)

    ### Problem

    This runs for (on my laptop) about 16 seconds every time I do a `clean-all`:
    ```
    22:27:23 00:02   [native-compile]
    22:27:23 00:02     [conan-prep]
    22:27:23 00:02       [create-conan-pex]
    22:27:39 00:18     [conan-fetch]
    ```

    It doesn't seem like we need to be putting this tool in the task workdir as the python requirements list is pretty static. Conan in particular will be instantiated by invoking almost every goal, and it is a nontrivial piece of software to resolve each time.

    Also, we aren't mixing in interpreter identity to the generated pex filename, which is a bug that has so far gone undetected: see pantsbuild#7236 (comment).

    ### Solution

    - Take the `stable_json_sha1()` of the requirements of each python tool generated by `PythonToolPrepBase` to generate a fingerprinted pex filename.
    - Stick it in the pants cachedir so it doesn't get blown away by a clean-all.
    - Add an `--interpreter-constraints` option to pex tools (where previously the repo's `--python-setup-interpreter-constraints` were implicitly used).
    - Ensure the selected interpreter identity is mixed into the fingerprinted filename.
    - Add a test for the pex filename fingerprinting and that the pex can be successfully executed for python 2 and 3 constraints.

    ### Result

    A significant amount of time spent waiting after clean builds is removed, and pex tools can have their own interpreter constraints as necessary.

commit 04c4ee0
Author: Eric Arellano <ericarellano@me.com>
Date:   Fri Feb 22 12:36:56 2019 -0700

    Move debugging to proper location

    It's failing before the release.sh script is even called. The bootstrap command is what's failing.

commit e502f58
Author: Eric Arellano <ericarellano@me.com>
Date:   Fri Feb 22 12:24:33 2019 -0700

    Fix linux ucs4 stage being overriden to cron instead of test

commit 2cd72e4
Author: Eric Arellano <ericarellano@me.com>
Date:   Fri Feb 22 09:47:16 2019 -0700

    Add back logging to debug osx ucs4

commit c9e1650
Merge: 7da092b 4097052
Author: Eric Arellano <ericarellano@me.com>
Date:   Fri Feb 22 09:40:38 2019 -0700

    Merge branch 'master' of github.com:pantsbuild/pants into py2-wheels-abi-specified

commit 4097052
Author: Stu Hood <stuhood@twitter.com>
Date:   Thu Feb 21 13:49:16 2019 -0800

    Prepare 1.14.0rc3 (pantsbuild#7274)

commit ea33c36
Author: Nora Howard <nh@baroquebobcat.com>
Date:   Wed Feb 20 12:43:32 2019 -0700

    [jvm-compile] fix typo: s/direcotry/directory/ (pantsbuild#7265)

    Fix a typo in `jvm_compile.py`

commit 761849e
Author: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Feb 20 11:38:39 2019 -0800

    Fix nightly cron ctypes enum failure (pantsbuild#7249)

    ### Problem

    Resolves pantsbuild#7247. `ToolchainVariant('gnu')` does not in fact `== 'gnu'`.

    ### Solution

    - Use `.resolve_for_enum_variant()` instead of comparing with equality in that one failing test (I missed this in pantsbuild#7226, I fixed the instance earlier in the file though).
    - Raise an error when trying to use `==` on an enum to avoid this from happening again.
    - Note that in Python 3 it appears that `__hash__` must be explicitly implemented whenever `__eq__` is overridden, and this appears undocumented.

    ### Result

    The nightly cron job should be fixed, and enums are now a little more difficult to screw up.

    # Open Questions
    It's a little unclear why this didn't fail in CI -- either the test was cached, or some but not all travis osx images are provisioned with the correct dylib, causing a nondeterministic error, or something else?

commit 0e6a144
Author: Daniel Wagner-Hall <dawagner@gmail.com>
Date:   Wed Feb 20 04:14:23 2019 +0000

    Node is Display (pantsbuild#7264)

    Use standard traits, rather than our own methods which happen to do the same thing.

commit 904e3f3
Author: Ekaterina Tyurina <tyurina.katty@yandex.ru>
Date:   Wed Feb 20 01:03:42 2019 +0000

    Allow passing floating point numbers from rust to python (pantsbuild#7259)

    PR allows passing float points from Rust to Python.
    ```
    externs::store_f64(v: f64)
    ```

commit 7da092b
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 19 17:03:59 2019 -0700

    Fix platform default shards bootstrapping

    - OSX UCS2 shard no longer was setting RUN_PANTS_FROM_PEX anywhere
    - Linux UCS4 had its before_script entry being override by travis_image.

commit 38e1cf7
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 19 15:35:15 2019 -0700

    Remove unncessary RUN_PANTS_FROM_PEX=0

    Now that we don't it in the base_build_wheels_env, we don't need to set this.

    This was actually causing a failure. ./pants only checks if the env var is set, and not what its value is.

commit 621137e
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 19 15:16:04 2019 -0700

    Fix improper call to {osx,linux}_config_env

    They don't exist apparently.

commit 0da5f91
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 19 15:13:11 2019 -0700

    Stop pulling down PEX

    Use the {osx,linux}_config images rather than {osx,linux}_test_config images.

commit 513cd50
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 19 14:35:19 2019 -0700

    release.sh still needs to run from PEX

    Change how we handle env var to not use PEX when first bootstrapping, then use it in the followup release.sh command.

commit dc36d94
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 19 14:29:19 2019 -0700

    Also deduplicate Pyenv env vars for OSX

    Realized this is a better design while working on pantsbuild#7261.

commit ad45b2d
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 19 14:25:28 2019 -0700

    Revert "Turn on PEX_VERBOSE for OSX ucs4 shard"

    This reverts commit 28b9e8b.

commit fb9ef9b
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 19 14:24:21 2019 -0700

    Revert "Run PEX with -v*9"

    This reverts commit edf81ef.

commit ab534e2
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 19 14:23:28 2019 -0700

    Bootstrap Pants when using new Python install

    We can't use the PEX from AWS because the Python versions do not match up.

commit 0b59e46
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 19 12:36:08 2019 -0700

    Fix gcc no input file issues by passing dummy file

    For Linux UCS2, the build was failing due to gcc complaining it could not find any files. This reproduced locally when running `./pants3 setup-py --run="bdist_wheel --py-limited-api cp36" src/python/pants:pants-packaged` on OSX.

    John suggested and gave the code snippet to pass a dummy file so this no longer happens. Thanks John!

commit edf81ef
Author: Eric Arellano <ericarellano@me.com>
Date:   Tue Feb 19 09:33:12 2019 -0700

    Run PEX with -v*9

    PEX_VERBOSE only impacts runtime output. -vvv... impacts build time output.

commit 7c17c0a
Merge: 6ecb550 222bc11
Author: Eric Arellano <ericarellano@me.com>
Date:   Mon Feb 18 17:47:13 2019 -0700

    Merge branch 'master' of github.com:pantsbuild/pants into py2-wheels-abi-specified

commit 222bc11
Author: Daniel Wagner-Hall <dawagner@gmail.com>
Date:   Mon Feb 18 21:12:30 2019 +0000

    Revert remote execution from tower to grpcio (pantsbuild#7256)

    We're seeing weird broken connection errors with tower.

    We'll probably just chuck some retries in and be happy, but for now, let's get back to a more stable time.

    * Revert "Remove unused operation wrapper (pantsbuild#7194)"

    This reverts commit 9400024.

    * Revert "Switch operation getting to tower (pantsbuild#7108)"

    This reverts commit 0375b30.

    * Revert "Remote execution uses tower-grpc to start executions (pantsbuild#7049)"

    This reverts commit 28683c7.

commit 3da2165
Author: Ekaterina Tyurina <tyurina.katty@yandex.ru>
Date:   Mon Feb 18 17:16:48 2019 +0000

    Scheduler returns metrics as a dictionary instead of a tuple of tuples (pantsbuild#7255)

    ### Problem
    Scheduler returns metrics as a tuple of tuples (key, value). And later this tuple is transformed into a dictionary.
    It is considered to use metrics to return to python part zipkin span info and dict type will be more convenient.

    ### Solution
    Scheduler_metrics returns a store_dict  instead of store_tuple.

commit 6ecb550
Author: Eric Arellano <ericarellano@me.com>
Date:   Sat Feb 16 19:57:25 2019 -0800

    Fix Dockerfile copyright year and add comment

commit 8d69dc2
Author: Eric Arellano <ericarellano@me.com>
Date:   Sat Feb 16 19:18:45 2019 -0800

    Improve naming of Build Wheels shards

    Make it more explicit how shard is configured / which wheel building config it has. Whereas for most shards we specify if they run with Py36 vs Py27 in parantheses, it is actually very important we make explicit the wheel building config, as it impacts which wheels we end up producing.

commit 170e9c8
Author: Eric Arellano <ericarellano@me.com>
Date:   Sat Feb 16 19:10:24 2019 -0800

    Install OpenSSL on OSX UCS4 shard

    The shard was failing when trying to build cryptography from an sdist because it could not find openssl. So, we now explicitly install it with Brew and modify the env vars to expose it. This is identical to how we install Py3 on OSX.

commit e87f567
Author: Eric Arellano <ericarellano@me.com>
Date:   Sat Feb 16 18:14:57 2019 -0800

    Fix typo in Dockerfile path

    The folder path is py27, not py2!

commit cb50136
Author: Stu Hood <stuhood@twitter.com>
Date:   Fri Feb 15 20:04:46 2019 -0800

    Prepare 1.14.0.rc2 instead. (pantsbuild#7251)

commit 28b9e8b
Author: Eric Arellano <ericarellano@me.com>
Date:   Fri Feb 15 13:38:21 2019 -0800

    Turn on PEX_VERBOSE for OSX ucs4 shard

    I can't reproduce the same failure locally. John was suspicious why for the problematic dependencies the sdist isn't being used to build the wheel when the bdist is not released for cp27mu. Hopefully this provides some insight.

commit f7472aa
Merge: 1e83f37 1ece461
Author: Eric Arellano <ericarellano@me.com>
Date:   Fri Feb 15 13:15:26 2019 -0800

    Merge branch 'master' of github.com:pantsbuild/pants into py2-wheels-abi-specified

commit 1ece461
Author: Stu Hood <stuhood@twitter.com>
Date:   Fri Feb 15 10:07:23 2019 -0800

    Prepare 1.14.0 (pantsbuild#7246)

commit fedc91c
Author: Stu Hood <stuhood@twitter.com>
Date:   Fri Feb 15 10:01:40 2019 -0800

    Avoid capturing Snapshots for previously digested codegen outputs (pantsbuild#7241)

    ### Problem

    As described in pantsbuild#7229, re-capturing `Snapshots` on noop runs in `SimpleCodegenTask` caused a performance regression for larger codegen usecases.

    ### Solution

    Remove features from the python-exposed `Snapshot` API that would prevent them from being roundtrippable via a `Digest` (including preservation of canonical paths, and preservation of literal ordering... ie. pantsbuild#5802), add support for optimistically loading a `Snapshot` from a `Digest`, and then reuse code to dump/load a `Digest` for the codegen directories to skip `Snapshot` capturing in cases where the `Digest` had already been stored.

    ### Result

    Very large codegen noop usecase runtimes reduced from `~15.2` seconds to `~3.05` seconds. Fixes pantsbuild#7229, and fixes pantsbuild#5802.

commit 594f91f
Author: Ekaterina Tyurina <tyurina.katty@yandex.ru>
Date:   Fri Feb 15 01:54:51 2019 +0000

    Add checks if values of flags zipkin-trace-id and zipkin-parent-id are valid (pantsbuild#7242)

    ### Problem
    When pants are called with flags zipkin-trace-id and zipkin-parent-id an assertion error is raised if the values of the flag are of the wrong format. The error is not informative.

    ### Solution
    Checks of values of flags zipkin-trace-id and zipkin-parent-id are added with a better error explanation. Users of the pants are asked to use 16-character  or 32-character hex string.
    Also, tests are added for these checks.

commit bc0536c
Author: Stu Hood <stuhood@twitter.com>
Date:   Thu Feb 14 13:59:26 2019 -0800

    Remove deprecated test classes (pantsbuild#7243)

    ### Problem

    `BaseTest` and the v1-aware `TaskTestBase` are long deprecated. Additionally, the `GraphTest` classes used v2 APIs that existed before `TestBase` came around.

    ### Solution

    Delete deprecated classes, and port `GraphTest` to `TestBase`.

commit e4456fd
Author: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Tue Feb 12 18:09:35 2019 -0800

    fix expected pytest output for pytest integration after pinning to 3.0.7 (pantsbuild#7240)

    ### Problem

    pantsbuild#7238 attempted to fix an upstream pytest issue (and therefore unbreak our CI) by pinning the default pytest version in our pytest subsystem to `pytest==3.0.7`. This worked, but broke a few of our other tests which relied on specific pytest output, and master is broken now (sorry!).

    I also hastily merged pantsbuild#7226, which introduced another test failure, which I have fixed. These are the only failing tests, and these all now pass locally on my laptop.

    ### Solution

    - Fix expected pytest output in pytest runner testing.

    ### Result

    I think it's still a good idea to string match pytest output unless we suddenly have to change pytest versions drastically like this again.

commit e382541
Author: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Tue Feb 12 12:54:49 2019 -0800

    Canonicalize enum pattern matching for execution strategy, platform, and elsewhere (pantsbuild#7226)

    ### Problem

    In pantsbuild#7092 we added [`NailgunTask#do_for_execution_strategy_variant()`](https://github.com/cosmicexplorer/pants/blob/70977ef064305b78406a627e07f4dae3a60e4ae4/src/python/pants/backend/jvm/tasks/nailgun_task.py#L31-L43), which allowed performing more declarative execution strategy-specific logic in nailgunnable tasks. Further work with rsc will do even more funky things with our nailgunnable task logic, and while we will eventually have a unified story again for nailgun and subprocess invocations with the v2 engine (see pantsbuild#7079), for now having this check that we have performed the logic we expect all execution strategy variants is very useful.

    This PR puts that pattern matching logic into `enum()`: https://github.com/pantsbuild/pants/blob/84cf9a75dbf68cf7126fe8372ab9b2f48720464d/src/python/pants/util/objects.py#L173-L174, among other things.

    **Note:** `TypeCheckError` and other exceptions are moved up from further down in `objects.py`.

    ### Solution

    - add `resolve_for_enum_variant()` method to `enum` which does the job of the previous `do_for_execution_strategy_variant()`
    - make the native backend's `Platform` into an enum.
    - stop silently converting a `None` argument to the enum's `create()` classmethod into its`default_value`.
    - add `register_enum_option()` helper method to register options based on enum types.

    ### Result

    We have a low-overhead way to convert potentially-tricky conditional logic into a checked pattern matching-style interface with `enum()`, and it is easier to register enum options.

commit d0432df
Author: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Tue Feb 12 12:50:59 2019 -0800

    Pin pytest version to avoid induced breakage from more-itertools transitive dep (pantsbuild#7238)

    ### Problem

    A floating transitive dependency of pytest, `more-itertools`, dropped support for python 2 in its 6.0.0 release -- see pytest-dev/pytest#4770. This is currently breaking our and our users' CI: see https://travis-ci.org/pantsbuild/pants/jobs/492004734. We could pin that dep, but as mentioned in pytest-dev/pytest#4770 (comment), pinning transitive deps of pytest would impose requirement constraints on users of pytest in pants.

    ### Solution

    - Pin `pytest==3.0.7` for now.

    ### Result

    python tests should no longer be broken.

commit 3d7a295
Author: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Mon Feb 11 22:02:21 2019 -0800

    add a TypedCollection type constraint to reduce boilerplate for datatype tuple fields (pantsbuild#7115)

    ### Problem

    *Resolves pantsbuild#6936.*

    There's been a [TODO in `pants.util.objects.Collection`](https://github.com/pantsbuild/pants/blob/c342fd3432aa0d73e402d2db7e013ecfcc76e9c8/src/python/pants/util/objects.py#L413) for a while to typecheck datatype tuple fields.

    pantsbuild#6936 has some thoughts on how to do this, but after realizing I could split out `TypeConstraint` into a base class and then introduce `BasicTypeConstraint` for type constraints which only act on the type, I think that ticket is invalidated as this solution is much cleaner.

    ### Solution

    - Split out logic for basic type checking (without looking at the object itself) into a `BasicTypeConstraint` class, which `Exactly` and friends inherit from.
    - Create the `TypedCollection` type constraint, which checks that its argument is iterable and then validates each element of the collection with a `BasicTypeConstraint` constructor argument.
      - Note that `TypedCollection` is a `TypeConstraint`, but not a `BasicTypeConstraint`, as it has to inspect the actual object object to determine whether each element matches the provided `BasicTypeConstraint`.
    - Move `pants.util.objects.Collection` into `src/python/pants/engine/objects.py`, as it is specifically for engine objects.
    - Use `TypedCollection` for the `dependencies` field of the datatype returned by `Collection.of()`.

    ### Result

    - `datatype` consumers and creators no longer have to have lots of boilerplate when using collections arguments, and those arguments can now be typechecked and made hashable for free!

    ### TODO in followup: `wrapper_type`

    See pantsbuild#7172.

commit 1e83f37
Author: Eric Arellano <ericarellano@me.com>
Date:   Mon Feb 11 17:37:31 2019 -0700

    Setup UCS2 vs UCS4 travis shards

    We must now build pantsbuild.pants with both unicode versions for Py2. So, we introduce Py2 to do this.

    We use Pyenv to install interpreter with the relevant encoding where necessary. See https://stackoverflow.com/questions/38928942/build-python-as-ucs-4-via-pyenv.

commit a02dde1
Author: Eric Arellano <ericarellano@me.com>
Date:   Thu Feb 7 11:33:53 2019 -0700

    Add ext_modules to BUILD entry

    This is going to be necessary to release Py3 with abi3.

    The line however results in the issue that this PR is going to aim to fix: now Python 2 will be built with abi `cp27m` or `cp27mu`, whereas earlier it was `none`. To test, try running `./pants setup-py --run="bdist_wheel --python-tag cp27 --plat-name=linux_x86_64" src/python/pants:pants-packaged` followed by `ls -l dist/pantsbuild.pants-1.14.0rc0/dist/`.

    Note also that ext_modules is deprecated in favor of distutils.Extension. We use this now as a temporary workaround until we add support for Extension.

commit 874ce34
Author: Chris Livingston <clivingston45@gmail.com>
Date:   Mon Feb 11 13:27:50 2019 -0500

    Validate and maybe prune interpreter cache run over run (pantsbuild#7225)

    * Purge stale interpreters from Interpreter Cache

commit 5d28cf8
Author: Daniel Wagner-Hall <dawagner@gmail.com>
Date:   Fri Feb 8 14:58:23 2019 +0000

    Prep for 1.15.0.dev0 (pantsbuild#7230)

commit 84cf9a7
Author: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Feb 6 13:56:35 2019 -0800

    deprecate implicit usage of binary_mode=True and mode='wb' in dirutil methods (pantsbuild#7120)

    ### Problem

    *Resolves pantsbuild#6543. See also [the python 3 migration project](https://github.com/pantsbuild/pants/projects/10).*

    There has been [a TODO](https://github.com/pantsbuild/pants/blob/6fcd7f7d0f8787910cfac01ec2895cdbd5cee66f/src/python/pants/util/dirutil.py#L109) pointing to pantsbuild#6543 to deprecate the `binary_mode` argument to `pants.util.dirutil.safe_file_dump()`, which wasn't canonicalized with a `deprecated_conditional`. This is because `binary_mode` doesn't quite make sense the way it does with file read methods `read_file()` and `maybe_read_file()`, because a file can be appended to as well as truncated (as opposed to reads).

    Separately, defaulting `binary_mode=True` for file read methods means more explicit conversions to unicode in a python 3 world,

    ### Solution

    - Deprecate the `binary_mode` argument to `safe_file_dump()`, as well as not explicitly specifying the `mode` argument.
      - `safe_file_dump()` now also defaults `payload=''`.
      - Also deprecate not specifying the `mode='wb'` argument in `safe_file_dump()`.
    - Deprecate not explicitly specifying the `binary_mode` argument in `{maybe_,}read_file()` and `temporary_file()` so that it can be given a default of unicode when pants finishes [migrating to python 3](https://github.com/pantsbuild/pants/projects/10) -- see pantsbuild#7121.
    - Update usages of `safe_file_dump()` across the repo.

    ### Result

    Pants plugins will see a deprecation warning if they fail to explicitly specify the `binary_mode` for file read methods in preparation for switching the default to unicode for [the python 3 switchover](https://github.com/pantsbuild/pants/projects/10). Several ambiguities in the `safe_file_dump()` method are alleviated.

    pantsbuild#7121 covers the eventual switchover to a default of `binary_mode=False` after the python 3 migration completes.

commit 224c2a0
Author: Borja Lorente <blorente@users.noreply.github.com>
Date:   Wed Feb 6 19:21:10 2019 +0000

    Make Resettable lazy again (pantsbuild#7222)

    ### Problem

    In the context of pantsbuild#6817, there is a logging issue that manifests when the daemon forks. In particular, in `fork_context`, both the daemon and the client reset some services that implement `Resettable`. Some of those services log at startup, when the client hasn't had time to reconfigure its logging to stderr, and therefore all these startup logs are intermingled in the `pantsd.log` file.

    ### Solution

    If we make `Resettable` lazy again, since we are ensured to only enter `fork_context` under a lock, the logging can only happen when the client has had time to configure its loggers.

    ### Result

    `Resettable` is now lazy. `Resettable::get()` is now implemented in terms of `Resettable::with`.

commit f281642
Author: Danny McClanahan <1305167+cosmicexplorer@users.noreply.github.com>
Date:   Wed Feb 6 10:26:07 2019 -0800

    fix _raise_deferred_exc() (pantsbuild#7008)

    ### Problem

    The `_raise_deferred_exc(self)` method in `daemon_pants_runner.py` hasn't ever been tested. As a result, it causes an easily fixable error: the issue can be easily reproduced if you register an option twice in the same task and then run with pantsd enabled (you get the wrong exception, because `exc_type` isn't needed to construct the exception again, that's what `exc_value` is for).

    ### Solution

    - Appropriately destructure `sys.exc_info()` (if that was what was used to populate `self._deferred_exception`) and re-raise the exception with its original traceback.

    ### Result

    This error is fixed, but not yet tested -- see pantsbuild#7220.

commit f0a1a9f
Author: Stu Hood <stuhood@twitter.com>
Date:   Wed Feb 6 07:17:10 2019 -0800

    Prepare 1.14.0rc1 (pantsbuild#7221)

commit b6f045d
Author: Daniel Wagner-Hall <dawagner@gmail.com>
Date:   Wed Feb 6 10:47:30 2019 +0000

    Resolve all platforms from all python targets (pantsbuild#7156)

    Don't just use the default configured targets.

    This means that _all_ transitive 3rdparty python will need to be
    resolvable in _all_ platforms in any target in the graph. This is not
    ideal (we really want to be doing per-root resolves), but because we
    currently do one global resolve, this is a decent fit.

commit b08c1fd
Author: Ekaterina Tyurina <tyurina.katty@yandex.ru>
Date:   Wed Feb 6 10:40:35 2019 +0000

    Add flag reporting-zipkin-sample-rate (pantsbuild#7211)

    ### Problem
    In the current implementation, every time the pants command is run with zipkin tracing turned on all the zipkin traces will be collected. It is not very convenient when the number of runs is very big.

    ### Solution
    Possibility to set the sample rate will allow us to have the number of traces that fits the constraints of Zipkin server.

    ### Result
    A flag `reporting-zipkin-sample-rate` was added that sets the sample rate at which to sample Zipkin traces. If flags `reporting-zipkin-trace-id` and `reporting-zipkin-parent-id` are set the sample rate will always be 100.0 (no matter what is set in `reporting-zipkin-sample-rate` flag).

commit 95638d3
Author: Stu Hood <stuhood@twitter.com>
Date:   Tue Feb 5 16:19:17 2019 -0800

    Only lint the direct sources of a linted target. (pantsbuild#7219)

    ### Problem

    The thrift linter currently redundantly lints the transitive dependencies of each target, leading to repetitive errors, and larger tool invokes than necessary.

    ### Solution

    Lint only the directly owned sources of a target, and expand unit tests.

commit 121f98c
Author: Stu Hood <stuhood@twitter.com>
Date:   Tue Feb 5 16:05:11 2019 -0800

    Do not render the coursier workunit unless it will run. (pantsbuild#7218)

    ### Problem

    Currently the `bootstrap-coursier` workunit is rendered repeatedly, although it only actually runs once.

    ### Solution

    Only render the workunit if it will run.

commit b2f5a49
Author: Marcin Podolski <marcinex7@gmail.com>
Date:   Wed Feb 6 00:17:09 2019 +0100

    documentation for grpcio (pantsbuild#7155)

    ### Problem

    Documentation for grpcio generation tool

commit f73f112
Author: Daniel Wagner-Hall <dawagner@gmail.com>
Date:   Tue Feb 5 15:33:43 2019 +0000

    Skip flaky test (pantsbuild#7209)

    Relates to pantsbuild#7199

commit f0bb0da
Author: Stu Hood <stuhood@twitter.com>
Date:   Mon Feb 4 20:51:34 2019 -0800

    Only run master-dependent commithooks on master (pantsbuild#7214)

    ### Problem

    See pantsbuild#7213: some commit hooks are only valid in a context where master is present (and should otherwise be skipped).

    ### Solution

    Move more hooks that reference `master` under the check for the presence of `master`.

    ### Result

    Fixes pantsbuild#7213, and unblocks further iteration on the `1.14.x` stable branch.
Originally we did not release Pex36 so the only option was Pex37. This was an issue because Pex37 means we need a Py37 interpreter available, which we do not have in the Py3 wheel building shards and do not want to setup as it would require bootstrapping Pants again.

Note this commit is going to be blocked on the upgrade to pex 1.6.2 to work properly. But once that's merged, this should work.
Eric-Arellano added a commit that referenced this pull request Mar 7, 2019
…erpreter constraints requested (#7285)

### Problem
When running `./pants binary`, we do not consider the global interpreter constraints in passing them to the Pex builder.

This resulted in a bug where the interpreter used to build a PEX in CI differed from the Python used to run the PEX. See pex-tool/pex#676. To resolve this issue, we need some way to specify the Pex builder must use the specific Python version 2.7.13 (UCS4). This blocks 
#7235.

### Solution
Modify `PexBuilderWrapper` to use `compatibility_or_constrains()`. This will first try to get the compatibility constraints from the target itself, and then resort to using the PythonSetup subsystem's constraints (for the global instance, resolved from `pants.ini`, `PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS`, and `--interpreter-constraints`).

### Result
Users of the function `add_interpreter_constraints_from()` will now add the global `--interpreter-constraints` to the Pex builder if the targets do not themselves specify a constraint. At the moment, this only impacts `./pants binary`, as `python_binary_create.py` is the only file which uses this function.

Note this is still not a great solution. It would be better to kill `add_interpreter_constraint()` and `add_interpreter_constraints_from()` to instead automatically set the interpreter constraints from the targets' graph. This PR does not make that change to avoid scope creep.

#### Skipped tests
Due to pex-tool/pex#655, we must now skip several tests that now fail. PEX does not correctly OR interpreter constraints, so these tests complain that they cannot find an appropriate interpreter to satisfy `['CPython>=3.6,<4', 'CPython>=2.7,<3']`.

Because this PR blocks #7235, which blocks #7197, we skip these tests until the upstream fix pex-tool/pex#678 is merged into PEX and Pants upgrades its PEX to the new version #7186.
Eric-Arellano added a commit that referenced this pull request Mar 9, 2019
…CS4 (#7235)

### Problem
We should be marking the [ABI (application binary interface)](https://docs.python.org/3/c-api/stable.html) for the `pantsbuild.pants` wheel because it uses native code. Currently, we mark the ABI as `none`, which is incorrect per https://www.python.org/dev/peps/pep-0513/#ucs-2-vs-ucs-4-builds.

In particular, in Python 2, Python may be installed with either UCS2 (UTF-16) or UCS4 (UTF-8). We should be marking the wheel as either `cp27m` for UCS2 or `cp27mu` for UCS4.

As a result of marking the ABI, we must now produce more wheels. macOS defaults to UCS2. For Linux, "ucs4 is much more widespread among Linux CPython distributions." We do not want to rely on these assumptions, however, when releasing, as some users may not have these default unicode settings. So, instead we must release `pantsbuild.pants` as both a `cp27m` and `cp27mu` wheel, and rely on Pip to resolve which the user should use.

### Solution
At a high level, this PR does two things:
1. Marks that the ABI should be specified, rather than `none`.
1. Sets up 4 Travis shards so that we build both a `cp27m` and `cp27mu` wheel for both Linux and OSX. See https://travis-ci.org/pantsbuild/pants/builds/503639333 for the end result of this.

To setup the new shards, we use Pyenv to install new versions of Python 2 with the appropriate unicode settings, thanks to the env var `PYTHON_CONFIGURE_OPTS=--enable-unicode=ucs{2,4}` (https://stackoverflow.com/a/38930764). 

Because both the OSX UCS4 shard and Linux UCS2 shard already have Python 2.7 installed, we must install a Python 2.7.x version different than what is already there, and use `PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS` to ensure Pants and PEX are using the exact interpreter we want. For this reason, this PR was blocked by #7285 to propagate interpreter constraints to PEX.

Specifically, we make these changes to achieve these two high level goals:

1. Modify [`src/python/pants/BUILD`](https://github.com/pantsbuild/pants/pull/7235/files#diff-3ce39309d74098493a1f3c8107292a8d) so that `bdist_wheel` knows it needs to mark the ABI. This achieves goal 1.
1. Change [`release.sh`](https://github.com/pantsbuild/pants/pull/7235/files#diff-9ed7102b7836807dc342cc2246ec4839) to allow pre-setting `$PY` and to also set `PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS` in order to use the specific Python interpreter we are targeting.
1. Create [`travis_ci_py27_ucs2/Dockerfile`](https://github.com/pantsbuild/pants/pull/7235/files#diff-b90425bcccb6969a98b9d1c4066422a8) to get Python 2 w/ UCS2 onto Linux.
1. Extract out `.travis.yml` `env` entries to get OpenSSL and Homebrew-installed Python working properly, along with launching a Docker image, in order to avoid duplication: [`env_osx_with_pyenv.mustache`](https://github.com/pantsbuild/pants/pull/7235/files#diff-c2ab029a1887e2e99f2391fc7568cc5f), [`launch_docker_image.mustache`](https://github.com/pantsbuild/pants/pull/7235/files#diff-3d4a0cecc373a624f233521f76d66999), and [`generate_travis_yml.py`](https://github.com/pantsbuild/pants/pull/7235/files#diff-c11e2f109e12527d2e1ac2c62161edf6).
1. Modify [`travis.yml.mustache`](https://github.com/pantsbuild/pants/pull/7235/files#diff-88af3146f5cc486b749ed790399bde46) to set up the 4 distinct wheel building shards. Similar to how we created a Dockerfile to use Pyenv to install Python 2 with UCS2 on Linux, we use Pyenv to install Python 2 with UCS4 on OSX. We also move the wheel building shards below unit tests.

#### Ensuring the correct abi is used
We need to ensure the `pants.pex` used by `release.sh` has the correct abi for its dependencies, and that `release.sh` is using the correct Python interpreter. We introduce a new script [`check_pants_pex_abi.py`](https://github.com/pantsbuild/pants/pull/7235/files#diff-8b857b8cee6cb9784bf37950220c587b) that inspects the pex's `PEX-INFO` to ensure the targeted abi was used.

An even better test would test the result of `release.sh` to ensure the built `pantsbuild.pants` wheel has the correct ABI and can be consumed properly. Currently `release.sh` verifies the wheel is valid, but it does not enforce which ABI it was built with. This could be a good followup PR.

### Result
We now properly mark the ABI for Python 2. Beyond the new script `check_pants_pex_abi.sh` proving this, we performed a run of this PR with verbose PEX logging turned on: https://travis-ci.org/pantsbuild/pants/builds/503639333. Inspecting the logs for the wheel building shards and searching for `Using the current platform` proves the 4 wheel building shards are using the correct interpreter and abi.

In addition to correctness for Python 2, this unblocks releasing Python 3 wheels (#7197).

Note this should have no significant impact on the end user, as Pip will resolve to the current ABI for their interpreter. It will change the name of our `pantsbuild.pants` wheel and will prevent using that wheel with an interpreter that uses a different UCS setting, but all users should be able to pull down whichever wheel they need as we provide wheels for both UCS2 and UCS4 on both OSX and Linux.

#### Downside: wheel building explosion
We currently are building more wheels than necessary. For wheels that are universal / platform-independent, we only need them to be built once, but we build them every time. See #7258.

This PR adds two new shards so adds ~30 unnecessary core wheels we build, in addition to 3rd party wheels that are universal / platform-independent.
I didn't realize we have a requirements function, which for the given name finds its entry in requirements.txt. This means that Py2 only requirements will already have the string "python_version < 3".
Smaller diff. Also it makes sense to keep the usage() closer to the source code for what each option does.

There's the issue of _OPTS being further from where most are used, but I still think this is a better design than having usage at the top far from the switch statement against each op.
Eric-Arellano added a commit that referenced this pull request Mar 11, 2019
…omments (#7351)

### Problem
#7235 introduced an initial deduplication of launching docker images, notably allowing the image name to be parametrized. While working on #7197, it became evident how frequently the same code was being repeated for calling `docker run`. The only thing that changes between these invocations is what goes inside `sh -c "my bash command"`.

This duplication makes our Travis code harder to understand, along with the usual arguments for DRY.

### Solution
Introduce a new `docker_run_image` mustache template that takes the parameters `$docker_image_name` and `$docker_run_command`.

Also changed:
* rename `launch_docker_image` -> `docker_build_image`
* remove script entry for `&travis_docker_image`, as it gets overridden every time by its caller so is unnecessary and confusing.
* introduce Mustache comments to some of our templates. These don't get copied over into `.travis.yml`, which is good for reducing noise, while still allowing us to explain the concept in the template. 

### Result
CI should perform the same with the benefit of a shorter and better abstracted `travis.yml.mustache`.
Copy link
Member

@stuhood stuhood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

build-support/bin/release.sh Outdated Show resolved Hide resolved
Copy link
Contributor

@benjyw benjyw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

# Once we release Python 3 wheels, we will expect 6 wheels: {linux,osx} * {cp27m,cp27mu,abi3}.
if [ "${cross_platform}" != "true" ] && [ ${#packages[@]} -ne 4 ]; then
# N.B. For platform-specific wheels, we expect 6 wheels: {linux,osx} * {cp27m,cp27mu,abi3}.
if [ "${cross_platform}" != "true" ] && [ ${#packages[@]} -ne 6 ]; then
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@benjyw this is updated, along with the release documentation.

Thanks for the review!

@Eric-Arellano
Copy link
Contributor Author

Merging now so that we have two days to test the dry release in CI. Happy to address any concerns inn a followup PR.

Big milestone here 😀

@Eric-Arellano Eric-Arellano merged commit b3ab38c into pantsbuild:master Mar 14, 2019
@Eric-Arellano Eric-Arellano deleted the py3-wheels branch March 14, 2019 03:35
@mateor
Copy link
Member

mateor commented Mar 14, 2019

Incredible work, Eric.

@@ -651,7 +683,7 @@ function usage() {
echo " -p Build a pex from prebuilt wheels for this release."
echo " -q Build a pex which only works on the host platform, using the code as exists on disk."
echo
echo "All options (except for '-d') are mutually exclusive."
echo "All options (except for '-d' and '-3') are mutually exclusive."
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Might be useful to capture the mutual exclusive args in a variable so that this CLI help doesn't drift from reality. Fine to punt or do as a followup.

    echo "All options (except for ${_PANTS_OPTS_MUTUAL_EXCLUSIVE}) are mutually exclusive."

or some better label.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea.

Even better will be using Python's argparse to handle this all for us + to allow us to use long-form arguments. Think it will become much more tenable when we drop Py2 and can start using subprocess.run([my, bash, command]) over subprocess.Popen().communicate().

Package("pantsbuild.pants.testinfra", "//tests/python/pants_test:test_infra"),
}
def core_packages(py3):
# N.B. When releasing with Python 3, we constrain the ABI (Application Binary Interface) to cp36 to allow
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

love the comment details man

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fun fact, you all taught me what NB means! And taught me that while self-documenting code is great, with highly technical situations like this, comments with links to standards and upstream issues are a great thing.

@mateor
Copy link
Member

mateor commented Mar 14, 2019

post-merge comments are all non-blocking and punt-worthy ;)

Eric-Arellano added a commit to pantsbuild/setup that referenced this pull request Mar 16, 2019
### Problem
Now that we're releasing Pants with Python 3 wheels (pantsbuild/pants#7197, overall migration tracked by pantsbuild/pants#6062), we need to add a way for every day Pants users to consume the upgrade.

See #30 for more context, including the design goals.

### Solution
Parse the `pants_runtime_python_version` from `pants.ini` to determine which Python interpreter to use. If it's missing, default to Python 2.7.

This requires modifying the venv folders to support multiple Python versions. Rather than using the naming scheme `${pants_version}`, we now consistently use `${pants_version}_py{py_major_minor}`, regardless of if the user is using `pants_runtime_python_version`.

It also requires upgrading virtualenv to avoid a deprecation warning when running with Python 3.

Finally, we add tests to ensure this all works.

### Result
Running `./pants` like normal for the first time will bootstrap Pants again and create the new folder `1.15.dev3_py{27,36,37}`.

Changing the `$PYTHON` env var or the `pants_runtime_python_version` in `pants.ini` will change which Python version is used to run Pants.

When changing interpreters for the first time, it will bootstrap Pants again. After that, you can change without issue.
Eric-Arellano added a commit that referenced this pull request Mar 20, 2019
### Problem
While wheels are now released with Python 3 support thanks to #7197, this is only one of two ways people consume Pants. The other way is through a Pex released to Github. 

We decided to release both a Py27 and Py36 Pex. 

We will not be releasing a Py37 Pex, as it is would require Py37 wheel building shards which would require a Centos7 docker image. We decided this is not worth the work for how few people we anticipate would use Py37 with Pex combined.

We will also not be releasing a more flexible Pex that works with either Py27 or Py36, as it is blocked by pex-tool/pex#654. We also want organizations pinning their Python version, so a flexible release is not particularly useful in this instance.

This implements #6450 (comment).

### Solution
Release both `pants.${VERSION}.py{27,36}.pex`.

This requires changing how we run `release.sh -p` and adding new CI shards to deploy on both Py2 and Py3.

### Result
The next release, we will release both `pants.1.15.0.rc1.py27.pex` and  `pants.1.15.0.rc1.py36.pex`.
stuhood pushed a commit that referenced this pull request Mar 21, 2019
### Problem
While wheels are now released with Python 3 support thanks to #7197, this is only one of two ways people consume Pants. The other way is through a Pex released to Github. 

We decided to release both a Py27 and Py36 Pex. 

We will not be releasing a Py37 Pex, as it is would require Py37 wheel building shards which would require a Centos7 docker image. We decided this is not worth the work for how few people we anticipate would use Py37 with Pex combined.

We will also not be releasing a more flexible Pex that works with either Py27 or Py36, as it is blocked by pex-tool/pex#654. We also want organizations pinning their Python version, so a flexible release is not particularly useful in this instance.

This implements #6450 (comment).

### Solution
Release both `pants.${VERSION}.py{27,36}.pex`.

This requires changing how we run `release.sh -p` and adding new CI shards to deploy on both Py2 and Py3.

### Result
The next release, we will release both `pants.1.15.0.rc1.py27.pex` and  `pants.1.15.0.rc1.py36.pex`.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants