From 7b2225e0e0e004db6d99abf14329c6bccac65bf8 Mon Sep 17 00:00:00 2001 From: Eric Arellano Date: Sat, 9 Mar 2019 09:09:16 -0700 Subject: [PATCH] Specify ABI for pantsbuild.pants wheel and build with both UCS2 and UCS4 (#7235) ### Problem We should be marking the [ABI (application binary interface)](https://docs.python.org/3/c-api/stable.html) for the `pantsbuild.pants` wheel because it uses native code. Currently, we mark the ABI as `none`, which is incorrect per https://www.python.org/dev/peps/pep-0513/#ucs-2-vs-ucs-4-builds. In particular, in Python 2, Python may be installed with either UCS2 (UTF-16) or UCS4 (UTF-8). We should be marking the wheel as either `cp27m` for UCS2 or `cp27mu` for UCS4. As a result of marking the ABI, we must now produce more wheels. macOS defaults to UCS2. For Linux, "ucs4 is much more widespread among Linux CPython distributions." We do not want to rely on these assumptions, however, when releasing, as some users may not have these default unicode settings. So, instead we must release `pantsbuild.pants` as both a `cp27m` and `cp27mu` wheel, and rely on Pip to resolve which the user should use. ### Solution At a high level, this PR does two things: 1. Marks that the ABI should be specified, rather than `none`. 1. Sets up 4 Travis shards so that we build both a `cp27m` and `cp27mu` wheel for both Linux and OSX. See https://travis-ci.org/pantsbuild/pants/builds/503639333 for the end result of this. To setup the new shards, we use Pyenv to install new versions of Python 2 with the appropriate unicode settings, thanks to the env var `PYTHON_CONFIGURE_OPTS=--enable-unicode=ucs{2,4}` (https://stackoverflow.com/a/38930764). Because both the OSX UCS4 shard and Linux UCS2 shard already have Python 2.7 installed, we must install a Python 2.7.x version different than what is already there, and use `PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS` to ensure Pants and PEX are using the exact interpreter we want. For this reason, this PR was blocked by https://github.com/pantsbuild/pants/pull/7285 to propagate interpreter constraints to PEX. Specifically, we make these changes to achieve these two high level goals: 1. Modify [`src/python/pants/BUILD`](https://github.com/pantsbuild/pants/pull/7235/files#diff-3ce39309d74098493a1f3c8107292a8d) so that `bdist_wheel` knows it needs to mark the ABI. This achieves goal 1. 1. Change [`release.sh`](https://github.com/pantsbuild/pants/pull/7235/files#diff-9ed7102b7836807dc342cc2246ec4839) to allow pre-setting `$PY` and to also set `PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS` in order to use the specific Python interpreter we are targeting. 1. Create [`travis_ci_py27_ucs2/Dockerfile`](https://github.com/pantsbuild/pants/pull/7235/files#diff-b90425bcccb6969a98b9d1c4066422a8) to get Python 2 w/ UCS2 onto Linux. 1. Extract out `.travis.yml` `env` entries to get OpenSSL and Homebrew-installed Python working properly, along with launching a Docker image, in order to avoid duplication: [`env_osx_with_pyenv.mustache`](https://github.com/pantsbuild/pants/pull/7235/files#diff-c2ab029a1887e2e99f2391fc7568cc5f), [`launch_docker_image.mustache`](https://github.com/pantsbuild/pants/pull/7235/files#diff-3d4a0cecc373a624f233521f76d66999), and [`generate_travis_yml.py`](https://github.com/pantsbuild/pants/pull/7235/files#diff-c11e2f109e12527d2e1ac2c62161edf6). 1. Modify [`travis.yml.mustache`](https://github.com/pantsbuild/pants/pull/7235/files#diff-88af3146f5cc486b749ed790399bde46) to set up the 4 distinct wheel building shards. Similar to how we created a Dockerfile to use Pyenv to install Python 2 with UCS2 on Linux, we use Pyenv to install Python 2 with UCS4 on OSX. We also move the wheel building shards below unit tests. #### Ensuring the correct abi is used We need to ensure the `pants.pex` used by `release.sh` has the correct abi for its dependencies, and that `release.sh` is using the correct Python interpreter. We introduce a new script [`check_pants_pex_abi.py`](https://github.com/pantsbuild/pants/pull/7235/files#diff-8b857b8cee6cb9784bf37950220c587b) that inspects the pex's `PEX-INFO` to ensure the targeted abi was used. An even better test would test the result of `release.sh` to ensure the built `pantsbuild.pants` wheel has the correct ABI and can be consumed properly. Currently `release.sh` verifies the wheel is valid, but it does not enforce which ABI it was built with. This could be a good followup PR. ### Result We now properly mark the ABI for Python 2. Beyond the new script `check_pants_pex_abi.sh` proving this, we performed a run of this PR with verbose PEX logging turned on: https://travis-ci.org/pantsbuild/pants/builds/503639333. Inspecting the logs for the wheel building shards and searching for `Using the current platform` proves the 4 wheel building shards are using the correct interpreter and abi. In addition to correctness for Python 2, this unblocks releasing Python 3 wheels (https://github.com/pantsbuild/pants/pull/7197). Note this should have no significant impact on the end user, as Pip will resolve to the current ABI for their interpreter. It will change the name of our `pantsbuild.pants` wheel and will prevent using that wheel with an interpreter that uses a different UCS setting, but all users should be able to pull down whichever wheel they need as we provide wheels for both UCS2 and UCS4 on both OSX and Linux. #### Downside: wheel building explosion We currently are building more wheels than necessary. For wheels that are universal / platform-independent, we only need them to be built once, but we build them every time. See https://github.com/pantsbuild/pants/issues/7258. This PR adds two new shards so adds ~30 unnecessary core wheels we build, in addition to 3rd party wheels that are universal / platform-independent. --- .travis.yml | 144 +++++++++++++++--- build-support/bin/check_pants_pex_abi.py | 70 +++++++++ build-support/bin/release.sh | 19 ++- .../docker/travis_ci_py27_ucs2/Dockerfile | 49 ++++++ .../travis/env_osx_with_pyenv.mustache | 5 + build-support/travis/generate_travis_yml.py | 19 ++- .../travis/launch_docker_image.mustache | 7 + build-support/travis/travis.yml.mustache | 134 +++++++++++----- src/python/pants/BUILD | 14 +- src/python/pants/dummy.c | 0 10 files changed, 388 insertions(+), 73 deletions(-) create mode 100755 build-support/bin/check_pants_pex_abi.py create mode 100644 build-support/docker/travis_ci_py27_ucs2/Dockerfile create mode 100644 build-support/travis/env_osx_with_pyenv.mustache create mode 100644 build-support/travis/launch_docker_image.mustache create mode 100644 src/python/pants/dummy.c diff --git a/.travis.yml b/.travis.yml index 707a37ef2df..4d7db0fd187 100644 --- a/.travis.yml +++ b/.travis.yml @@ -170,7 +170,6 @@ py36_osx_config: &py36_osx_config packages: &py36_osx_config_brew_packages - openssl env: - # Fix Python 3 issue linking to OpenSSL - &py36_osx_config_env > PATH="/usr/local/opt/openssl/bin:$PATH" LDFLAGS="-L/usr/local/opt/openssl/lib" @@ -234,14 +233,14 @@ travis_docker_image: &travis_docker_image before_script: - ulimit -c unlimited script: - # NB: This script definition is very likely to be overridden in a consumer, so we alias it - # for re-inclusion. - - &travis_docker_image_launch docker build --rm -t travis_ci + - > + docker build + --rm -t ${docker_image_name} --build-arg "TRAVIS_USER=$(id -un)" --build-arg "TRAVIS_UID=$(id -u)" --build-arg "TRAVIS_GROUP=$(id -gn)" --build-arg "TRAVIS_GID=$(id -g)" - build-support/docker/travis_ci/ + build-support/docker/${docker_image_name}/ # ------------------------------------------------------------------------- # Bootstrap engine shards @@ -252,14 +251,21 @@ base_linux_build_engine: &base_linux_build_engine <<: *travis_docker_image stage: *bootstrap script: - - *travis_docker_image_launch + - > + docker build + --rm -t ${docker_image_name} + --build-arg "TRAVIS_USER=$(id -un)" + --build-arg "TRAVIS_UID=$(id -u)" + --build-arg "TRAVIS_GROUP=$(id -gn)" + --build-arg "TRAVIS_GID=$(id -g)" + build-support/docker/${docker_image_name}/ # Note that: # * We mount ${HOME} to cache the ${HOME}/.cache/pants/rust-toolchain. # * We also build fs_util, to take advantage of the rust code built during bootstrapping. - docker run --rm -t -v "${HOME}:/travis/home" -v "${TRAVIS_BUILD_DIR}:/travis/workdir" - travis_ci:latest + ${docker_image_name}:latest sh -c "./build-support/bin/ci.sh ${BOOTSTRAP_ARGS} && ./build-support/bin/release.sh -f" - aws --no-sign-request --region us-east-1 s3 cp ${TRAVIS_BUILD_DIR}/pants.pex ${BOOTSTRAPPED_PEX_URL_PREFIX}.${BOOTSTRAPPED_PEX_KEY_SUFFIX} @@ -268,6 +274,7 @@ py27_linux_build_engine: &py27_linux_build_engine <<: *base_linux_build_engine name: "Build Linux native engine and pants.pex (Py2.7 PEX)" env: + - docker_image_name=travis_ci # NB: Only the Py2.7 shard sets PREPARE_DEPLOY to cause the fs_util binary to be uploaded to S3: # either linux shard could upload this binary, since it does not depend on python at all. - PREPARE_DEPLOY=1 @@ -280,6 +287,7 @@ py36_linux_build_engine: &py36_linux_build_engine <<: *base_linux_build_engine name: "Build Linux native engine and pants.pex (Py3.6 PEX)" env: + - docker_image_name=travis_ci - CACHE_NAME=linuxpexbuild.py36 - BOOTSTRAPPED_PEX_KEY_SUFFIX=py36.linux - BOOTSTRAP_ARGS='-b' @@ -381,41 +389,128 @@ cargo_audit: &cargo_audit # Build wheels # ------------------------------------------------------------------------- +# N.B. With Python 2, we must build pantsbuild.pants with both UCS2 and UCS4 to provide full +# compatibility for end users. This is because we constrain our ABI due to the native engine. +# See https://www.python.org/dev/peps/pep-0513/#ucs-2-vs-ucs-4-builds. Note this distinction is +# not necessary with Python 3.3+ due to flexible storage of Unicode strings (https://www.python.org/dev/peps/pep-0393/). +# +# We treat both Linux UCS4 and OSX UCS2 normally, as these are the defaults for those environments. +# The Linux UCS2 and OSX UCS4 shards, however, must rebuild Python with +# `PYTHON_CONFIGURE_OPTS=--enable-unicode=ucs{2,4}` set, along with bootstrapping Pants again rather +# than pulling the PEX from AWS. + base_build_wheels: &base_build_wheels stage: *test env: - - &base_build_wheels_env RUN_PANTS_FROM_PEX=1 PREPARE_DEPLOY=1 + - &base_build_wheels_env PREPARE_DEPLOY=1 -linux_build_wheels: &linux_build_wheels +py27_linux_build_wheels_no_ucs: &py27_linux_build_wheels_no_ucs # Similar to the bootstrap shard, we build Linux wheels in a docker image to maximize # compatibility. This is a Py2.7 shard, so it is not subject to #6985. <<: *travis_docker_image - <<: *py27_linux_test_config <<: *base_build_wheels - name: "Build Linux wheels (No PEX)" + +py27_linux_build_wheels_ucs2: &py27_linux_build_wheels_ucs2 + <<: *py27_linux_config + <<: *py27_linux_build_wheels_no_ucs + <<: *native_engine_cache_config + name: "Build wheels - Linux and cp27m (UCS2)" + env: + - *base_build_wheels_env + - docker_image_name=travis_ci_py27_ucs2 + - CACHE_NAME=linuxwheelsbuild.ucs2 + script: + - > + docker build + --rm -t ${docker_image_name} + --build-arg "TRAVIS_USER=$(id -un)" + --build-arg "TRAVIS_UID=$(id -u)" + --build-arg "TRAVIS_GROUP=$(id -gn)" + --build-arg "TRAVIS_GID=$(id -g)" + build-support/docker/${docker_image_name}/ + - docker run --rm -t + -v "${HOME}:/travis/home" + -v "${TRAVIS_BUILD_DIR}:/travis/workdir" + ${docker_image_name}:latest + sh -c "./build-support/bin/ci.sh -2b + && ./build-support/bin/check_pants_pex_abi.py cp27m + && RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n" + +py27_linux_build_wheels_ucs4: &py27_linux_build_wheels_ucs4 + <<: *py27_linux_build_wheels_no_ucs + <<: *py27_linux_test_config + # `py27_linux_test_config` overrides the stage set by `base_build_wheels`, so we re-override it. + stage: *test + name: "Build wheels - Linux and cp27mu (UCS4)" env: - *py27_linux_test_config_env - *base_build_wheels_env - - CACHE_NAME=linuxwheelsbuild + - docker_image_name=travis_ci + - CACHE_NAME=linuxwheelsbuild.ucs4 script: - - *travis_docker_image_launch + - > + docker build + --rm -t ${docker_image_name} + --build-arg "TRAVIS_USER=$(id -un)" + --build-arg "TRAVIS_UID=$(id -u)" + --build-arg "TRAVIS_GROUP=$(id -gn)" + --build-arg "TRAVIS_GID=$(id -g)" + build-support/docker/${docker_image_name}/ - docker run --rm -t -v "${HOME}:/travis/home" -v "${TRAVIS_BUILD_DIR}:/travis/workdir" - travis_ci:latest - sh -c "RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n" + ${docker_image_name}:latest + sh -c "./build-support/bin/check_pants_pex_abi.py cp27mu + && RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n" -osx_build_wheels: &osx_build_wheels - <<: *py27_osx_test_config +py27_osx_build_wheels_no_ucs: &py27_osx_build_wheels_no_ucs <<: *base_build_wheels - name: "Build OSX wheels (No PEX)" osx_image: xcode8 + +py27_osx_build_wheels_ucs2: &py27_osx_build_wheels_ucs2 + <<: *py27_osx_test_config + <<: *py27_osx_build_wheels_no_ucs + name: "Build wheels - OSX and cp27m (UCS2)" env: - *py27_osx_test_config_env - *base_build_wheels_env - - CACHE_NAME=osxwheelsbuild + - CACHE_NAME=osxwheelsbuild.ucs2 + script: + - ./build-support/bin/check_pants_pex_abi.py cp27m + - RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n + +py27_osx_build_wheels_ucs4: &py27_osx_build_wheels_ucs4 + <<: *py27_osx_config + <<: *py27_osx_build_wheels_no_ucs + <<: *native_engine_cache_config + name: "Build wheels - OSX and cp27mu (UCS4)" + addons: + brew: + packages: + - openssl + env: + - *base_build_wheels_env + - CACHE_NAME=osxwheelsbuild.ucs4 + - > + PATH="/usr/local/opt/openssl/bin:$PATH" + LDFLAGS="-L/usr/local/opt/openssl/lib" + CPPFLAGS="-I/usr/local/opt/openssl/include" + PYENV_ROOT="${HOME}/.pyenv" + PATH="${PYENV_ROOT}/shims:${PATH}" + - PYTHON_CONFIGURE_OPTS=--enable-unicode=ucs4 + # We set $PY to ensure the UCS4 interpreter is used when bootstrapping the PEX. + - PY=${PYENV_ROOT}/shims/python2.7 + before_install: + - curl -L https://github.com/stedolan/jq/releases/download/jq-1.5/jq-osx-amd64 -o /usr/local/bin/jq + - chmod 755 /usr/local/bin/jq + - ./build-support/bin/install_aws_cli_for_ci.sh + - git clone https://github.com/pyenv/pyenv ${PYENV_ROOT} + - ${PYENV_ROOT}/bin/pyenv install 2.7.13 + - ${PYENV_ROOT}/bin/pyenv global 2.7.13 script: - - ./build-support/bin/release.sh -n + - ./build-support/bin/ci.sh -2b + - ./build-support/bin/check_pants_pex_abi.py cp27mu + - RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n # ------------------------------------------------------------------------- # Rust tests @@ -607,7 +702,6 @@ matrix: include: - <<: *py27_linux_build_engine - <<: *py36_linux_build_engine - - <<: *py27_osx_build_engine - <<: *py36_osx_build_engine @@ -617,9 +711,6 @@ matrix: - <<: *linux_rust_clippy - <<: *cargo_audit - - <<: *linux_build_wheels - - <<: *osx_build_wheels - - <<: *py27_linux_test_config name: "Unit tests for pants and pants-plugins (Py2.7 PEX)" stage: *test @@ -637,6 +728,11 @@ matrix: script: - ./build-support/bin/travis-ci.sh -lp + - <<: *py27_linux_build_wheels_ucs2 + - <<: *py27_linux_build_wheels_ucs4 + - <<: *py27_osx_build_wheels_ucs2 + - <<: *py27_osx_build_wheels_ucs4 + - <<: *py36_linux_test_config name: "Integration tests for pants - shard 0 (Py3.6 PEX)" env: diff --git a/build-support/bin/check_pants_pex_abi.py b/build-support/bin/check_pants_pex_abi.py new file mode 100755 index 00000000000..107888e4fa2 --- /dev/null +++ b/build-support/bin/check_pants_pex_abi.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python2.7 +# coding=utf-8 +# Copyright 2019 Pants project contributors (see CONTRIBUTORS.md). +# Licensed under the Apache License, Version 2.0 (see LICENSE). + +# Check that the ./pants.pex was built using the passed abi specification. + +from __future__ import absolute_import, division, print_function, unicode_literals + +import argparse +import json +import os.path +import zipfile + + +RED = "\033[31m" +BLUE = "\033[34m" +RESET = "\033[0m" + + +def main(): + if not os.path.isfile("pants.pex"): + die("pants.pex not found! Ensure you are in the repository root, then run " \ + "'./build-support/bin/ci.sh -b' to bootstrap pants.pex with Python 3 or " \ + "'./build-support/bin/ci.sh -2b' to bootstrap pants.pex with Python 2.") + expected_abi = create_parser().parse_args().abi + with zipfile.ZipFile("pants.pex", "r") as pex: + with pex.open("PEX-INFO", "r") as pex_info: + pex_info_content = str(pex_info.readline()) + parsed_abis = { + parse_abi_from_filename(filename) + for filename in json.loads(pex_info_content)["distributions"].keys() + if parse_abi_from_filename(filename) != "none" + } + if len(parsed_abis) < 1: + die("No abi tag found. Expected: {}.".format(expected_abi)) + elif len(parsed_abis) > 1: + die("Multiple abi tags found. Expected: {}, found: {}.".format(expected_abi, parsed_abis)) + found_abi = list(parsed_abis)[0] + if found_abi != expected_abi: + die("pants.pex was built with the incorrect ABI. Expected: {}, found: {}.".format(expected_abi, found_abi)) + success("Success. As expected, pants.pex was built with the ABI {}.".format(expected_abi)) + + +def create_parser(): + parser = argparse.ArgumentParser( + description="Check that ./pants.pex was built using the passed abi specification." + ) + parser.add_argument("abi", help="The expected abi, e.g. `cp27m` or `abi3`") + return parser + + +def parse_abi_from_filename(filename): + """This parses out the abi from a wheel filename. + + For example, `configparser-3.5.0-py2-abi3-any.whl` would return `abi3`. + See https://www.python.org/dev/peps/pep-0425/#use for how wheel filenames are defined.""" + return filename.split("-")[-2] + + +def success(message): + print("{}{}{}".format(BLUE, message, RESET)) + + +def die(message): + raise SystemExit("{}{}{}".format(RED, message, RESET)) + + +if __name__ == "__main__": + main() diff --git a/build-support/bin/release.sh b/build-support/bin/release.sh index 2df73d4216c..b8c479b78d1 100755 --- a/build-support/bin/release.sh +++ b/build-support/bin/release.sh @@ -7,9 +7,22 @@ set -e ROOT=$(cd $(dirname "${BASH_SOURCE[0]}") && cd "$(git rev-parse --show-toplevel)" && pwd) source ${ROOT}/build-support/common.sh -PY=$(which python2.7 || exit 0) -[[ -n "${PY}" ]] || die "You must have python2.7 installed and on the path to release." -export PY +# Set the Python interpreter to be used for the virtualenv. Note we allow the user to +# predefine this value so that they may point to a specific interpreter, e.g. 2.7.13 vs. 2.7.15. +export PY="${PY:-python2.7}" +if ! which "${PY}" >/dev/null; then + die "Python interpreter ${PY} not discoverable on your PATH." +fi +py_major_minor=$(${PY} -c 'import sys; print(".".join(map(str, sys.version_info[0:2])))') +if [[ "${py_major_minor}" != "2.7" ]]; then + die "Invalid interpreter. The release script requires python2.7, and you are using python${py_major_minor}." +fi + +# Also set PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS. We set this to the exact Python version +# to resolve any potential ambiguity when multiple Python interpreters are discoverable, such as +# Python 2.7.13 vs. 2.7.15. +py_major_minor_patch=$(${PY} -c 'import sys; print(".".join(map(str, sys.version_info[0:3])))') +export PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS="${PANTS_PYTHON_SETUP_INTERPRETER_CONSTRAINTS:-['CPython==${py_major_minor_patch}']}" function run_local_pants() { ${ROOT}/pants "$@" diff --git a/build-support/docker/travis_ci_py27_ucs2/Dockerfile b/build-support/docker/travis_ci_py27_ucs2/Dockerfile new file mode 100644 index 00000000000..ca729381db7 --- /dev/null +++ b/build-support/docker/travis_ci_py27_ucs2/Dockerfile @@ -0,0 +1,49 @@ +# Copyright 2019 Pants project contributors (see CONTRIBUTORS.md). +# Licensed under the Apache License, Version 2.0 (see LICENSE). + +# NB: this file duplicates travis_ci/Dockerfile, except it installs Python 2.7 with UCS2. +# We do not include this change directly in centos6/Dockerfile nor in travis_ci/Dockerfile +# because we only want access to this Python interpreter in the Build Wheels Linux UCS2 shard, +# so it is not helpful to other shards. Rather, it would make those shards more finicky to deal +# with by resulting in two Python 2.7 installs: system vs. pyenv. + +# Use our custom Centos6 image for binary compatibility with old linux distros. +FROM pantsbuild/centos6:latest + +# Note we use 2.7.15, rather than 2.7.13, as the centos6 image already comes with 2.7.13 +# installed, which uses UCS4 instead of UCS2. This allows us to disambiguate which Python 2 +# interpreter to use when `ci.sh` and `release.sh` set the interpreter constraints for +# Pants, and thus for the built ./pants.pex. We set $PY to the exact Python 2.7 version we want +# to ensure the PEX is bootstrapped with UCS 2. +ARG PYTHON_2_VERSION=2.7.15 +# TODO(7064): remove this yum install line once we update the base Centos6 image to include this dependency. +RUN yum install sqlite-devel -y +ENV PYENV_ROOT /pyenv-docker-build +RUN mkdir ${PYENV_ROOT} +RUN git clone https://github.com/pyenv/pyenv ${PYENV_ROOT} +ENV PYTHON_CONFIGURE_OPTS --enable-unicode=ucs2 +RUN /usr/bin/scl enable devtoolset-7 -- bash -c '\ + ${PYENV_ROOT}/bin/pyenv install ${PYTHON_2_VERSION} \ + && ${PYENV_ROOT}/bin/pyenv global ${PYTHON_2_VERSION}' +ENV PATH "${PYENV_ROOT}/shims:${PATH}" +ENV PY "${PYENV_ROOT}/shims/python2.7" +ENV PEX_PYTHON_PATH "${PYENV_ROOT}/shims/python2.7" + +# Setup mount points for the travis ci user & workdir. +VOLUME /travis/home +VOLUME /travis/workdir + +# Setup a non-root user to execute the build under (avoids problems with npm install). +ARG TRAVIS_USER=travis_ci +ARG TRAVIS_UID=1000 +ARG TRAVIS_GROUP=root +ARG TRAVIS_GID=0 + +RUN groupadd --gid ${TRAVIS_GID} ${TRAVIS_GROUP} || true +RUN useradd -d /travis/home -g ${TRAVIS_GROUP} --uid ${TRAVIS_UID} ${TRAVIS_USER} +USER ${TRAVIS_USER}:${TRAVIS_GROUP} + +# Our newly created user is unlikely to have a sane environment: set a locale at least. +ENV LC_ALL="en_US.UTF-8" + +WORKDIR /travis/workdir diff --git a/build-support/travis/env_osx_with_pyenv.mustache b/build-support/travis/env_osx_with_pyenv.mustache new file mode 100644 index 00000000000..df639a77a9f --- /dev/null +++ b/build-support/travis/env_osx_with_pyenv.mustache @@ -0,0 +1,5 @@ +PATH="/usr/local/opt/openssl/bin:$PATH" +LDFLAGS="-L/usr/local/opt/openssl/lib" +CPPFLAGS="-I/usr/local/opt/openssl/include" +PYENV_ROOT="${HOME}/.pyenv" +PATH="${PYENV_ROOT}/shims:${PATH}" diff --git a/build-support/travis/generate_travis_yml.py b/build-support/travis/generate_travis_yml.py index 0977b2e60b7..a029d2c6166 100644 --- a/build-support/travis/generate_travis_yml.py +++ b/build-support/travis/generate_travis_yml.py @@ -25,12 +25,15 @@ def generate_travis_yml(): """Generates content for a .travis.yml file from templates.""" - template = pkg_resources.resource_string( - __name__, 'travis.yml.mustache').decode('utf-8') - before_install_linux = pkg_resources.resource_string( - __name__, 'before_install_linux.mustache').decode('utf-8') - before_install_osx = pkg_resources.resource_string( - __name__, 'before_install_osx.mustache').decode('utf-8') + def get_mustache_file(file_name): + return pkg_resources.resource_string(__name__, file_name).decode('utf-8') + + template = get_mustache_file('travis.yml.mustache') + before_install_linux = get_mustache_file('before_install_linux.mustache') + before_install_osx = get_mustache_file('before_install_osx.mustache') + env_osx_with_pyenv = get_mustache_file('env_osx_with_pyenv.mustache') + launch_docker_image = get_mustache_file('launch_docker_image.mustache') + context = { 'header': HEADER, 'py3_integration_shards': range(0, num_py3_integration_shards), @@ -42,6 +45,8 @@ def generate_travis_yml(): } renderer = pystache.Renderer(partials={ 'before_install_linux': before_install_linux, - 'before_install_osx': before_install_osx + 'before_install_osx': before_install_osx, + 'env_osx_with_pyenv': env_osx_with_pyenv, + 'launch_docker_image': launch_docker_image }) print(renderer.render(template, context)) diff --git a/build-support/travis/launch_docker_image.mustache b/build-support/travis/launch_docker_image.mustache new file mode 100644 index 00000000000..9a8ce0e1080 --- /dev/null +++ b/build-support/travis/launch_docker_image.mustache @@ -0,0 +1,7 @@ +docker build +--rm -t ${docker_image_name} +--build-arg "TRAVIS_USER=$(id -un)" +--build-arg "TRAVIS_UID=$(id -u)" +--build-arg "TRAVIS_GROUP=$(id -gn)" +--build-arg "TRAVIS_GID=$(id -g)" +build-support/docker/${docker_image_name}/ diff --git a/build-support/travis/travis.yml.mustache b/build-support/travis/travis.yml.mustache index 08d48816df3..b2c1374abe1 100644 --- a/build-support/travis/travis.yml.mustache +++ b/build-support/travis/travis.yml.mustache @@ -156,13 +156,8 @@ py36_osx_config: &py36_osx_config packages: &py36_osx_config_brew_packages - openssl env: - # Fix Python 3 issue linking to OpenSSL - &py36_osx_config_env > - PATH="/usr/local/opt/openssl/bin:$PATH" - LDFLAGS="-L/usr/local/opt/openssl/lib" - CPPFLAGS="-I/usr/local/opt/openssl/include" - PYENV_ROOT="${HOME}/.pyenv" - PATH="${PYENV_ROOT}/shims:${PATH}" + {{>env_osx_with_pyenv}} before_install: {{>before_install_osx}} # Clone pyenv directly from GitHub. For multiple osx images, brew's version of pyenv is too old to get @@ -192,11 +187,7 @@ py36_osx_test_config: &py36_osx_test_config env: # Must duplicate py36_osx_config's env because it cannot be merged into a new anchor - &py36_osx_test_config_env > - PATH="/usr/local/opt/openssl/bin:$PATH" - LDFLAGS="-L/usr/local/opt/openssl/lib" - CPPFLAGS="-I/usr/local/opt/openssl/include" - PYENV_ROOT="${HOME}/.pyenv" - PATH="${PYENV_ROOT}/shims:${PATH}" + {{>env_osx_with_pyenv}} BOOTSTRAPPED_PEX_KEY_SUFFIX=py36.osx linux_with_fuse: &linux_with_fuse @@ -213,14 +204,8 @@ travis_docker_image: &travis_docker_image before_script: - ulimit -c unlimited script: - # NB: This script definition is very likely to be overridden in a consumer, so we alias it - # for re-inclusion. - - &travis_docker_image_launch docker build --rm -t travis_ci - --build-arg "TRAVIS_USER=$(id -un)" - --build-arg "TRAVIS_UID=$(id -u)" - --build-arg "TRAVIS_GROUP=$(id -gn)" - --build-arg "TRAVIS_GID=$(id -g)" - build-support/docker/travis_ci/ + - > + {{>launch_docker_image}} # ------------------------------------------------------------------------- # Bootstrap engine shards @@ -231,14 +216,15 @@ base_linux_build_engine: &base_linux_build_engine <<: *travis_docker_image stage: *bootstrap script: - - *travis_docker_image_launch + - > + {{>launch_docker_image}} # Note that: # * We mount ${HOME} to cache the ${HOME}/.cache/pants/rust-toolchain. # * We also build fs_util, to take advantage of the rust code built during bootstrapping. - docker run --rm -t -v "${HOME}:/travis/home" -v "${TRAVIS_BUILD_DIR}:/travis/workdir" - travis_ci:latest + ${docker_image_name}:latest sh -c "./build-support/bin/ci.sh ${BOOTSTRAP_ARGS} && ./build-support/bin/release.sh -f" - aws --no-sign-request --region us-east-1 s3 cp ${TRAVIS_BUILD_DIR}/pants.pex ${BOOTSTRAPPED_PEX_URL_PREFIX}.${BOOTSTRAPPED_PEX_KEY_SUFFIX} @@ -247,6 +233,7 @@ py27_linux_build_engine: &py27_linux_build_engine <<: *base_linux_build_engine name: "Build Linux native engine and pants.pex (Py2.7 PEX)" env: + - docker_image_name=travis_ci # NB: Only the Py2.7 shard sets PREPARE_DEPLOY to cause the fs_util binary to be uploaded to S3: # either linux shard could upload this binary, since it does not depend on python at all. - PREPARE_DEPLOY=1 @@ -259,6 +246,7 @@ py36_linux_build_engine: &py36_linux_build_engine <<: *base_linux_build_engine name: "Build Linux native engine and pants.pex (Py3.6 PEX)" env: + - docker_image_name=travis_ci - CACHE_NAME=linuxpexbuild.py36 - BOOTSTRAPPED_PEX_KEY_SUFFIX=py36.linux - BOOTSTRAP_ARGS='-b' @@ -360,41 +348,110 @@ cargo_audit: &cargo_audit # Build wheels # ------------------------------------------------------------------------- +# N.B. With Python 2, we must build pantsbuild.pants with both UCS2 and UCS4 to provide full +# compatibility for end users. This is because we constrain our ABI due to the native engine. +# See https://www.python.org/dev/peps/pep-0513/#ucs-2-vs-ucs-4-builds. Note this distinction is +# not necessary with Python 3.3+ due to flexible storage of Unicode strings (https://www.python.org/dev/peps/pep-0393/). +# +# We treat both Linux UCS4 and OSX UCS2 normally, as these are the defaults for those environments. +# The Linux UCS2 and OSX UCS4 shards, however, must rebuild Python with +# `PYTHON_CONFIGURE_OPTS=--enable-unicode=ucs{2,4}` set, along with bootstrapping Pants again rather +# than pulling the PEX from AWS. + base_build_wheels: &base_build_wheels stage: *test env: - - &base_build_wheels_env RUN_PANTS_FROM_PEX=1 PREPARE_DEPLOY=1 + - &base_build_wheels_env PREPARE_DEPLOY=1 -linux_build_wheels: &linux_build_wheels +py27_linux_build_wheels_no_ucs: &py27_linux_build_wheels_no_ucs # Similar to the bootstrap shard, we build Linux wheels in a docker image to maximize # compatibility. This is a Py2.7 shard, so it is not subject to #6985. <<: *travis_docker_image - <<: *py27_linux_test_config <<: *base_build_wheels - name: "Build Linux wheels (No PEX)" + +py27_linux_build_wheels_ucs2: &py27_linux_build_wheels_ucs2 + <<: *py27_linux_config + <<: *py27_linux_build_wheels_no_ucs + <<: *native_engine_cache_config + name: "Build wheels - Linux and cp27m (UCS2)" + env: + - *base_build_wheels_env + - docker_image_name=travis_ci_py27_ucs2 + - CACHE_NAME=linuxwheelsbuild.ucs2 + script: + - > + {{>launch_docker_image}} + - docker run --rm -t + -v "${HOME}:/travis/home" + -v "${TRAVIS_BUILD_DIR}:/travis/workdir" + ${docker_image_name}:latest + sh -c "./build-support/bin/ci.sh -2b + && ./build-support/bin/check_pants_pex_abi.py cp27m + && RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n" + +py27_linux_build_wheels_ucs4: &py27_linux_build_wheels_ucs4 + <<: *py27_linux_build_wheels_no_ucs + <<: *py27_linux_test_config + # `py27_linux_test_config` overrides the stage set by `base_build_wheels`, so we re-override it. + stage: *test + name: "Build wheels - Linux and cp27mu (UCS4)" env: - *py27_linux_test_config_env - *base_build_wheels_env - - CACHE_NAME=linuxwheelsbuild + - docker_image_name=travis_ci + - CACHE_NAME=linuxwheelsbuild.ucs4 script: - - *travis_docker_image_launch + - > + {{>launch_docker_image}} - docker run --rm -t -v "${HOME}:/travis/home" -v "${TRAVIS_BUILD_DIR}:/travis/workdir" - travis_ci:latest - sh -c "RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n" + ${docker_image_name}:latest + sh -c "./build-support/bin/check_pants_pex_abi.py cp27mu + && RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n" -osx_build_wheels: &osx_build_wheels - <<: *py27_osx_test_config +py27_osx_build_wheels_no_ucs: &py27_osx_build_wheels_no_ucs <<: *base_build_wheels - name: "Build OSX wheels (No PEX)" osx_image: xcode8 + +py27_osx_build_wheels_ucs2: &py27_osx_build_wheels_ucs2 + <<: *py27_osx_test_config + <<: *py27_osx_build_wheels_no_ucs + name: "Build wheels - OSX and cp27m (UCS2)" env: - *py27_osx_test_config_env - *base_build_wheels_env - - CACHE_NAME=osxwheelsbuild + - CACHE_NAME=osxwheelsbuild.ucs2 script: - - ./build-support/bin/release.sh -n + - ./build-support/bin/check_pants_pex_abi.py cp27m + - RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n + +py27_osx_build_wheels_ucs4: &py27_osx_build_wheels_ucs4 + <<: *py27_osx_config + <<: *py27_osx_build_wheels_no_ucs + <<: *native_engine_cache_config + name: "Build wheels - OSX and cp27mu (UCS4)" + addons: + brew: + packages: + - openssl + env: + - *base_build_wheels_env + - CACHE_NAME=osxwheelsbuild.ucs4 + - > + {{>env_osx_with_pyenv}} + - PYTHON_CONFIGURE_OPTS=--enable-unicode=ucs4 + # We set $PY to ensure the UCS4 interpreter is used when bootstrapping the PEX. + - PY=${PYENV_ROOT}/shims/python2.7 + before_install: + {{>before_install_osx}} + - git clone https://github.com/pyenv/pyenv ${PYENV_ROOT} + - ${PYENV_ROOT}/bin/pyenv install 2.7.13 + - ${PYENV_ROOT}/bin/pyenv global 2.7.13 + script: + - ./build-support/bin/ci.sh -2b + - ./build-support/bin/check_pants_pex_abi.py cp27mu + - RUN_PANTS_FROM_PEX=1 ./build-support/bin/release.sh -n # ------------------------------------------------------------------------- # Rust tests @@ -586,7 +643,6 @@ matrix: include: - <<: *py27_linux_build_engine - <<: *py36_linux_build_engine - - <<: *py27_osx_build_engine - <<: *py36_osx_build_engine @@ -596,9 +652,6 @@ matrix: - <<: *linux_rust_clippy - <<: *cargo_audit - - <<: *linux_build_wheels - - <<: *osx_build_wheels - - <<: *py27_linux_test_config name: "Unit tests for pants and pants-plugins (Py2.7 PEX)" stage: *test @@ -616,6 +669,11 @@ matrix: script: - ./build-support/bin/travis-ci.sh -lp + - <<: *py27_linux_build_wheels_ucs2 + - <<: *py27_linux_build_wheels_ucs4 + - <<: *py27_osx_build_wheels_ucs2 + - <<: *py27_osx_build_wheels_ucs4 + {{#py3_integration_shards}} - <<: *py36_linux_test_config name: "Integration tests for pants - shard {{.}} (Py3.6 PEX)" diff --git a/src/python/pants/BUILD b/src/python/pants/BUILD index 06d97891250..8cff8c232ab 100644 --- a/src/python/pants/BUILD +++ b/src/python/pants/BUILD @@ -10,7 +10,11 @@ target( python_library( name='pants-packaged', - sources=[], + # NB: we must include at least one file in `sources` to avoid clang/gcc complaining + # `error: no input files`. We don't actually need to put any meaningful files here, + # though, because we use `with_binaries()` to link to the actual native code, + # so clang/gcc do not need to build any native code. Instead, we pass a dummy file. + sources=['dummy.c'], dependencies=[ ':version', ], @@ -18,6 +22,14 @@ python_library( name='pantsbuild.pants', description='A scalable build tool for large, complex, heterogeneous repos.', namespace_packages=['pants', 'pants.backend'], + # NB: by setting `ext_modules`, we signal to setup_py and bdist_wheel that this library + # has native code. As a consquence, bdist_wheel pins the ABI (application binary interface) + # used when creating the wheel, which is a good thing. We should be setting this ABI to ensure + # consumers of pantsbuild.pants are using a compatible interpreter, for example + # that we correctly distinguish when using Python 2 between interpreters built with UCS2 vs. UCS4 + # (see https://www.python.org/dev/peps/pep-0513/#ucs-2-vs-ucs-4-builds). + # TODO(7344): the tuple syntax for ext_modules is deprecated. Use Extension once we support it. + ext_modules=[('native_engine', {'sources': ['src/pants/dummy.c']})], ).with_binaries( pants='src/python/pants/bin:pants', ) diff --git a/src/python/pants/dummy.c b/src/python/pants/dummy.c new file mode 100644 index 00000000000..e69de29bb2d