Add support for running with Python 3.6 and 3.7 #32

Eric-Arellano · 2019-03-12T08:19:00Z

Problem

Now that we're releasing Pants with Python 3 wheels (pantsbuild/pants#7197, overall migration tracked by pantsbuild/pants#6062), we need to add a way for every day Pants users to consume the upgrade.

See #30 for more context, including the design goals.

Solution

Parse the pants_runtime_python_version from pants.ini to determine which Python interpreter to use. If it's missing, default to Python 2.7.

This requires modifying the venv folders to support multiple Python versions. Rather than using the naming scheme ${pants_version}, we now consistently use ${pants_version}_py{py_major_minor}, regardless of if the user is using pants_runtime_python_version.

It also requires upgrading virtualenv to avoid a deprecation warning when running with Python 3.

Finally, we add tests to ensure this all works.

Result

Running ./pants like normal for the first time will bootstrap Pants again and create the new folder 1.15.dev3_py{27,36,37}.

Changing the $PYTHON env var or the pants_runtime_python_version in pants.ini will change which Python version is used to run Pants.

When changing interpreters for the first time, it will bootstrap Pants again. After that, you can change without issue.

Rather than using a sentinal value, read in the pants.ini value for `pants_engine_python_version`. Default to Py27 because that's what we've always used and most people will have this.

We first need to land pantsbuild/pants#7363 and then release Pants with it or else the option will error out.

…ound in pants.ini For some reason it was exiting the script?! I have no idea why, because the same code works for parsing the pants_version. This does the same thing and has the benefit of using way less lines.

commit e8cf735 Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 09:36:16 2019 -0700 Trusty Py34 is from system, not pyenv commit 1f7d236 Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 09:30:08 2019 -0700 OSX requires language: general commit 4941956 Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 09:23:28 2019 -0700 Move constant to top of file Think this is the style used in most Python projects and ours. commit c478ecb Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 09:19:29 2019 -0700 Set Pyenv for Trusty commit 30aab56 Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 09:16:58 2019 -0700 Try pyenv approach Path wrangling didn't work. commit 9b9e6be Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 09:12:11 2019 -0700 Global python entry -> build matrix See if moving back to osx_setup and linux_setup works. commit 0dc87dc Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 09:09:18 2019 -0700 Maybe fix Trusty using Py3.4 commit aaab4bf Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 09:05:50 2019 -0700 Test if OSX comes pre-installed with Python? commit 96cb6ac Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 09:02:44 2019 -0700 Remove unneeded anchors from OSX shards Turns out Precise comes installed with 3.6 when requested! So we only need Pyenv for OSX, not Linux. commit 4a312a4 Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 09:01:31 2019 -0700 Try to debug Trusty image using Py3.4 commit d511d99 Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 08:48:28 2019 -0700 Remove stray whitespace commit 126ae2f Merge: cd28677 4aee51a Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 08:47:56 2019 -0700 Merge branch 'gh-pages' of github.com:pantsbuild/setup into ci-py commit 4aee51a Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 08:46:38 2019 -0700 Test Ubuntu Xenial (pantsbuild#31) Travis added Xenial back in November. See https://blog.travis-ci.com/2018-11-08-xenial-release. Adding this test results in better coverage. commit cd28677 Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 08:36:55 2019 -0700 Add prefix to script. Oops commit 4afcdcf Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 08:24:43 2019 -0700 Add comment explaining why we rewrite pants.ini like this commit ef5bf71 Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 08:22:55 2019 -0700 Tweak ci.py formatting commit 599ae46 Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 08:19:38 2019 -0700 Try to install Py36 on each shard Also modernize .travis.yml to use all the cool goodies we've learned in Pants's CI like anchors and the brew addon package! commit 717271f Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 07:50:10 2019 -0700 Remove PythonVersion and fix --pants-version unspecified --pants-version unspecified would fail because of a bad config, that the contrib module would refer to pants_version even though that line was deleted. commit d1f9f0f Author: Eric Arellano <ericarellano@me.com> Date: Tue Mar 12 07:40:38 2019 -0700 Add ci.py to CI to allow parameterizing pants_version and python_version

Will look like this: 1.14.dev1_py36.

We won't be able to resolve Py3 from PyPi until the release anyway, so we should setup the test correctly from the start. Once the Pypi release happens, we can rebase and get green CI

Easier on the eyes.

We'll need this version to get CI passing so that it can consume Py3 wheels and parse pants_engine_python_version.

### Problem The `./pants` script claims to support running with both a pinned `pants_version` in the `pants.ini` and with the `pants_version` unspecified. We currently do not test this unspecified branch, which we should be doing to guarantee that functionality always works. Further, this is pre-work for #32, which will add `--python-version` as another parameter for `ci.py` to allow an unspecified Python version vs. `pants.ini` specifying 2.7 or 3.6 (FYI resulting [`.travis.yml`](https://github.com/pantsbuild/setup/pull/32/files#diff-354f30a63fb0907d4ad57269548329e3)). ### Solution Introduce a new `ci.py` helper script that uses argparse to have the parameter `--pants-version {unspecified,config}`. #### Requires adding Python 3 to CI shards Because this script uses modern Python 3—i.e. enums, f-strings, type hints, keyword-only args, and `subprocess.run()`—it requires Python 3.5+ to run. Even though we can't yet run `./pants` with Python 3 until #32 lands, this PR does some of the pre-work in adding Python 3 to each of the shards so that we can run this script. This is valid to do because every single shard is going to run the tests with both Py27 and Py36—unlike Pants where we have dedicated shards for each interpreter—so there is only benefit in ensuring each shard has Py27 and Py36. #### Also updates test's pinned `pants_version`

Eric-Arellano · 2019-03-13T00:49:50Z

Reviewers: CI is going to fail on this one until we release 1.15.0.dev4, because we need the Py3 wheels and pantsbuild/pants#7363 to parse pants_engine_python_version in pants.ini.

Stu and I came up with a strategy to (hopefully) still land this change before sending the release email to pants-devel. Right after we finish the release to PyPi, I'll update the pants.ini to point to the new version and run CI. If all goes green, we'll merge this PR and send the email to pants-devel about how to consume this upgrade. If CI fails, we simply push off the announced release one week and go with a silent release.

stuhood

Thanks. One blocker related to the option name, but the rest looks reasonable.

ci.py

pants

Copy idiom we use in Pants' `virtualenv` file.

Usually I don't like nested conditionals, but it reads better here.

We try to read in $PYTHON, which we store into $PYTHON_BIN_NAME. The important part is we check for the env var $PYTHON.

stuhood

Looks good: thanks!

pants

Thanks Stu!

Same fix we made in pantsbuild#36 for setting up pants versions. Forgot to also apply it to setup python version.

illicitonion

Works for me, though I don't know really anything about the setup repo :)

ci.py

Daniel pointed out we don't need this. We not only don't need it, but we shouldn't have it. No matter what, we need to do the cleanup, regardless of the exception caught.

Do not repeat yourself. Makes it a bit more clear also why we need to read original config and have the try finally.

pants

cosmicexplorer · 2019-03-15T22:19:34Z

ci.py

@@ -22,9 +22,19 @@ def __str__(self):
      return self.value


+class PythonVersion(Enum):


Love enums!!

Sorry for accidently pushing!

No longer hardcoded to Python.

cosmicexplorer

Thanks!!

We gotta get Pylint in this repo. Python only complains about unknown values once it's evaluated.

Running ./pants with Py3.6 or 3.7 would result in this error: ``` ./pants WARN] /Users/eric/.cache/pants/setup/bootstrap-Darwin-x86_64/1.15.0.dev4_py36/lib/python3.6/site.py:165: DeprecationWarning: 'U' mode is deprecated f = open(fullname, "rU") WARN] /Users/eric/.cache/pants/setup/bootstrap-Darwin-x86_64/1.15.0.dev4_py36/lib/python3.6/site.py:165: DeprecationWarning: 'U' mode is deprecated f = open(fullname, "rU") No goals specified. Use `pants goals` to list goals. Use `pants help` to get help. ``` To fix this, we must upgrade to 16.4.3, as the deprecation warning was fixed between 16.0.0 and this release.

I think the issue is that the wrong pip is being used? This looks like a Pip issue rather than twitter commons.

This reverts commit fa1f224. Still fails :/ but passes locally.

commit 4799cfa Author: Eric Arellano <ericarellano@me.com> Date: Fri Mar 15 23:59:37 2019 -0700 Remove pinned setuptools

commit c738904 Author: Eric Arellano <ericarellano@me.com> Date: Sat Mar 16 01:11:30 2019 -0700 Remove bad leftover line commit 4799cfa Author: Eric Arellano <ericarellano@me.com> Date: Fri Mar 15 23:59:37 2019 -0700 Remove pinned setuptools

### Problem Due to pantsbuild/pants#6714, and maybe pantsbuild/pants#7323, the OSX 12 shard frequently flakes when Pantsd is enabled, as seen [here](https://travis-ci.org/pantsbuild/setup/jobs/507257503#L432), [here](https://travis-ci.org/pantsbuild/setup/jobs/507238744#L545), and [here](https://travis-ci.org/pantsbuild/setup/jobs/507238744#L681). This blocks #32, as that PR will result in each shard running with Pantsd a total of 6 more times, which results in an almost certain guarantee at least one shard will flake. ### Solution Temporarily skip pantsd tests for OSX 10.12. We do not completely remove the shard, as the first two tests still work and are useful to include.

Eric-Arellano added 5 commits March 11, 2019 18:19

Parse targeted Python version via pants.ini value

4be519a

Rather than using a sentinal value, read in the pants.ini value for `pants_engine_python_version`. Default to Py27 because that's what we've always used and most people will have this.

Update docs to mention pinning Python version

5f9c5d3

Remove pants_engine_python_version from pants.ini

2dcdcfc

We first need to land pantsbuild/pants#7363 and then release Pants with it or else the option will error out.

Fix interpreter_version failing script when interpreter_version not f…

3b1c763

…ound in pants.ini For some reason it was exiting the script?! I have no idea why, because the same code works for parsing the pants_version. This does the same thing and has the benefit of using way less lines.

Add ci.py to CI to allow parameterizing pants_version and python_version

5531a2b

Eric-Arellano mentioned this pull request Mar 12, 2019

Add ci.py helper script to allow parametrizing the pants_version #33

Merged

Eric-Arellano mentioned this pull request Mar 12, 2019

Update docs to explain how to use Python 3 pantsbuild/pants#7368

Closed

Eric-Arellano added 4 commits March 12, 2019 14:02

Implement naming scheme

65cae1e

Will look like this: 1.14.dev1_py36.

Update interpreter selection test to modify pants.ini

481a2d3

We won't be able to resolve Py3 from PyPi until the release anyway, so we should setup the test correctly from the start. Once the Pypi release happens, we can rebase and get green CI

Rename venv folders to py27 etc., not py2.7

28f3c41

Easier on the eyes.

Upgrade version to 1.15.0.dev4

a9d1f77

We'll need this version to get CI passing so that it can consume Py3 wheels and parse pants_engine_python_version.

Merge branch 'gh-pages' of github.com:pantsbuild/setup into py3

37bdbde

Eric-Arellano marked this pull request as ready for review March 13, 2019 00:42

Eric-Arellano requested review from stuhood, jsirois, illicitonion, cosmicexplorer, benjyw and mateor March 13, 2019 00:45

stuhood approved these changes Mar 14, 2019

View reviewed changes

ci.py Outdated Show resolved Hide resolved

ci.py Show resolved Hide resolved

pants Outdated Show resolved Hide resolved

Eric-Arellano added 7 commits March 13, 2019 17:24

Rename to pants_runtime_python_version

30321a1

Refactor ci.py

a2ef6e2

Merge branch 'gh-pages' of github.com:pantsbuild/setup into py3

dc7257c

Check interpreter is valid

3382787

Copy idiom we use in Pants' `virtualenv` file.

Fix not using keyword arg

b1fd519

Remove stry pdb import

ed937fa

Nest conditionals in write_config_entry

21ad22f

Usually I don't like nested conditionals, but it reads better here.

Clarify in high level flow the env var name $PYTHON

ec4e5ae

We try to read in $PYTHON, which we store into $PYTHON_BIN_NAME. The important part is we check for the env var $PYTHON.

stuhood approved these changes Mar 14, 2019

View reviewed changes

pants Outdated Show resolved Hide resolved

Eric-Arellano added 3 commits March 14, 2019 16:33

Fix undefined name from bad merge

f388027

Thanks Stu!

Remove bad \ from .travis.yml script entry

9a75f9f

Restore pants.ini if command fails and --python-version changed

03b13d3

Same fix we made in pantsbuild#36 for setting up pants versions. Forgot to also apply it to setup python version.

illicitonion approved these changes Mar 15, 2019

View reviewed changes

ci.py Outdated Show resolved Hide resolved

Eric-Arellano added 2 commits March 15, 2019 09:50

Remove unnecessary except clause

f231cfd

Daniel pointed out we don't need this. We not only don't need it, but we shouldn't have it. No matter what, we need to do the cleanup, regardless of the exception caught.

Extract temporarily_rewrite_config() context manager

4731d58

Do not repeat yourself. Makes it a bit more clear also why we need to read original config and have the try finally.

cosmicexplorer reviewed Mar 15, 2019

View reviewed changes

Eric-Arellano added 4 commits March 15, 2019 15:43

Use get_exe_path_or_die per Danny's suggestion

0c128f8

Revert bad setting of '-x'

fbbde0b

Sorry for accidently pushing!

Rename local variables to reflect new function name

0b9ee0e

No longer hardcoded to Python.

Remove stray white line

1b5a6bc

cosmicexplorer approved these changes Mar 15, 2019

View reviewed changes

Eric-Arellano added 8 commits March 15, 2019 15:59

Fix unnamed value in ci.py

cac7e02

We gotta get Pylint in this repo. Python only complains about unknown values once it's evaluated.

Upgrade to 1.15.0.dev4!

be6f731

See if python -m ensurepip works

fa1f224

I think the issue is that the wrong pip is being used? This looks like a Pip issue rather than twitter commons.

Revert "See if python -m ensurepip works"

741a238

This reverts commit fa1f224. Still fails :/ but passes locally.

Squashed commit of the following:

e362d8e

commit 4799cfa Author: Eric Arellano <ericarellano@me.com> Date: Fri Mar 15 23:59:37 2019 -0700 Remove pinned setuptools

Squashed commit of the following:

b9d4b25

commit c738904 Author: Eric Arellano <ericarellano@me.com> Date: Sat Mar 16 01:11:30 2019 -0700 Remove bad leftover line commit 4799cfa Author: Eric Arellano <ericarellano@me.com> Date: Fri Mar 15 23:59:37 2019 -0700 Remove pinned setuptools

Merge branch 'gh-pages' of github.com:pantsbuild/setup into py3

d79a553

Eric-Arellano mentioned this pull request Mar 16, 2019

Skip flaky Pantsd tests for OSX 10.12 shard #40

Merged

Eric-Arellano added 3 commits March 16, 2019 14:57

Merge branch 'gh-pages' of github.com:pantsbuild/setup into py3

9a1800e

Fix bad merge leaving off --python-version arg

215ea00

Remove bad __pycache__ from wip that made it into here

4322fa2

Eric-Arellano merged commit 416ab35 into pantsbuild:gh-pages Mar 16, 2019

Eric-Arellano deleted the py3 branch March 16, 2019 22:40

Eric-Arellano mentioned this pull request Mar 19, 2019

Add support for Python 3 #30

Closed

3 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for running with Python 3.6 and 3.7 #32

Add support for running with Python 3.6 and 3.7 #32

Eric-Arellano commented Mar 12, 2019 •

edited

Loading

Eric-Arellano commented Mar 13, 2019

stuhood left a comment

stuhood left a comment

illicitonion left a comment

cosmicexplorer Mar 15, 2019

cosmicexplorer left a comment

		@@ -22,9 +22,19 @@ def __str__(self):
		return self.value


		class PythonVersion(Enum):

Add support for running with Python 3.6 and 3.7 #32

Add support for running with Python 3.6 and 3.7 #32

Conversation

Eric-Arellano commented Mar 12, 2019 • edited Loading

Problem

Solution

Result

Eric-Arellano commented Mar 13, 2019

stuhood left a comment

Choose a reason for hiding this comment

stuhood left a comment

Choose a reason for hiding this comment

illicitonion left a comment

Choose a reason for hiding this comment

cosmicexplorer Mar 15, 2019

Choose a reason for hiding this comment

cosmicexplorer left a comment

Choose a reason for hiding this comment

Eric-Arellano commented Mar 12, 2019 •

edited

Loading