Add support for Python 3

[Eric-Arellano]

With Pants soon to be released with Python 3 wheels pantsbuild/pants#7197 and the Python 3 migration nearing completion pantsbuild/pants#6062, we need to modify this setup script to allow Pants end users to consume the upgrade.

Goals for an ideal solution

1. Decision of which interpreter to use applies to the whole organization.
   • Typically, an individual or team will be responsible for upgrading Pants. They should be the only ones to have to know this is even a decision. Every day users should not be concerned with this upgrade.
   • Thus, solution must be tracked by VCS.
   • Solution should not require something like each individual modifying their .bashrc with their individual preference. It should be an organization-level decision set for the whole repo.
2. Transparent what is being used.
   • We certainly shouldn't print out to the console every time or anything. But we should make it possible to find, e.g. in a config file.
   • Will help with debugging.
3. Possible to change the interpreter version.
   • For example, deciding to upgrade from 2.7 -> 3.6 or 3.6 -> 3.7.
   • This decision isn't expected to be very frequent, but it will sometimes happen.
   • When a change happens, must ensure that no one has to do anything beyond the admin. For example, we can't expect people to have to rm -rf their cache.
   • Non-goal: make it extremely easy to change back-and-forth between interpreter versions, a la ./pants2 and ./pants. We are not expecting the average Pants user to worry about which Python interpreter they use to run Pants as both should work equally as well and it is more cognitive load than necessary.
4. Simple to make the first-time decision.
   • Clearly explain what the options are.
   • Don't overwhelm with too intricate of details.
   • Emphasize Py3 will soon be required for runtime.
   • Maybe interactive?
5. Backwards compatible.
   • If the user doesn't change their behavior at all, Pants should continue working as before, i.e. continue using Python 2.7.
   • Note this doesn't mean that we can't use the new venv naming scheme. That's okay. Just Pants needs to keep working when the user doesn't change their behavior from before.

Proposed solution

Users should add pants_runtime_python_version: {2.7,3.6,3.7} to their pants.ini [GLOBAL] entry, just as they set pants_version.

We would update the website at https://www.pantsbuild.org/install.html to provide instructions, along the lines of:

Pants may be ran with Python 2.7, 3.6, or 3.7. This does not mean your code has to use this Python
version, as Pants will spawn subprocesses to run with the interpreter you want to use for your own
codebase; rather, this is what Pants uses for its engine.

While Pants defaults to Python 2.7, we recommend pinning the interpreter you prefer in `pants.ini` by
adding an entry like this:

``ini
[GLOBAL]
pants_version: 1.15.0
pants_runtime_python_version: 3.6
``

Any time you want to change the interpreter Pants uses, update this value in your `pants.ini`.

Some notes when choosing an interpreter:
- Ensure you have this Python version accessible on your PATH everywhere Pants is used in your organization.
- Pants will require Python 3.6+ to run as of release 1.17.0.

Tasks

• Determine naming scheme for the venv folder.
  • Must specify the Py version so that changing interpreters does not require removing the cache.
  • Delete the legacy name folders? I think so, if they happened to already have a folder for that version.
• Read in the value from pants.ini. Use this to set $PYTHON and determine bootstrap_venv.
• Add testing.

[cosmicexplorer]

For background, pantsbuild/pants#7151 is where we have previously used the .pants-python-version convention.

[cosmicexplorer]

And this sounds like a plan!

[Eric-Arellano]

FYI the proposal has changed from the original one of using a .pants-python-version sentinel file to instead read in the value from pants.ini, similar to how we resolve pants_version. See the edit history of the PR description and look for the entry on March 11, 2019 if you'd like to see the original proposal.

[cosmicexplorer]

For the record, we could definitely turn:

setup/pants

Lines 69 to 72 in 6dfccab

	pants_version=$(
	grep -E "^[[:space:]]*pants_version" pants.ini 2>/dev/null | \
	cut -f2 -d: | tr -d " "
	)

into:

 pants_version=$(
    # use extended regexps, don't print every line
    # when a `pants_version: ` line is found,
    # replace all the text from the start of the line up to `: ` with an empty string, and print
    sed -rne '/\s*pants_version: / s#^.*: ##p' pants.ini 2>/dev/null
    # could add `| head -n1` if there might be multiple such lines
  )

which I personally find easier to read.

[Eric-Arellano]

This was closed by #32.