Accept Python 3 when NODE_GYP_PYTHON3 is defined #1815

joaocgreis · 2019-07-08T22:50:50Z

Checklist

npm install && npm test passes
tests are included
commit message follows commit guidelines

Description of change

~~This depends on #1812, I'll rebase when that PR lands.~~

This allows for Python 3 to be tested without breaking node-gyp for users.

This uses two new environment variables: NODE_GYP_FORCE_PYTHON is the only Python binary checked when it is defined (so we can be sure of what version is being tested in Travis), and NODE_GYP_PYTHON3 conditionally enables Python 3.

When Python 3 is fully supported, enabling it unconditionally should be as simple as removing NODE_GYP_PYTHON3, which is only used on one line.

This includes two commits. One just moves Python detection to a new file. The other contains the rest of the changes. Separately, they should be easy to review.

The tests that check for a "Python version too new" were deleted. Hopefully we won't need them again.

joaocgreis · 2019-07-08T22:55:01Z

CI: https://ci.nodejs.org/view/All/job/nodegyp-test-pull-request/141/

rvagg · 2019-07-09T00:17:34Z

When Python 3 is fully supported

I thought we were there already? Don't we have Travis tests for this? This python garbage is doing my head in and I wish we had more actual python people working on this or at least helping review. The sooner we get to gyp.js the better.

joaocgreis · 2019-07-09T00:34:44Z

I thought so too. Node-gyp was doing what it's supposed to do, find a compatible version of Python somewhere and run it. The tests don't show info output so we had no confirmation of version. All the Linux machines have Python 2 installed, so the tests pass with no errors. This shows that when actually using Python 3 the tests don't pass, but I don't know how hard those issues are to fix.

richardlau · 2019-07-09T01:00:14Z

Thia shows that when actually using Python 3 the tests don't pass, but I don't know how hard those issues are to fix.

At least https://travis-ci.com/nodejs/node-gyp/jobs/214237690#L2234-L2241

    +Traceback (most recent call last):                                         
    +  File "/home/travis/build/nodejs/node-gyp/gyp/gyp_main.py", line 13, in <module>
    +    import gyp                                                             
    +  File "/home/travis/build/nodejs/node-gyp/gyp/pylib/gyp/__init__.py", line 10, in <module>
    +    import gyp.input                                                       
    +  File "/home/travis/build/nodejs/node-gyp/gyp/pylib/gyp/input.py", line 7, in <module>
    +    from compiler.ast import Const                                         
    +ModuleNotFoundError: No module named 'compiler'

looks like it would be fixed by refack/GYP3@f989ef9.

cclauss · 2019-07-09T03:31:34Z

.travis.yml

@@ -4,32 +4,61 @@ cache: pip
 matrix:
  include:
    - name: "Python 2.7 on Linux"
+      env: NODE_GYP_FORCE_PYTHON=python2


~~On Linux runs Travis will set the environment variable $TRAVIS_PYTHON_VERSION == 2.7 so perhaps that it would be better to leverage that rather than create a new environment variable.~~

As I think about it more, I now understand that you are looking for a more general solution that is useful well beyond just Travis CI. What if we name the variable NODE_GYP_PYTHON2 and set it to True in legacy Python environments and set it to False or leave it undefined in Python 3 environments? In this way we would not need to carry this baggage into future years. With just 176 days left before Python 2 end of life we should be making execution on Python 3 the default behavior and making running on Python 2 be the special case.

cclauss

Awesome. Thanks for doing this.

cclauss

NODE_GYP_PYTHON3 --> NODE_GYP_PYTHON2

joaocgreis · 2019-07-09T14:08:42Z

@cclauss I don't understand which variable you'd like to change. Changing NODE_GYP_FORCE_PYTHON would break the point of having it in the first place, and changing NODE_GYP_PYTHON3 would break node-gyp for end users. My goal with these variables is:

NODE_GYP_FORCE_PYTHON: To have a way of forcing node-gyp to use a certain Python binary and not trying anything else. This is useful in Travis to specify a Python binary to use. I understand this adds some lines to .travis.yml, but when we're sure there is no other Python version in the machines we can remove this (hopefully next year Python 2 will start to disappear). I also plan to use this to add a test where python is a bat file, which is common in Windows. I think this is a good mechanism to have, even in the future when there is only Python 3.
NODE_GYP_PYTHON3: Python 3 needs to be opt-in for now, because it doesn't work. If we make this NODE_GYP_PYTHON2, then every user who wants to use node-gyp would have to set this before running! I purposefully used this variable only in one line:

node-gyp/lib/find-python.js

Line 21 in 7bcc33b

semverRange: process.env.NODE_GYP_PYTHON3 ? '>=2.7.0' : '>=2.6.0 <3.0.0',

So, when Python 3 works we can simply change that line to semverRange: '>=2.7.0', remove the variable from .travis.yml and be done with it.

cclauss · 2019-07-09T15:27:57Z

I like the idea behind NODE_GYP_FORCE_PYTHON. It will not be needed on Travis CI Linux because those jobs run in virtualenvs where by definition python points to the version of Python which is specified in the python: tag. On Travis CI Mac, NODE_GYP_FORCE_PYTHON plays a very useful role because both python2 and and python3 will be present and you know the Windows story better than I do.

I would like to resist the NODE_GYP_PYTHON3 variable unless we have an agreed upon method of deprecating it. My sense is that once we introduce new configuration vocabulary, it is nearly impossible to drop it. For example, our README.md clearly states that we no longer support Python 2.6 yet contributors continue to try to push it back into our scripts.

As I wrote above, Python 3 should the default and Python 2 should be the opt-in. Python 3 has been available for 8+ years and is the present and future of the language. Python 2 is legacy and is no longer supported at yearend. Therefore, I would much rather see us introduce a variable called NODE_GYP_PYTHON2 and force those who wish to remain on legacy Python to set that variable. I advocate that continuing to rely on a soon-to-be-EOLed platform should to be a deliberate, conscious, and documented choice.

cclauss · 2019-07-09T15:30:28Z

lib/find-python.js

+  log: logWithPrefix(log, 'find Python'),
+  argsExecutable: [ '-c', 'import sys; print(sys.executable);' ],
+  argsVersion: [ '-c', 'import sys; print("%s.%s.%s" % sys.version_info[:3]);' ],
+  semverRange: process.env.NODE_GYP_PYTHON3 ? '>=2.7.0' : '>=2.6.0 <3.0.0',


Please remove v2.6.0 to align with this project's README.md.

For this project:

Python2 is >=2.7.0 <3.0.0

Python3 is >=3.5.0

This is only to extend the range of acceptable versions, not select which version to use.

joaocgreis · 2019-07-09T16:13:53Z

@cclauss I might be missing something here, but I don't understand where you're trying to get to. Node-gyp does not work with Python 3, as Travis shows. By making Python 3 the default, node-gyp will be broken by default for every user. How is that acceptable?

cclauss · 2019-07-09T16:22:32Z

This PR is supposed to make node-gyp Python 3 compatible. We all already thought that it was with all the tests passing until you demonstrated to us that find_python.js was actually silently preventing us from finding Python. That script was hardcoded to legacy which is exactly want I would like us to avoid continuing in this PR. I have been on this mission for a year now... I also worked on https://github.com/refack/GYP which was more advanced in Python 3 support but unfortunately was not adopted.

joaocgreis · 2019-07-09T17:16:09Z

This PR does not make node-gyp Python 3 compatible, it only allows testing with it.

find-python.js is doing what it's supposed to do, produces a lot of logging at several levels and clearly indicates what version of Python it chose and why. If you run node-gyp locally on any project, you'll see that output. I also added it to Travis here.

The only test that actually runs node-gyp end-to-end is test-addon.js. Those tests are not supposed to have any output, so they're not suitable to test this kind of change. With this PR we can explicitly select an executable to run, which should allow us to reliably test different versions of Python from now on.

@cclauss it's great that you can put some effort into making node-gyp Python 3 compatible, thank you for moving that forward. I can't look into it myself, but I think this is a very important goal and that's why I'm trying to help where I can. With all changes that happened so far, I was not finding it surprising that node-gyp was already compatible until I started looking into the Windows failures. I hope that even if this is a bit disappointing, there won't be much left to do.

This allows us to start testing Python 3 without breaking node-gyp for users. This also adds support for NODE_GYP_FORCE_PYTHON, which will be the only Python binary acceptable when defined.

cclauss · 2019-07-09T20:21:25Z

The ModuleNotFoundError: No module named 'compiler' errors are caused by the fact that the compiler package was removed in Python 3 in favor of the ast module that exists in both Python 2 and Python 3.

refack/GYP3@f989ef9

joaocgreis · 2019-07-09T20:29:17Z

I renamed NODE_GYP_PYTHON3 to EXPERIMENTAL_NODE_GYP_PYTHON3, I hope this helps it be clear that the variable is not to be used by end users, only for testing. Also excluded outdated versions 3.0-3.4 of Python if it is used.

@cclauss when this lands (or even before by depending on this), fixing that error should require only picking those changes and adapting for any possible difference.

cclauss · 2019-07-10T09:43:34Z

Thanks for our WebEx on this topic and the resulting changes that you have made. If you can fix the Travis CI Py2 issues then I will give this PR a LGTM.

It is a bummer that our Py3 tests now fail but at least with PR we are able to run the desired version of Python. We can leave the Py3 Travis tests in allow_failures mode in this PR and then we can work to make those tests green in future PRs.

joaocgreis · 2019-07-10T13:29:40Z

@cclauss fixed, thanks!

CI: https://ci.nodejs.org/view/All/job/nodegyp-test-pull-request/142/

cclauss

Nice work here! Thanks.

PR-URL: #1815 Reviewed-By: Christian Clauss <cclauss@me.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

This allows us to start testing Python 3 without breaking node-gyp for users. This also adds support for NODE_GYP_FORCE_PYTHON, which will be the only Python binary acceptable when defined. PR-URL: #1815 Reviewed-By: Christian Clauss <cclauss@me.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

joaocgreis · 2019-07-11T04:23:15Z

Landed in bb92c76...66ad305

cclauss · 2019-07-11T06:06:35Z

Why have we created a new NODE_GYP_FORCE_PYTHON environment variable when there is already a well documented gyp --python option? Do they do the same thing or different things? Which one takes precedence?

joaocgreis · 2019-07-11T16:08:52Z

The --python option is only a hint, it will be used if it can be used. If it can't be used, node-gyp will look for something else. I can elaborate on the rationale of how this works a bit:

Node-gyp also checks the PYTHON environment variable. As I understand it, this variable is quite common and might be defined for some other project that users are working on, or might even be a forgotten variable from some past project that was never undefined. Therefore, I don't think it makes sense for node-gyp to fail immediately if it points to an invalid version of Python, it should keep looking for a Python of the right version.
The --python option is the same as defining NPM_CONFIG_PYTHON or setting python in npm config. This is a harder case to make, but I think the same logic should (and currently does) apply here. We have plenty of users who are not familiar with node-gyp, and at some point in the past might have set the npm configuration for some reason (actually, we still recommend this in the README, and shouldn't remove until v5.0.0 becomes the default). If this Python is of the versions that we accept, we use it. If not, just look for something else. I think most users want node-gyp to work and not have to worry about this.
The NODE_GYP_FORCE_PYTHON is not a hint. It's either this or nothing. This variable is only used for node-gyp.

Note that the verbose log output shows clearly what is used, what can't be used, by what order and why.

Precedence works like this:

If NODE_GYP_FORCE_PYTHON is defined, check it's version and use it, or fail. Stop here, don't look for anything else.
Check --python/NPM_CONFIG_PYTHON.
Check PYTHON.
Look for a python in the PATH.
Look for a python2 in the PATH.
Try to use the py launcher on Windows.
Try the default locations on Windows.

The code is here:

node-gyp/lib/find-python.js

Lines 47 to 125 in 66ad305

    
           function getChecks () { 
        
             if (this.env.NODE_GYP_FORCE_PYTHON) { 
        
               return [ { 
        
                 before: () => { 
        
                   this.addLog( 
        
                     'checking Python explicitly set from NODE_GYP_FORCE_PYTHON') 
        
                   this.addLog('- process.env.NODE_GYP_FORCE_PYTHON is ' + 
        
                     `"${this.env.NODE_GYP_FORCE_PYTHON}"`) 
        
                 }, 
        
                 check: this.checkCommand, 
        
                 arg: this.env.NODE_GYP_FORCE_PYTHON 
        
               } ] 
        
             } 
        
             var checks = [ 
        
               { 
        
                 before: () => { 
        
                   if (!this.configPython) { 
        
                     this.addLog( 
        
                       'Python is not set from command line or npm configuration') 
        
                     return SKIP 
        
                   } 
        
                   this.addLog('checking Python explicitly set from command line or ' + 
        
                     'npm configuration') 
        
                   this.addLog('- "--python=" or "npm config get python" is ' + 
        
                     `"${this.configPython}"`) 
        
                 }, 
        
                 check: this.checkCommand, 
        
                 arg: this.configPython 
        
               }, 
        
               { 
        
                 before: () => { 
        
                   if (!this.env.PYTHON) { 
        
                     this.addLog('Python is not set from environment variable ' + 
        
                       'PYTHON') 
        
                     return SKIP 
        
                   } 
        
                   this.addLog('checking Python explicitly set from environment ' + 
        
                     'variable PYTHON') 
        
                   this.addLog(`- process.env.PYTHON is "${this.env.PYTHON}"`) 
        
                 }, 
        
                 check: this.checkCommand, 
        
                 arg: this.env.PYTHON 
        
               }, 
        
               { 
        
                 before: () => { this.addLog('checking if "python" can be used') }, 
        
                 check: this.checkCommand, 
        
                 arg: 'python' 
        
               }, 
        
               { 
        
                 before: () => { this.addLog('checking if "python2" can be used') }, 
        
                 check: this.checkCommand, 
        
                 arg: 'python2' 
        
               } 
        
             ] 
        
             if (this.win) { 
        
               checks.push({ 
        
                 before: () => { 
        
                   this.addLog( 
        
                     'checking if the py launcher can be used to find Python 2') 
        
                 }, 
        
                 check: this.checkPyLauncher 
        
               }) 
        
               for (var i = 0; i < this.winDefaultLocations.length; ++i) { 
        
                 const location = this.winDefaultLocations[i] 
        
                 checks.push({ 
        
                   before: () => { 
        
                     this.addLog('checking if Python is ' + 
        
                       `${location}`) 
        
                   }, 
        
                   check: this.checkExecPath, 
        
                   arg: location 
        
                 }) 
        
               } 
        
             } 
        
             return checks 
        
           }

I wrote that hoping that at least that part would be easy to read and extend.

cclauss reviewed Jul 9, 2019

View reviewed changes

cclauss approved these changes Jul 9, 2019

View reviewed changes

cclauss mentioned this pull request Jul 9, 2019

build: set Python semverRange: '>=2.7.0 <3.8.0' #1813

Closed

4 tasks

cclauss requested changes Jul 9, 2019

View reviewed changes

cclauss reviewed Jul 9, 2019

View reviewed changes

joaocgreis added 2 commits July 9, 2019 18:58

python: move Python detection to its own file

c79708d

python: accept Python 3 when NODE_GYP_PYTHON3=1

e2c75b4

This allows us to start testing Python 3 without breaking node-gyp for users. This also adds support for NODE_GYP_FORCE_PYTHON, which will be the only Python binary acceptable when defined.

joaocgreis force-pushed the joaocgreis-J77-python3 branch 2 times, most recently from ea08a83 to e2c75b4 Compare July 9, 2019 20:17

joaocgreis added 2 commits July 9, 2019 21:20

fixup: rename NODE_GYP_PYTHON3 to EXPERIMENTAL_NODE_GYP_PYTHON3

cc6c659

fixup: reject outdated versions 3.0-3.4

4617ecb

fixup: fix linting

19f02f6

cclauss approved these changes Jul 10, 2019

View reviewed changes

richardlau approved these changes Jul 10, 2019

View reviewed changes

joaocgreis added a commit that referenced this pull request Jul 11, 2019

python: move Python detection to its own file

7e7fce3

PR-URL: #1815 Reviewed-By: Christian Clauss <cclauss@me.com> Reviewed-By: Richard Lau <riclau@uk.ibm.com>

joaocgreis closed this Jul 11, 2019

cclauss added the Python label Jul 12, 2019

richardlau mentioned this pull request Jul 12, 2019

build: compile module was removed in Python 3 #1820

Merged

4 tasks

cclauss mentioned this pull request Jul 16, 2019

Travis CI build failures in test/test-find-python.js #1826

Closed

rvagg mentioned this pull request Jul 16, 2019

v5.0.3 proposal #1829

Closed

skewty mentioned this pull request Sep 5, 2019

Why no support Python3 #1687

Closed

paulocoghi mentioned this pull request Sep 7, 2019

Upgrade to Python3 nodegui/nodegui#78

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Accept Python 3 when NODE_GYP_PYTHON3 is defined #1815

Accept Python 3 when NODE_GYP_PYTHON3 is defined #1815

joaocgreis commented Jul 8, 2019 •

edited

Loading

joaocgreis commented Jul 8, 2019

rvagg commented Jul 9, 2019

joaocgreis commented Jul 9, 2019 via email •

edited

Loading

richardlau commented Jul 9, 2019

cclauss Jul 9, 2019 •

edited

Loading

cclauss Jul 9, 2019 •

edited

Loading

cclauss left a comment

cclauss left a comment

joaocgreis commented Jul 9, 2019

cclauss commented Jul 9, 2019 •

edited

Loading

cclauss Jul 9, 2019 •

edited

Loading

joaocgreis Jul 9, 2019

joaocgreis commented Jul 9, 2019

cclauss commented Jul 9, 2019 •

edited

Loading

joaocgreis commented Jul 9, 2019

cclauss commented Jul 9, 2019 •

edited

Loading

joaocgreis commented Jul 9, 2019

cclauss commented Jul 10, 2019 •

edited

Loading

joaocgreis commented Jul 10, 2019

cclauss left a comment

joaocgreis commented Jul 11, 2019

cclauss commented Jul 11, 2019 •

edited

Loading

joaocgreis commented Jul 11, 2019

Accept Python 3 when NODE_GYP_PYTHON3 is defined #1815

Accept Python 3 when NODE_GYP_PYTHON3 is defined #1815

Conversation

joaocgreis commented Jul 8, 2019 • edited Loading

Checklist

Description of change

joaocgreis commented Jul 8, 2019

rvagg commented Jul 9, 2019

joaocgreis commented Jul 9, 2019 via email • edited Loading

richardlau commented Jul 9, 2019

cclauss Jul 9, 2019 • edited Loading

Choose a reason for hiding this comment

cclauss Jul 9, 2019 • edited Loading

Choose a reason for hiding this comment

cclauss left a comment

Choose a reason for hiding this comment

cclauss left a comment

Choose a reason for hiding this comment

joaocgreis commented Jul 9, 2019

cclauss commented Jul 9, 2019 • edited Loading

cclauss Jul 9, 2019 • edited Loading

Choose a reason for hiding this comment

joaocgreis Jul 9, 2019

Choose a reason for hiding this comment

joaocgreis commented Jul 9, 2019

cclauss commented Jul 9, 2019 • edited Loading

joaocgreis commented Jul 9, 2019

cclauss commented Jul 9, 2019 • edited Loading

joaocgreis commented Jul 9, 2019

cclauss commented Jul 10, 2019 • edited Loading

joaocgreis commented Jul 10, 2019

cclauss left a comment

Choose a reason for hiding this comment

joaocgreis commented Jul 11, 2019

cclauss commented Jul 11, 2019 • edited Loading

joaocgreis commented Jul 11, 2019

joaocgreis commented Jul 8, 2019 •

edited

Loading

joaocgreis commented Jul 9, 2019 via email •

edited

Loading

cclauss Jul 9, 2019 •

edited

Loading

cclauss Jul 9, 2019 •

edited

Loading

cclauss commented Jul 9, 2019 •

edited

Loading

cclauss Jul 9, 2019 •

edited

Loading

cclauss commented Jul 9, 2019 •

edited

Loading

cclauss commented Jul 9, 2019 •

edited

Loading

cclauss commented Jul 10, 2019 •

edited

Loading

cclauss commented Jul 11, 2019 •

edited

Loading