Refactor: Update our python scripts to Python 3

[ph]

Our current python scripts and test frameworks work only under Python 2.7; there are a few changes that we need to make to upgrade our scripts to python 3, notably concerning encoding of strings. But toolings exist to helps us update and compatibility layers also exist to ease the transition.

I understand that python 2.7 EOL is in 2020, but we should still work on that.

related to a discuss post: https://discuss.elastic.co/t/step-3-init-and-create-the-metricset-failure/112046/5

[rvigeant8]

I want to use python 2.7, not python 3.

Here are some details about the problem:

1- I'm using Python 2.7 all the way.
I even removed all references to Python3 in .usr/bin/ as you can see here:

which python
/usr/bin/python
python --version
Python 2.7.12

2- I've also tried using virtenv before running the command "make setup" but I get the same result.

virtualenv -p python2.7 workspace/python2.7
Running virtualenv with interpreter /usr/bin/python2.7
New python executable in /home/ubuntu/workspace/python2.7/bin/python2.7
Not overwriting existing python script /home/ubuntu/workspace/python2.7/bin/python (you must use /home/ubuntu/workspace/python2.7/bin/python2.7)
Installing setuptools, pip, wheel...done.

source workspace/python2.7/bin/activate
(python2.7) ubuntu@beats:~$

It is still pointing to python 2.7 as shown here:

(python2.7) ubuntu@beats: which python
/home/ubuntu/workspace/python2.7/bin/python
(python2.7) ubuntu@beats: python --version
Python 2.7.12

The problems occurs during the processing of "make setup"
I ran the command make setup with the debug option and I see that at one stage python 3 gets setup in directory

3- The recipe is this:

go get github.com/elastic/beats/metricbeat
cd $GOPATH/src/github.com
mkdir rvigeant8
cd rvigeant8
python ${GOPATH}/src/github.com/elastic/beats/script/generate.py --type=metricbeat
Questions answered

Now I run make in debug mode to see all details:
make -d setup > make_setup.logs
``

Traceback (most recent call last):
File "./vendor/github.com/elastic/beats/metricbeat/scripts/create_metricset.py", line 105, in
args.module = raw_input("Module name: ")
NameError: name 'raw_input' is not defined
make[1]: *** [create-metricset] Error 1
make: *** [setup] Error 2

Now if we look at the debug log data, I cut a few interesting part:s:

Updating goal targets....
Considering target file 'create-metricset'.
 File 'create-metricset' does not exist.
  Considering target file 'python-env'.
   File 'python-env' does not exist.
    Considering target file 'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.
     Looking for an implicit rule for 'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.
     Trying pattern rule with stem 'requirements.txt'.

(more...).

In the following log section we see references to python3:

     No implicit rule found for 'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.
     Finished prerequisites of target file 'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.
    No need to remake target 'vendor/github.com/elastic/beats/libbeat/tests/system/requirements.txt'.
   Finished prerequisites of target file 'python-env'.
  Must remake target 'python3    <<--- here
python-env'.
Putting child 0x15955d0 (python-env) PID 1775 on the chain.
Live child 0x15955d0 (python-env) PID 1775
Using base prefix '/usr'
New python executable in /home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/rvigeant8/dpbeat/build/python-env/bin/python3    <<--- here
Also creating executable in /home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/rvigeant8/dpbeat/build/python-env/bin/python
Installing setuptools, pip, wheel...done.
Reaping winning child 0x15955d0 PID 1775
Live child 0x15955d0 (python-env) PID 1785
Reaping winning child 0x15955d0 PID 1785
Removing child 0x15955d0 PID 1785 from chain.
  Successfully remade target file 'python-env'.
 Finished prerequisites of target file 'create-metricset'.

The question is: Why is python-env set to python3 as we can see here?

ubuntu@beats: ls -l build/python-env/bin/python
lrwxrwxrwx 1 ubuntu ubuntu 7 Dec 22 17:38 build/python-env/bin/python -> python3
Please do not hesitate if you more details
Thanks

[ph]

@rvigeant8 Lets the rm -rf build directory and rebuild the python-env with the make python-env command. Can you attach the output of the command to this issue?

[ph]

I can reproduce the problem on Mac using the following scenario:

• remove virtualenv with pip uninstall virtualenv.
• Install python3 with brew install python3.
• Install virtualenv with pip3 install virtualenv.
• looking at the virtualenv file, we can see that the use script python 3 to start #!/usr/local/opt/python3/bin/python3.6 So it defaults to that.
• running make python-env will use python 3.6.

Using base prefix '/usr/local/Cellar/python3/3.6.3/Frameworks/Python.framework/Versions/3.6'
New python executable in /Users/ph/go/src/github.com/elastic/beats/build/python-env/bin/python3.6
Also creating executable in /Users/ph/go/src/github.com/elastic/beats/build/python-env/bin/python
Installing setuptools, pip, wheel...done.

So to use python 2.7 I have to pass params to virtualenv using this command:

VIRTUALENV_PARAMS="-p python2.7" make python-env

Running virtualenv with interpreter /usr/local/bin/python2.7
New python executable in /Users/ph/go/src/github.com/elastic/beats/build/python-env/bin/python2.7
Also creating executable in /Users/ph/go/src/github.com/elastic/beats/build/python-env/bin/python
Installing setuptools, pip, wheel...done.

Its probably better to export the variable

[rvigeant8]

I tried this: export VIRTUALENV_PARAMS="-p python2.7" so looking at the evironment variables I get this: env | grep VIRT VIRTUALENV_PARAMS=-p python2.7 Then it looks better but I get a different problem: the path seems somwhat too long for the pip command. rm -rf build make python-env Running virtualenv with interpreter /usr/bin/python2.7 New python executable in /home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/rvigeant8/dpbeat/build/python-env/bin/python2.7 Also creating executable in /home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/rvigeant8/dpbeat/build/python-env/bin/python Installing setuptools, pip, wheel...done. bash: /home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/rvigeant8/dpbeat/build/python-env/bin/pip: /home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.co: bad interpreter: No such file or directory bash: /home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.com/rvigeant8/dpbeat/build/python-env/bin/pip: /home/ubuntu/workspace/SecOpsTools/IBMDataPowerXMLGW/metricbeat/src/github.co: bad interpreter: No such file or directory vendor/github.com/elastic/beats/libbeat/scripts/Makefile:222: recipe for target 'python-env' failed make: *** [python-env] Error 126 I will shorten the path and check if it completes succesfully

[rvigeant8]

Here are the latest development with my problems creating a metricbeat : I get much further now, however make setup fails with this message: cp: cannot stat 'module/*/_meta/kibana': No such file or directory Here I ran make with debug output to get more details: make -p --debug update # Copyright (C) 1988-2014 Free Software Foundation, Inc. # License GPLv3+: GNU GPL version 3 or later < http://gnu.org/licenses/gpl.html> # This is free software: you are free to change and redistribute it. # There is NO WARRANTY, to the extent permitted by law. Reading makefiles... Updating goal targets.... File 'update' does not exist. File 'python-env' does not exist. Must remake target 'python-env'. Successfully remade target file 'python-env'. File 'collect' does not exist. File 'fields' does not exist. Must remake target 'fields'. Successfully remade target file 'fields'. File 'collect-docs' does not exist. Must remake target 'collect-docs'. Successfully remade target file 'collect-docs'. File 'configs' does not exist. Must remake target 'configs'. Successfully remade target file 'configs'. File 'kibana' does not exist. Must remake target 'kibana'. cp: cannot stat 'module/*/_meta/kibana': No such file or directory vendor/github.com/elastic/beats/metricbeat/Makefile:23: recipe for target 'kibana' failed make: [kibana] Error 1 (ignored) Successfully remade target file 'kibana'. File 'imports' does not exist. Must remake target 'imports'. Traceback (most recent call last): File "./vendor/github.com/elastic/beats/script/generate_imports.py", line 7, in <module> from generate_imports_helper import comment, get_importable_lines ImportError: No module named generate_imports_helper vendor/github.com/elastic/beats/metricbeat/Makefile:57: recipe for target 'imports' failed make: *** [imports] Error 1