Migrate scripts to python 3

[jsoriano]

PR to identify and fix issues with python 3.

If you want to help:

• Pick up an issue from the list of issues identified
• If you have identified an issue that is not in the list, add it there
• Add your name next to the item in the list (if you cannot edit this description, add a comment)
• When you have a fix, open a PR, with feature/python3 as base branch
• Once the issue is solved and the branch merged, check it in the list

How to use Python 3 locally:

If Python 3 is not the default in your system you may need to:

• Remove existing virtual environments you have for beats (you can selectively remove build/python-env and build/ve directories, or just run make clean)
• Install Python 3 in your system if it was not already installed

Issues:

Log of changes that we may consider to revert before merging:

• Forced use of python3 using PYTHON_EXE env var
• Don't check vendor and build directories on make check from the root directory
• Move check to the test stage in Travis so failures there don't block other jobs so we can see their problems.

Log of issues identified/done:

• Remove functools2 library from requirements.txt
• Apply recommendations of 2to3 tool
• Apply recommendations from futurize
• Don't reuse python virtualenvs from the host when running tests in docker Make independent python virtualenvs in docker #14903 @jsoriano
• Python 2 compatibility shims and casts removed remove Python 2 compatibility shims and casts #14989 @beniwohli
• Use python3 in the shebang for python scripts. python is going to be reserved for python2 in Ubuntu, and PEP394 mentions that python should be used in the shebang line only for scripts that are source compatible with both Python 2 and 3 Update script shebangs to python3 #15701
• Review docs to look for pending references to Python 2 @ChrsMark Update documented Python version to 3.7 #15702
• Replace virtualenv with python3 -m venv and get rid of PYTHON_EXE. Use 'python3 -m venv' instead of virtualenv #15754
• Builds are slower in Travis, hitting timeouts in some cases. Upgrade python instead of the base image for python 3 #15931
• make check Fix make check for python 3 #15226 @mikemadden42
  • dev-tools/generate_notice.py:
    • Changed byte array to string in dev-tools/generate_notice.py @jsoriano
    • Something related to encoding fails in CI
    • Alternative to solve issues in dev-tools/generate_notice.py: migrate it to mage
  • Once green, revert change in travis so check is run in a previous check stage Revert changes in travis for the migration to python 3 #15817
  • Different order in fields.asciidoc makes make check to randomly fail on Jenkins Use JSON string for dictionary example #15898 @jsoriano

    -example: {'application': 'foo-bar', 'env': 'production'}
    +example: {'env': 'production', 'application': 'foo-bar'}

• dev-tools/cherrypick_pr Migrate cherrypick_pr to Python 3 #14900 Remove leftover encode from cherrypick_pr #14929 @kvch
• Fix versions of python in CI
  • Install Python 3 in jenkins workers https://github.com/elastic/infra/pull/16269 @jsoriano
  • Upgrade Python 3 (or ubuntu base image) in Travis, it is using 3.4 that is discontinued Fix make check for python 3 #15226 @mikemadden42
  • Python 3 venv not available in Ubuntu CIs in beats-ci @jsoriano
  • Windows jenkins workers are using python 2 Use mage target instead of nosetests on Windows jenkins CI #16141
  • Consider keeping support also with Python 2 if some CI environment cannot be upgraded
• Update Dockerfiles used for running beats during testing to have Python3 Use 'python3 -m venv' instead of virtualenv #15754
• libbeat testsuite
  • Failed to connect: x509: certificate is valid for 127.0.0.1, not 127.0.1.1 (probably similar to Listen explicitly on localhost on heartbeat TCP tests #15583) Regenerate test certificates to include 127.0.1.1 #15591 @jsoriano
  • There is still something running on python 2.7 Explicitly use python3 on libbeat Dockerfile virtualenv #15608 @jsoriano
  • Failures in test_management.py (see this build) Read hits.total.value to get number of hits #15728
• Filebeat testsuite @kvch
  • x-pack suite is still being tested on Python 2 Pass PYTHON_EXE when integration tests are executed from mage #14912 @jsoriano
  • AssertionError: '0o640' != '0640' Migrate Filebeat integration tests to Python3 #15030
  • Type errors ('bool' object is not iterable, a bytes-like object is required, not 'str', 'float' object cannot be interpreted as an integer). Migrate Filebeat integration tests to Python3 #15030
  • ValueError: can't have unbuffered text I/O Migrate Filebeat integration tests to Python3 #15030
  • AttributeError: 'str' object has no attribute 'decode' Migrate Filebeat integration tests to Python3 #15030
• Heartbeat testsuite
  • ImportError: No module named http.server Migrate Auditbeat, Heartbeat and Metricbeat to Python3 #15111 @kvch
  • Local IP not the same (after upgrading ubuntu image?) objects not equal: actual(string(127.0.1.1)) != expected(string(127.0.0.1)) Listen explicitly on localhost on heartbeat TCP tests #15583 @jsoriano
• Auditbeat testsuite
  • Type errors (a bytes-like object is required, not 'str') Migrate Auditbeat, Heartbeat and Metricbeat to Python3 #15111 @kvch
  • test connected UDP IPv4 flow(needs cherry-pick to branch Socket dataset: Workaround for bogus dereference in kernel 5.x #15771) fix backport as part of Update python3 fix auditbeat and dashboards #15807
  • Tests were skipped on Linux Fix python3 sys.platform check for Linux #15727
• Metricbeat testsuite
  • ImportError: No module named cli.command (from compose.cli.command import get_project)
  • ImportError: No module named request (import urllib.request, was autogenerated by 2to3)
  • AttributeError: 'Test' object has no attribute 'assertNotRegexpMatches' Migrate Auditbeat, Heartbeat and Metricbeat to Python3 #15111
  • ActiveMQ test failures and dashbards with incorrect versions Update python3 fix auditbeat and dashboards #15807
  • Missing skip for ES integration tests Metricbeat system test fixes (Python3 branch) #15779
  • Appsearch tests fail Try to fix flakiness of appsearch system tests #15878 🤞 @jsoriano
• Packetbeat testsuite
  • TypeError: Unicode-objects must be encoded before hashing
• Functionbeat testsuite
  • TypeError: 'str' does not support the buffer interface Migrate Functionbeat to Python3 #14901 @kvch
• Winlogbeat pythonUnitTest
  • Fix unicode issues Python3 Windows and Winlogbeat updates #15831
• make docs - to be removed @jsoriano Remove docs build from travis #15816
  • Docker build fails in travis
• vagrant file

  beats/Vagrantfile

  Line 85 in b74e19c

  echo "Installing python2"

  • Test Windows 2019 Vagrantbox with Python3 Python3 Windows and Winlogbeat updates #15831

To do when build is green, before merging:

• Prepare migration guidelines for contributors working with Python 2 @jsoriano Update devguide with information about Python 3 and venv #15833
• Sync with APM Server to decide when to merge @jsoriano

Fixes #5950.

[kvch]

Opened small PR: #14900

[jsoriano]

jenkins, test this again to see if python 3 is already available in workers

[beniwohli]

Great stuff, fantastic to see some more Python 3 adoption!

Is the plan to switch over from Python 2 to Python 3, or to support both at the same time? If it's the former, there's a lot of changes in the PR that aren't really necessary (I assume those were introduced by 2to3 or futurize). Some examples:

• all from __future__ imports

• all from builtins imports

• class SomeClass: -> class SomeClass(object) (in Python 3, all classes inherit from object, there aren't old-style classes anymore)

• many of the the instances where an iterator is cast to a list, only to iterate over it, e.g.

   for _, deps in list(dependencies.items()):
  

•  from future import standard_library
   standard_library.install_aliases()
  

  This installs aliases in Python 2 for modules that moved around in Python 3, so not needed if you target Python 3 only

While these changes don't really hurt, it makes the diff quite a bit larger than it could be, and the code itself is not quite as readable.

If it helps, I can open a PR that removes these unnecessary changes (again, assuming you don't plan to support running the tests with Python 2 going forward).

[andrewkroh]

Ignore the OSX packaging job. The OSX packaging build needs to be disabled/removed until there's a more powerful worker available. It doesn't have enough memory to complete the build.

[jsoriano]

jenkins, test this again please

[jsoriano]

Opened PR with the squashed changes to prepare the backport to 7.x #16302

[urso]

x-pack/metricbeat did fail on travis on go vet running out of memory.

Checking beats-ci and travis-ci, it looks like each testsuite did succeed at least once.

[urso]

The default version should be 8.0.0-SNAPSHOT, so to test against the app-search master branch.

[jsoriano]

Appsearch version is set to 7.5.0 also in master, we will review it as a follow up.

[andrewkroh]

LGTM. No changes needed - just pointing out my own typo.

[andrewkroh]

Suggested change

	// well. So if this applies to your development environment set PYTHON_EXE
	// well. So if this applies to your development environment set PYTHON_ENV

[jsoriano]

Fixed in #16364, where I am also adding a changelog entry for this.