From 15dba7498a9bbec538acf6c18b9adbc5802c0615 Mon Sep 17 00:00:00 2001 From: Jaime Soriano Pastor Date: Mon, 27 Jan 2020 11:27:39 +0100 Subject: [PATCH] Update devguide with information about python 3 (#15833) --- docs/devguide/contributing.asciidoc | 15 +++- .../creating-beat-from-metricbeat.asciidoc | 11 ++- docs/devguide/metricset-details.asciidoc | 2 +- docs/devguide/modules-dev-guide.asciidoc | 3 +- docs/devguide/newbeat.asciidoc | 2 +- docs/devguide/python.asciidoc | 90 +++++++++++++++++++ docs/devguide/testing.asciidoc | 2 +- 7 files changed, 115 insertions(+), 10 deletions(-) create mode 100644 docs/devguide/python.asciidoc diff --git a/docs/devguide/contributing.asciidoc b/docs/devguide/contributing.asciidoc index 7d94e44a342..fcf6781cc3b 100644 --- a/docs/devguide/contributing.asciidoc +++ b/docs/devguide/contributing.asciidoc @@ -106,16 +106,23 @@ Another command properly formats go source files and adds a copyright header: make fmt -------------------------------------------------------------------------------- +Both of these commands should be run before submitting a PR. You can view all +the available make targets with `make help`. + These commands have the following dependencies: * Python >= {python} -* https://virtualenv.pypa.io/en/latest/[virtualenv] for Python +* Python https://docs.python.org/3/library/venv.html[venv module] * https://github.com/magefile/mage[Mage] -Virtualenv can be installed with the command `easy_install virtualenv` or `pip -install virtualenv`. More details can be found -https://virtualenv.pypa.io/en/latest/installation.html[here]. Both of these commands should be run before submitting a PR. You can view all the available make targets with `make help`. +Python venv module is included in the standard library in Python 3. On Debian/Ubuntu +systems it also requires to install the `python3-venv` package, that includes +additional support scripts: +[source,shell] +-------------------------------------------------------------------------------- +sudo apt-get install python3-venv +-------------------------------------------------------------------------------- [float] [[build-target-env-vars]] diff --git a/docs/devguide/creating-beat-from-metricbeat.asciidoc b/docs/devguide/creating-beat-from-metricbeat.asciidoc index 0e4c46736f9..0dbf0e0cbca 100644 --- a/docs/devguide/creating-beat-from-metricbeat.asciidoc +++ b/docs/devguide/creating-beat-from-metricbeat.asciidoc @@ -11,10 +11,17 @@ To create your own Beat, you must have Go {go-version} or later installed, and t must be set up correctly. In addition, the following tools are required: * https://www.python.org/downloads/[python] -* https://virtualenv.pypa.io/en/stable/[virtualenv] +* https://docs.python.org/3/library/venv.html[python venv module] * https://github.com/magefile/mage[mage] -Virtualenv is easiest installed with your package manager or https://pip.pypa.io/en/stable/[pip]. +Python venv module is already included in the standard library. In Ubuntu/Debian +it requires additional support scripts that can be installed with the +`python3-venv` package: + +[source,bash] +---- +sudo apt-get install python3-venv +---- [float] ==== Step 1 - Get the metricbeat source code diff --git a/docs/devguide/metricset-details.asciidoc b/docs/devguide/metricset-details.asciidoc index 32e92f05b3e..a68d45dfc9a 100644 --- a/docs/devguide/metricset-details.asciidoc +++ b/docs/devguide/metricset-details.asciidoc @@ -149,7 +149,7 @@ written in Go and have no dependencies. Integration tests are also written in Go but require the service from which the module collects metrics to also be running. System tests for Metricbeat also require the service to be running in most cases and are written in Python based on our small Python test framework. -We use `virtualenv` to deal with Python dependencies. +We use https://docs.python.org/3/library/venv.html[venv] to deal with Python dependencies. You can simply run the command `make python-env` and then `. build/python-env/bin/activate` . You should use a combination of the three test types to test your metricsets because diff --git a/docs/devguide/modules-dev-guide.asciidoc b/docs/devguide/modules-dev-guide.asciidoc index e7083052733..0ee9bfce171 100644 --- a/docs/devguide/modules-dev-guide.asciidoc +++ b/docs/devguide/modules-dev-guide.asciidoc @@ -102,7 +102,8 @@ $ ./export_dashboards -yml '../../../filebeat/module/{module}/module.yml' ---- New Filebeat modules might not be compatible with Kibana 5.x. To export dashboards -that are compatible with 5.x, run the following command inside the developer virtualenv: +that are compatible with 5.x, run the following command inside the developer +virtual environment: [source,shell] ---- diff --git a/docs/devguide/newbeat.asciidoc b/docs/devguide/newbeat.asciidoc index f6e157d7c0c..02bae35078b 100644 --- a/docs/devguide/newbeat.asciidoc +++ b/docs/devguide/newbeat.asciidoc @@ -148,7 +148,7 @@ You now have a raw template of the Beat, but you still need to <= {python}. + +[[installing-python]] +=== Installing Python and venv + +Python uses to be installed in many operating systems. If it is not installed in +your system you can follow the instructions available in https://www.python.org/downloads/ + +In Ubuntu/Debian systems, Python 3 can be installed with: + +["source","sh"] +---- +sudo apt-get install python3 python3-venv +---- + +There are packages for specific minor versions, so for example if Python 3.7 +wants to be used, it can be installed with the following command: + +["source","sh"] +---- +sudo apt-get install python3.7 python3.7-venv +---- + +It is recommended to use Python >= {python}. + +[[python-virtual-environments]] +=== Working with virtual environments + +All `make` and `mage` targets manage their own virtual environments in a transparent +way, so for the most common operations required when contributing to beats, +nothing special needs to be done. + +Virtual environments used by `make` can be found in most Beats directories under +`build/python-env`, they are created by targets that need it, or can be +explicitly created by running `make python-env`. The ones used by `mage` are +created when required under `build/ve`. + +There are some environment variables that can be used to customize the creation +of these virtual environments: + +* `PYTHON_EXE`: Python executable to be used in the virtual environment. It has + to exist in the path. +* `PYTHON_ENV`: Path to the virtual environment to use. If it doesn't exist, it + is created by `make` or `mage` targets when needed. + +Virtual environments can also be used without `make` or `mage`, this is usual +for example when running individual system tests with `nosetests`. There are two +ways to run commands from the virtual environment: + +* "Activating" the virtual environment in your current terminal running + `source ./build/python-env/bin/activate`. Virtual environment can be + deactivated by running `deactivate`. +* Directly running commands from the virtual environment path. For example + `nosetests` can be executed as `./build/python-env/bin/nosetests`. + +To recreate a virtual environment, remove its directory. All virtual +environments are also removed with `make clean`. + +[[python-older-versions]] +=== Working with older versions + +Older versions of Beats were not compatible with Python 3, if you need to +temporary work on one of these versions of Beats, and you don't want to remove +your current virtual environments, you can use environment variables to run +commands in a temporary virtual environment. + +For example you can run `make update` with Python 2.7 with the following +command: + +["source","sh"] +----- +PYTHON_EXE=python2.7 PYTHON_ENV=/tmp/venv2 make update +----- + +If you need to run tests you can also create a virtual environment and then +activate it to run commands from there: +["source","sh"] +----- +PYTHON_EXE=python2.7 PYTHON_ENV=/tmp/venv2 make python-env +source /tmp/venv2/bin/activate +... +----- diff --git a/docs/devguide/testing.asciidoc b/docs/devguide/testing.asciidoc index 3091cd01cd7..a21a8d65ce0 100644 --- a/docs/devguide/testing.asciidoc +++ b/docs/devguide/testing.asciidoc @@ -23,7 +23,7 @@ All Go tests are in the same package as the tested code itself and have the post ==== Running Python Tests -The system tests require a testing binary to be available and the python environment to be set up. To create the testing binary run `make {beatname}.test`. This will create the test binary in the beat directory. To setup the testing environment `make python-env` can be run which will use `virtualenv` to load the dependencies. Then `nosetests` has to be run inside `tests/system`. +The system tests require a testing binary to be available and the python environment to be set up. To create the testing binary run `make {beatname}.test`. This will create the test binary in the beat directory. To setup the testing environment `make python-env` can be run which will use `venv` to load the dependencies. Then `nosetests` has to be run inside `tests/system`. To automate all these steps into one `make system-tests` can be run. This creates the binary, the environment and runs all tests which do not require and external service.