Use mage target instead of nosetests on Windows jenkins CI #16141

jsoriano · 2020-02-06T17:37:22Z

What does this PR do?

Replaces use of nosetests with mage wrapper in script for Windows CI workers.

Mage manages their own python virtual environments, this is preferred to directly
call python commands.

Installs python 3 using chocolatey, as it is still not available in windows CI workers.

future was failing to install in Windows, but we don't actually need it anymore,
so it is removed.

Why is it important?

To fix Windows builds in Python 3 branch (#14798).

Mage targets manage their own virtual environments.

jsoriano · 2020-02-06T17:40:29Z

dev-tools/jenkins_ci.ps1

@@ -65,5 +65,5 @@ exec { go test -race -c -cover -covermode=atomic -coverpkg $packages } "go test

 if (Test-Path "tests\system") {
 Set-Location -Path tests\system
- exec { nosetests --with-timer --with-xunit --xunit-file=../../build/TEST-system.xml } "System test FAILURE"
+ exec { mage pythonUnitTest } "System test FAILURE"


Command executed by pythonUnitTest is:

exec: <...>/github.com/elastic/beats/build/ve/linux/bin/nosetests --process-timeout=90 --with-timer -v --with-xunit --xunit-file=build/TEST-python-unit.xml <...>

jsoriano · 2020-02-11T12:53:01Z

This is progressing, only serious pending problem is that x-pack builds for Metricbeat and auditbeat are failing when creating the virtual environment, with the following error:

11:37:24 Error: Command '['C:\\Users\\jenkins\\workspace\\elastic+beats+pull-request+multijob-windows\\beat\\x-pack\\metricbeat\
11:37:24 \label\\windows\\src\\github.com\\elastic\\beats\\build\\ve\\windows\\Scripts\\python.exe', '-Im', 'ensurepip', 
11:37:24 '--upgrade', '--default-pip']' returned non-zero exit status 1.
11:37:24 Error: running "python -m venv C:\Users\jenkins\workspace\elastic+beats+pull-request+multijob-windows\beat\x-pack\metri
11:37:24 cbeat\label\windows\src\github.com\elastic\beats\build\ve\windows" failed with exit code 1
11:37:25 System test FAILURE

For example here: https://beats-ci.elastic.co/job/elastic+beats+pull-request+multijob-windows/6197/beat=x-pack%2Fmetricbeat,label=windows/console

It is interesting because it doesn't happen in other builds that run in principle the same command, I am trying to reproduce it locally.

There are also a couple of errors that I am considering ignoring because they don't affect the builds:

I see the following error, but then tests are run correctly, it also seems to happen with other calls to this exec, is it expected?

11:29:24 At C:\Users\jenkins\workspace\elastic+beats+pull-request+multijob-windows\beat\filebeat\label\windows\src\github.com\el
11:29:24 astic\beats\dev-tools\jenkins_ci.ps1:72 char:12
11:29:24 +     exec { mage pythonUnitTest } "System test FAILURE"
11:29:24 +            ~~~~~~~~~~~~~~~~~~~
11:29:24     + CategoryInfo          : NotSpecified: (2020/02/11 11:2...m\elastic\beats:String) [], RemoteException
11:29:24     + FullyQualifiedErrorId : NativeCommandError
11:29:24

For example here: https://beats-ci.elastic.co/job/elastic+beats+pull-request+multijob-windows/6197/beat=filebeat,label=windows/console

Sometimes Python 3 installation "fails" with the following error, but then everything works ok:

11:25:17 Failures
11:25:17  - python3 (exited -1) - Error while running 'C:\ProgramData\chocolatey\lib\python3\tools\chocolateyInstall.ps1'.

For example here: https://beats-ci.elastic.co/job/elastic+beats+pull-request+multijob-windows/6197/beat=auditbeat,label=windows/console

// cc @andrewkroh @mikemadden42 @kuisathaverat any idea about any of these errors?

kuisathaverat · 2020-02-11T13:58:22Z

11:29:24 At C:\Users\jenkins\workspace\elastic+beats+pull-request+multijob-windows\beat\filebeat\label\windows\src\github.com\el
11:29:24 astic\beats\dev-tools\jenkins_ci.ps1:72 char:12
11:29:24 + exec { mage pythonUnitTest } "System test FAILURE"
11:29:24 + ~~~~~~~~~~~~~~~~~~~
11:29:24 + CategoryInfo : NotSpecified: (2020/02/11 11:2...m\elastic\beats:String) [], RemoteException
11:29:24 + FullyQualifiedErrorId : NativeCommandError
11:29:24

I really hate PowerShell error reporting, I guess it continues because we do not have defined $ErrorActionPreference = 'STOP'

Sometimes Python 3 installation "fails" with the following error, but then everything works ok:

I think that the real failure is that can not change the PATH variable

13:25:09 ERROR: Cannot bind argument to parameter 'Path' because it is null.
13:25:09   python3 can be automatically uninstalled.
13:25:09 Environment Vars (like PATH) have changed. Close/reopen your shell to
13:25:09  see the changes (or in powershell/cmd.exe just type `refreshenv`).
13:25:09 The install of python3 was NOT successful.

11:37:24 Error: Command '['C:\Users\jenkins\workspace\elastic+beats+pull-request+multijob-windows\beat\x-pack\metricbeat
11:37:24 \label\windows\src\github.com\elastic\beats\build\ve\windows\Scripts\python.exe', '-Im', 'ensurepip',
11:37:24 '--upgrade', '--default-pip']' returned non-zero exit status 1.
11:37:24 Error: running "python -m venv C:\Users\jenkins\workspace\elastic+beats+pull-request+multijob-windows\beat\x-pack\metri
11:37:24 cbeat\label\windows\src\github.com\elastic\beats\build\ve\windows" failed with exit code 1
11:37:25 System test FAILURE

The only thing that comes to my mind is that can be the length of the path, I will try to replicate it on a bat file to see if the error is reported.

andrewkroh

This is progressing, only serious pending problem is that x-pack builds for Metricbeat and auditbeat are failing when creating the virtual environment, with the following error:
...
11:37:24 \label\windows\src\github.com\elastic\beats\build\ve\windows\Scripts\python.exe', '-Im', 'ensurepip',
11:37:24 '--upgrade', '--default-pip']' returned non-zero exit status 1.

Try setting PTYHON_ENV to a tmp directory that is shorter in length. That paths to the VE might be too long.

I saw a similar error when updating the Vagrantfile in #15831. When I moved the PYTHON_ENV location off of the virtualbox mount point it started working. I don't think this is same issue since its not a network mount location, but moving the PYTHON_ENV directory to a tmp location might help.

Regarding the exec in powershell, I concur on the dislike of PowerShell. You need a lot of experience to get it right. My strong recommendation is to ditch this script. Bootstrap Go with gvm then immediately invoke something like go run jenkins_windows.go that does all of the other setup that the script used to do.

andrewkroh · 2020-02-11T15:13:40Z

filebeat/tests/system/test_crawler.py

@@ -528,7 +528,7 @@ def test_utf8(self):
 lambda: self.output_has(lines=1), max_timeout=10)

 # Append utf-8 chars to check if it keeps reading
- with codecs.open(testfile, "a") as f:
+ with codecs.open(testfile, "a", "utf-8") as f:


utf_8 is the name of the codec in Python. utf-8 is an alias name. I think I used utf_8 in other places. So to prevent having a mix of "utf-8" and "utf_8" can you change this to utf_8.

jsoriano · 2020-02-11T18:05:54Z

I have changed the PYTHON_ENV path and blue ball 🔵! I think we could go on and merge this.

I agree with rewriting this script in go (or mage), but wdyt about doing it in a follow up?

Thanks @kuisathaverat and @andrewkroh!

andrewkroh · 2020-02-11T18:07:20Z

Awesome! Yeah, let's merge this.

…6141) Replaces use of nosetests with mage wrapper in script for Windows CI workers. Mage manages their own python virtual environments, this is preferred to directly call python commands. Installs python 3 using chocolatey, as it is still not available in Windows CI workers. future was failing to install in Windows, but we don't actually need it anymore, so it is removed.

Use mage target instead of nosetests on Windows jenkins CI

26f4b28

Mage targets manage their own virtual environments.

jsoriano added :Windows :Testing labels Feb 6, 2020

jsoriano self-assigned this Feb 6, 2020

jsoriano commented Feb 6, 2020

View reviewed changes

This was referenced Feb 6, 2020

Migrate scripts to python 3 #14798

Merged

[Draft] Windows use pyenv #16182

Closed

jsoriano added 15 commits February 6, 2020 23:17

Don't change directory

73c59b8

Add message

b7c5614

Set PYTHON_EXE

269a17a

Try to install python in build directory

02b0f75

Try to install using python installer

dd6b251

Wait for installation to finish

0de4308

Revert installation

7a16bbb

Install python 3 with choco

8e39c99

Update path

a288015

Fix path

da5a6f7

Fix path again

2b81703

Remove future from requirements

e1a5a9b

Remove uses of future/past

a1fbf12

Explictly open file as utf-8

ded3557

Add unittest common targets

c901e07

jsoriano requested a review from a team as a code owner February 10, 2020 17:29

Add pythonUnitTest to x-pack libbeat

11616cf

andrewkroh reviewed Feb 11, 2020

View reviewed changes

jsoriano added 3 commits February 11, 2020 17:15

Set PYTHON_ENV to a temporal directory

22c723b

Use utf_8 as encoding in python test

5245db7

Typo

058cfd3

andrewkroh approved these changes Feb 11, 2020

View reviewed changes

jsoriano merged commit 1d2252d into elastic:feature/python3 Feb 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use mage target instead of nosetests on Windows jenkins CI #16141

Use mage target instead of nosetests on Windows jenkins CI #16141

jsoriano commented Feb 6, 2020 •

edited

Loading

jsoriano Feb 6, 2020 •

edited

Loading

jsoriano commented Feb 11, 2020 •

edited

Loading

kuisathaverat commented Feb 11, 2020

andrewkroh left a comment •

edited

Loading

andrewkroh Feb 11, 2020

jsoriano commented Feb 11, 2020

andrewkroh commented Feb 11, 2020

Use mage target instead of nosetests on Windows jenkins CI #16141

Use mage target instead of nosetests on Windows jenkins CI #16141

Conversation

jsoriano commented Feb 6, 2020 • edited Loading

What does this PR do?

Why is it important?

jsoriano Feb 6, 2020 • edited Loading

Choose a reason for hiding this comment

jsoriano commented Feb 11, 2020 • edited Loading

kuisathaverat commented Feb 11, 2020

andrewkroh left a comment • edited Loading

Choose a reason for hiding this comment

andrewkroh Feb 11, 2020

Choose a reason for hiding this comment

jsoriano commented Feb 11, 2020

andrewkroh commented Feb 11, 2020

jsoriano commented Feb 6, 2020 •

edited

Loading

jsoriano Feb 6, 2020 •

edited

Loading

jsoriano commented Feb 11, 2020 •

edited

Loading

andrewkroh left a comment •

edited

Loading