[ci] Add 'mage test' to each project

[andrewkroh]

What does this PR do?

This is a small enhancement to make sure that each project has a mage test. And where applicable a mage unitTest and mage integTest.

It also allows you to pass variables like NOSE_TESTMATCH=test_kafka to limit what system tests execute.

Why is it important?

This will make easier to update the Jenkinsfile because each project will have a mage test target. It will make address #17411 easier.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• [ ] I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Related issues

• Relates Missing CI targets for x-pack software #17411

Commits

commit c9272cf563a5216cc1006d634cc549cdf5079b19

Test Metricbeat using mage directly

commit 91ecc2e7013d1081fd12b6805e7aeb8cdc948097

Decrease verbosity of the chown function when using mage -v

commit 2411d4fd5dae93f52db29c422d17960d021293b6

Remove unused file

commit 0b0d5731e2bb5786643ffd2a24185723c53ed418

Update libbeat integ tests to work under mage

The mage integTest environment does not create a test.env file to pass to docker-compose. So
move the variables into the docker-compose.yml file. If variables from a file are wanted at a
later time then the built-in support for a ".env" file in docker-compose should be used instead.

The python integration test was written to require code coverage instrumentation in the Go
binaries. I updated the test to work when code coverage is not enabled.

commit f6c56fe445eb6ec442e75c2bbb92ec9a2865b408

Fix libbeat test_export_dashboard_cmd_export_dashboard_by_id_unknown_id

The Kibana error message changed so I updated the regex to work on both versions of the error message.

commit 0e01162e0ae179674de80d3d5a97c011e1536730

Add kafka-python to python requirements

Each project is sharing the same python virtual environment as an optimization when using
Mage. Metricbeat had one additional dependency than what was listed in libbeat's requirements.txt.

commit 23655244650db51f3feef9436ba68f0d937ef2d0

Fix metricbeat dashboard test

Mage writes dashboards to build/kibana. The old Makefile convention was to use _meta/kibana.generated.

commit e40f0cc36ca743a0a7238f32da9e8b4de0612c99

Ensure all projects have a test target

Ensure that each project has a test target. Additionally, if the project has unit tests then
make sure it has an unitTest target. And likewise, an integTest target if the project has
integration tests.

This also ensures that for python notetests that the NOSE_TESTMATCH environment
variable can be set to select a subset of tests to execute. For example
`NOSE_TESTMATCH=test_kafka mage pythonIntegTest`.

commit 989b4b177c874654ff2eb4af92c573a6cbe2161e

Remove legacy Windows CI mage targets and files

GoTestUnit and Fields are not longer required for Windows CI tests.

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[jsoriano]

Nice! There are some things overlapping with #17799, the good thing is that they both go in the same direction 🙂 // cc @blakerouse

[andrewkroh]

@jsoriano I made the updates.

To help clarify the purpose here. I'm laying the ground work for Ivan to update Jenkinsfile to use Mage instead of Make base on the spreadsheet linked in #17411.

[andrewkroh]

All tests are green except the macOS generator test which is being fixed in #17782.