Make use of prometheus filter settings on IBM-MQ Metricbeat module

[ma-hei]

What does this PR do?

Refactoring the manifest file of the qmgr metricset. Previously a processor was used to filter out non ibmmq metrics. This commit replaces the processor with the more elegant metrics_filter that was introduced recently.

Why is it important?

Using the metrics_filter is cleaner and easier to read.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• [ ] I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• [ ] I have added tests that prove my fix is effective or that my feature works

How to test this PR locally

• Run the IBM MQ container ("docker-compose up" in x-pack/metricbeat/module/ibmmq) locally.
• Run metricbeats with the ibmmq module enabled.
• In Kibana look at the imported metric data -> For metricset qmgr, you should see only metrics beginning with "prometheus.metrics.ibmmq_qmgr"
• Remove the metrics_filter from the manifest file and restart metricbeats
• In Kibana look at the imported metric data -> For metricset qmgr, you should see a number of metrics not beginning with "prometheus.metrics.ibmmq_qmgr" besides metrics beginning with that prefix.

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[elasticmachine]

Since this is a community submitted pull request, a Jenkins build has not been kicked off automatically. Can an Elastic organization member please verify the contents of this patch and then kick off a build manually?

[ChrsMark]

jenkins, test this

[ChrsMark]

@ma-hei thank you for contributing this! This is looking good to me!

@mtojek any comments?

[elasticmachine]

Pinging @elastic/integrations (Team:Integrations)

[elasticmachine]

Pinging @elastic/integrations-services (Team:Services)

[ma-hei]

looking at test failures..

[ma-hei]

@ChrsMark The test build was previously failing because the test expects the "up" event in the returned metrics. The "up" event is now filtered out by the metrics_filter. Before this commit, this test was succeeding because the previously used javascript processor doesn't get applied in the test setting (correct me if I'm wrong). I removed the "up" event from the expected data file because I believe it's filtered out on purpose (judging from logic in "addUpEvent" in prometheus/collector/collector.go). The test should succeed now (and I now know how to run tests locally, before creating a PR :) ) The other option would be to add the "up" metric to the default metrics_filter.

[ChrsMark]

Hey @ma-hei, good job with this! I would see value in reporting up metric since it would give the user the insight of the monitoring endpoint 's health. Since we are collecting from a Prometheus Exporter, it would be meaningful to report the health of it along with the ibmmq specific metrics. By health we actually mean that the endpoint is reachable.

[ma-hei]

@ChrsMark makes sense. I added "up" to the default metrics_filters and re-added the "up" event to the expected test metrics.

[ChrsMark]

jenkins, test this

[ChrsMark]

jenkins, test this please

[ma-hei]

@ChrsMark could you run Jenkins on this once more? I'm fairly confident that the test will succeed now (I forgot to add "up" as an expected metric to another integration test that I wasn't aware of). I can't seem to get this test to run locally.. I'm running "mage update build integTest" from x-pack/metricbeat. It seems to stall at this point.
>> go test: Integration-aws Testing
It might be that my dev environment is in a bad state. If there's any documentation on how to run these tests, I'd be happy to read it.

[ycombinator]

jenkins, test this please

[ChrsMark]

lgtm @ma-hei! Thank you for working on this!