[Metricbeat] Remove xpack enabled flag on ES, Logstash, Beats and Kibana

[sayden]

What does this PR do?

Elasticsearch, Kibana, Logstash and Beat modules has a flag called xpack.enabled: true/false that can be activated to ensure that all metrics required on Stack Monitoring UI in Kibana are included in events from those modules. Before that, only a subset of the metrics were included.

At the same time, all metrics have been "refactored" to be fully compatible with ECS and the common structure of Metricbeat events.

Finally, metrics will appear in indices of Metricbeat (the good ol' metricbeat-* indices) instead of .monitoring indices. This way users will also be able to build their own dashboards around those modules more easily.

xpack.enabled: true flag will exist for some time more to help users activate the Metricbeat modules required by SM UI in Kibana.

Why is it important?

ECS is a benefit for all users and Elasticsearch, Kibana, Logstash and Beat modules weren´t publishing events following the schema yet. OSS users will also benefit from this change, as the metrics will now be available in metricbeat-* indices.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

To test this PR locally, a SNAPSHOT version of Metricbeat and Kibana is necessary. Launch Elasticsearch, Kibana and Logstash and setup the same Metricbeat modules to monitor them. Metrics from those modules should appear in metricbeat-* indices and the Stack Monitoring UI in Kibana should work without issues.

To test Beats, you need to setup another Metricbeat or use this same one to monitor itself.

Related issues

• Relates Make beats module work for Stack Monitoring without xpack.enabled flag #21272 Make elasticsearch/ccr metricset work for Stack Monitoring without xpack.enabled flag #21348 Make elasticsearch/cluster_stats metricset work for Stack Monitoring without xpack.enabled flag #21099 Make elasticsearch/enrich metricset work for Stack Monitoring without xpack.enabled flag #20821 Make elasticsearch/index metricset work for Stack Monitoring without xpack.enabled flag #21438 Make elasticsearch/index_recovery metricset work for Stack Monitoring without xpack.enabled flag #20810 Make elasticsearch/index_recovery metricset work for Stack Monitoring without xpack.enabled flag #20810 Make elasticsearch/index_summary metricset work for Stack Monitoring without xpack.enabled flag #20615 Make elasticsearch/ml_job metricset work for Stack Monitoring without xpack.enabled flag #21043 Make elasticsearch/node_stats metricset work for Stack Monitoring without xpack.enabled flag #19747 Make elasticsearch/shard metricset work for Stack Monitoring without xpack.enabled flag #21389 Make kibana/stats metricset work for Stack Monitoring without xpack.enabled flag  #21496 [Metricbeat][SM] Extract a kibana/settings metricset from kibana/stats #23200 Make logstash/node_stats metricset work for Stack Monitoring without xpack.enabled flag #21546 [Monitoring] Adapt UI code to read from metricbeat-* indices kibana#73864
• Requires [Monitoring] Ensure Stack Monitoring UI works with metricbeat-* indices kibana#96205

[sayden]

/test

[sayden]

/test

[sayden]

/test

[chrisronline]

Here are the changes I needed to make to get beats/apm working: https://gist.github.com/chrisronline/f18740f00601251f979cec3dccbac320

At a high level, the apm-server aliases were all pointing to apm_server when it's really apm-server. I just kept it apm-server in both places but feel free to change it and lemme know what you decide.

I also noticed the version was not properly passed for ES cluster_stats so that change is in the diff too

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b feature-stack-monitoring-mb-ecs upstream/feature-stack-monitoring-mb-ecs
git merge upstream/master
git push upstream feature-stack-monitoring-mb-ecs

[ycombinator]

I skimmed through the changes in this PR as it's quite large. In any case, most of these changes were reviewed in smaller PRs that targeted the feature-stack-monitoring-mb-ecs branch.