Investigate collecting background task load metric from Kibana using Metricbeat

[ymao1]

With elastic/kibana#153600, we have added a new HTTP API in Kibana to expose a background task load metric. We initially planned to add a new metricset to the Kibana metricbeat module but have recently learned that we might be able to use the HTTP module to collect this metric with no code changes to metricbeat and are interested in evaluating if this is a feasible path for us.

[botelastic]

This issue doesn't have a Team:<team> label.

[ruflin]

As describe above, the recommended approach is not to extend the existing modules but instead do the collection through one of the http inputs. This also aligns well with the integrations (https://github.com/elastic/integrations/tree/main/packages/kibana) that in general do the collection purely through input configs and not backend code.

I would even encourage to start with the integrations change and then work backwards to enable it in metricbeat as config if needed. @klacabane might have some inputs here from stack monitoring perspective and @joshdover that recently build a change for ingest pipeline monitoring.

[ymao1]

Closing as we decided to update the Kibana integration directly instead of updating the Kibana metricbeat module.