Allow plugins to register their own stats for an _xpack/usage API

[stacey-gammon]

We should provide a mechanism for plugins to extend the Stats API and include their own stats.

---

With #12504, plugins will be able to register their own stats as "extended" stats for the Kibana stats API. The intention for that API is to host operational stats such as performance metrics.

Plugins also want to have a server method available for registering "usage" stats that indicate user activity. Kibana should have a separate API to host these stats, similar to the _xpack/usage API in Elasticsearch.

Currently there is the /api/_kibana/v1/stats endpoint in xpack_main that is close to doing the job of what we need. We want to rename (duplicate it until it can be removed in a breaking change) the endpoint to _xpack/usage. The internals of the endpoint should change as well, so that instead of plucking collector objects out of monitoring/server/kibana_monitoring and calling the fetch method of each one, collector objects can be registered. That will make ownership more clear, and avoid having to hardcode internal specifics across different plugins.

cc @pickypg @alexfrancoeur

[tsullivan]

In some sense we are allowing plugins to define their own usage collection. But I think it's a little subjective on what's meant by "define their own."

There is not a registration method that plugins can call and pass some function into, but Reporting does have a x-pack/plugins/reporting/server/usage/get_usage.js module. Telemetry has hardcoding to import that module and connect it to everything else on its own.

It seems like a better implementation to build a registration method that Reporting can pull in, instead of having Reporting export this magical function.

[tsullivan]

Reopening this issue to track building a registration function that telemetry provides to plugins, as per my previous comment.

[tsullivan]

I re-titled this issue to be more specific about "usage" stats.

The new /stats API (#17773, elastic/beats#6746) will be more about operational and performance stats. This issue needs to be more specific to "usage" collection for telemetry.

---

Added to the description:

With #12504, plugins will be able to register their own stats as "extended" stats for the Kibana stats API. The intention for that API is to host operational stats such as performance metrics.

Plugins also want to have a server method available for registering "usage" stats that indicate user activity. Kibana should have a separate API to host these stats, similar to the _xpack/usage API in Elasticsearch.

Currently there is the /api/_kibana/v1/stats endpoint in xpack_main that is close to doing the job of what we need. We want to rename (duplicate it until it can be removed in a breaking change) the endpoint to _xpack/usage. The internals of the endpoint should change as well, so that instead of plucking collector objects out of monitoring/server/kibana_monitoring and calling the fetch method of each one, collector objects can be registered. That will make ownership more clear, and avoid having to hardcode internal specifics across different plugins.

[tsullivan]

My plan is to work on this in phases:

• Plugins will depend on monitoring to expose the collectorSet object so they can register their stats function with it. That means that monitoring needs to have a yellow status initially, and then change to green right after the collectorSet is exposed: [Monitoring] Add a yellow status phase in plugin init #18939
• When working in the code, I found a few semantic problems with the names used for collection services. There's a typeCollector that lets you register a collector -- what? I'm changing the name of typeCollector to collectorSet for semantics: [Monitoring] Rename TypeCollector to CollectorSet for semantics #18987
• Third PR will actually move the Reporting Usage Stats Collection pieces out of Monitoring and update Reporting's plugin init to call the collectorSet.register method with it's own collector. [Monitoring/Reporting Stats] Move hardcoded Reporting stats type collection out of monitoring plugin #18894
• Fourth PR adds a new API, _xpack/usage which is similar to the same endpoint in ES. This will tell the collectorSet to do a fetch for all of its usage collectors, and then do a small summarization to convert the payload data into a consumable structure. The code is ready but needs to be pulled out of the last commits of the 3rd PR.

All this will need to be done without breaking the 6.x behavior of having the collector set fetch data and combine into a single payload for _xpack/monitoring/_bulk in ES. In 7.0, we're hoping to be able to eliminate that API, and that will allow us to delete some code in the kibana_monitoring module. Until then, there's going to be some deprecation comments around "temporary" code.

[tsullivan]

whoops, it got closed accidentally