[Stack Monitoring] Remove custom Monitoring Bulk API endpoint

[ycombinator]

The work described in this issue is a breaking change. As such, it is intended to be released only in a major release (i.e. no backports).

We now have the ability to monitor Elasticsearch, Kibana, Logstash, Beats, and APM Server with Metricbeat. The previous method of monitoring stack products, using collection code internal to each product that shipped monitoring data to a custom Monitoring Bulk API endpoint, is now deprecated. And in the next major release (likely 8.0.0), all code related to this internal collection approach should be removed.

This issue tracks the removal of one specific item related to the internal collection approach: the custom Monitoring Bulk API endpoint, i.e. POST _monitoring/bulk.

By removing this endpoint, various stack products (other than Elasticsearch) will no longer be able to ship their internally-collected monitoring metrics to Elasticsearch via this endpoint. Instead these products will now have to be monitored externally by Metricbeat, which will ship monitoring metrics (to the monitoring Elasticsearch cluster) using the standard POST _bulk API endpoint.

[elasticmachine]

Pinging @elastic/es-core-features

[romseygeek]

This would enormously simplify the removal of types from the Bulk API, as the monitoring bulk endpoint still uses the _type field to distinguish between input types.

[cachedout]

@jakelandis When we're ready to do this, is this something that you think the ES team can take on? I'm just hoping we can find an owner for this work.

[jakelandis]

@cachedout - yes this is something that ES team would take care of.

[dakrone]

Internal monitoring has been deprecated for quite a while, and we're no longer doing any active development on it. I'm going to close this issue.