Add prometheus-exporter-plugin to Opensearch builtin plugins

[saeed-mcu]

Currently, it is not possible to monitor Opensearch performance from the Opensearch dashboard, while Elasticsearch can be monitored through Stack Monitoring easily.

If the prometheus-exporter-plugin is added to the Built-in plugins , monitoring of opensearch will be easy, and it can be have dashboard something like stack monitoring or monitored through Prometheus externally.

[lukas-vlcek]

Hi,

OpenSearch Community projects page lists Prometheus exporter plugin which can be installed very easily (like any other OpenSearch plugin). On top of that the Prometheus exporter plugin repo has a branch mixin which contains Grafana dashboard and Prometheus Alerts (if you do not want to build the dashboard from sources you can download its release here: https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/tag/mixin-1.0.0-rc.1).

Is that what you are looking for?

Regards,
Lukáš

[peternied]

This seems like a good feature to consider - I know there hasn't been much traction, but I'm going to tag in a number of feature areas that might want to take ownership here.

[coredump17]

Any progress on this? Dashboards already supports Prometheus as a data source.

[AdaptiveStep]

Hi,

OpenSearch Community projects page lists Prometheus exporter plugin which can be installed very easily (like any other OpenSearch plugin). On top of that the Prometheus exporter plugin repo has a branch mixin which contains Grafana dashboard and Prometheus Alerts (if you do not want to build the dashboard from sources you can download its release here: https://github.com/aiven/prometheus-exporter-plugin-for-opensearch/releases/tag/mixin-1.0.0-rc.1).

Is that what you are looking for?

Regards, Lukáš

Why is this not a default plugin inside opensearch?!!! Its extremely useful!

[lukas-vlcek]

@AdaptiveStep There is a more general RFC to introduce metric framework directly into OpenSearch which should be able to support Prometheus servers as well.

[dblock]

Why is this not a default plugin inside opensearch?!!! Its extremely useful!

I think the maintainers of https://github.com/aiven/prometheus-exporter-plugin-for-opensearch should first voice whether they want their plugin to be included in the default distribution or not. Consider opening an issue there?

In general, every plugin we add proportionally increases the work to make a release because OpenSearch is tightly coupled, which isn't great.

[saeed-mcu]

@dblock
prometheus-exporter-plugin-for-opensearch is very very essential and useful for performance analysis of opensearch and I think it MUST be added as default plugin inside opensearch as soon as possible

[lukas-vlcek]

I think it would make sense to bring the Prometheus exporter plugin a lot closer to the OpenSearch core and I would support this transition.

Let's discuss some cons and pros (feel free to add more, the following are just the ones I can think of right now).

Cons

Increased cost for the core maintainers and committers:

• The main "downside" I can see is that this will add "yet another core plugin" that needs maintenance.

  • However; I see this as temporary only because once the [RFC] Metrics Framework matures enough then the Prometheus exporter plugin should not be needed at all – see below in Pros section for further details.
  • As long as I am the main maintainer of the Prometheus exporter plugin I do not think it is that important which repository/organization I do this work in – except for releases, which I do not have any privs to do if the code is in one of the OpenSearch upstream repositories.

• New plugin PRs will probably need reviews from more people than a single one (currently it is me). Actually, I don't think this is a bad thing, but yes, technically this needs to be accounted for in the cost.

Pros

• The [RFC] Metrics Framework should replace the need for Prometheus exporter plugin in a longer run (or at least that is my understanding). Having the Prometheus exporter plugin directly under the same "umbrella" the Metric Framework effort could benefit from having a real goal to aim for. I might be mistaken but it seems to me that the Metric Framework focuses a lot on Node Stats API but in practice useful metrics come also from other places (such as Node Info, Indices Stats or Cluster configuration to name a few and individual pieces of the data need to be aligned). This can be a way how to make sure the Metric Framework provides something that is really requested by existing users using Prometheus to monitor OpenSearch cluster(s).

Other things to consider

• There is more to the monitoring story than just metric exporting from OpenSearch. In case of Prometheus/Grafana there are also Prometheus recording rules and alerts and Grafana dashboard. These parts can be currently found in the mixin branch of the Prometheus exporter plugin repo. They may still need a home... and I am not sure if upstream OpenSearch repo is the right place for it. On the other hand these parts are usually custom specific and ready-to-customize alternatives could be found elsewhere.

Regards,
Lukáš

[lukas-vlcek]

FYI, related: #13879

[spapadop]

Hi everyone,

Unless this request goes through, there is essentially currently no official option to monitor OpenSearch clusters, with the industry-standard prometheus way. In other words, the prometheus-exporter plugin is too important to be kept outside the core official plugins.

Whatismore, it is currently a one-person repository, requiring @lukas-vlcek to be constantly available and closely follow-up on every OpenSearch release (at least till #13187 is implemented). He has done that for far too long and I am personally very thankful, but this is not sustainable. Of course, we can each fork the repo and do the releases on our side, but this is not really a sign of a healthy community. If he agrees, more people could be added as maintainers on the repo, but again, it would be more reassuring (and look better in my eyes) if the plugin gets within the opensearch-project.

@dblock @krisfreedain it would be nice to have an official decision on whether the prometheus exporter plugin should move into the opensearch-project or if this is of no interest for the OpenSearch roadmap, so that we know how to proceed (e.g., end up with multiple forks or try to maintain the repo under Aiven).

#10141 which could be a future-proof alternative looks pretty stale to me, being experimental since 2.11.