Proposed OLM-based Monitoring Stack Solution for RH Managed Services and future needs.

[bwplotka]

As an effect of the collaboration of the Monitoring Team with Service Delivery org in our Monitoring Enhancement Working Group, we on behalf of our team we would like to propose with @fpetkovski a solution for immediate needs for Managed Services (SaaS/Layered Services/Addons).

Feedback is very welcome!

PTAL @alanmoran @smarterclayton @kbsingh @pb82 @simonpasquier @brampling

Please keep all offline discussions in this PR so everyone is up-to-date.

Signed-by: Bartlomiej Plotka bwplotka@gmail.com

[bwplotka]

Got bit upset by linter, produced issue: #869 (: (with some ideas how to improve it)

[bwplotka]

/test markdownlint

[simonpasquier]

This is very good! Thanks for the effort.

[simonpasquier]

I don't see this issue as a blocker since it's already possible to configure the global Alertmanager configuration with a secret. Having said that supporting the same structures for global and per-namespace Alertmanager configurations would be nice indeed.

[bwplotka]

👍🏽

[fpetkovski]

@simonpasquier is it possible to use both the secret and an AlertmanagerConfig? I thought that with the AlertmanagerConfig everything gets scoped to the namespace from which the config is coming. If that is the case, how would a tenant reference a receiver from the common secret in their AlertmanagerConfig?

[aditya-konarde]

Thanks a lot for working on this! ❤️

Added some suggestions and clarifying questions :)

[matej-g]

Solid chunk of work 💪

Mostly minor suggestions from my side

[jan--f]

Nice, this sounds really good! Left a few comments/suggestions inline.

I'm not sure whether the following is out of scope for this document or not, but should we consider maintenance of this somewhere? Maybe this risk section?
I assume the Monitoring team will maintain this. Adding an additional operator to the teams maintenance workload seems worth mentioning.

[simonpasquier]

/lgtm

[jeremyeder]

@cooktheryan our robot overlords have you set as the next reviewer. Is that correct?

[cooktheryan]

@jeremyeder I wish that was the case but I don't believe I am the correct individual to be making that decision. This should be tagged for someone else

[sugarraysam]

Hello, I am part of the MTSRE team and would like to thank you for putting this proposal together. This is awesome and I am looking forward to working together in order to make this new project happen.

I posted a few comments/suggestions, and they all have the same two underlying goals:

1. Expose an SLO based interface to our tenants
2. Make sure we perform active reconciliation on this interface (operator-pattern)

This is very important as the end goal here is to be able to measure SLOs/SLIs and error budgets. We should not ask our MTO to tweak or touch any prometheus/alertmanager/grafana configs. They need to spend their time doing feature development and bug fixing.

There exists various tools out there that could help us abstract away prometheus/alertmanager and grafana configs:

• Sloth: https://github.com/slok/sloth
• Keptn: https://github.com/keptn/keptn

[sugarraysam]

I want to challenge the interface we want to expose. Instead of ServiceMonitor it would be so much better to provide an SLO-based interface. The reasoning is that we want to abstract as much of the monitoring stack as we can. We want MTO to focus on SLOs and SLIs, not being Prometheus experts. The story I suggest is this one:

As a MTSRE/MTO, I would like to define a SLO based CR that will create Prometheus-compatible config to scrape metrics from my managed service by a Prometheus-compatible system.

Example implementation:

• PrometheusServiceLevel CR from the sloth project: https://sloth.dev/examples/kubernetes/getting-started/

[sugarraysam]

Again proposing to abstract alerting and recording rules away, in favor of PrometheusServiceLevel CR or something similar (sloth):

As an MTSRE/MTO, I would like to create SLO-oriented CRs, that will translate to Prometheus alerting and recording rules for my managed service.

Example implementation:

• PrometheusServiceLevel CR from the sloth project: https://sloth.dev/examples/kubernetes/getting-started/

[sugarraysam]

Again proposing to abstract routing away, in favor of PrometheusServiceLevel CR or something similar (sloth):

As an MTSRE/MTO, I would like to create SLO-oriented CRs, that will configure alerts defined on Prometheus metrics exposed by my SaaS/Addon.

Example implementation:

• PrometheusServiceLevel CR from the sloth project: https://sloth.dev/examples/kubernetes/getting-started/

[sugarraysam]

I think SLO dashboard can be abstracted as well. Creating grafana dashboards is extremely time consuming, and as an MTSRE engineer, I want to have an homogeneous environment when switching from one addon to another. Plus, we do not want MTO to become grafana dashboard experts, or spend any time on this. This is why I would highly suggest:

As an MTO, I would like to provide an SLO based CR, that automatically creates dashboards that allow me to visualize SLO/SLI and error budgets for certain metrics across all clusters where the SaaS/Addon is deployed or used.

Example implementation:

• Sloth SLO oriented grafana dashboards: https://sloth.dev/dashboards/

[sugarraysam]

Adding an extra requirement for MTSRE/MTO:

### Story 8

As an MTO, I would like to provide an SLO based CR, that is actively reconciled by a kubernetes-based operator, so that I can update my SLO/SLI configurations smoothly.

Example implementation:

• sloth operator/controller: https://sloth.dev/usage/kubernetes/

[fpetkovski]

Hi @sugarraysam, thanks for the suggestions, having a higher-level abstraction on top of ServiceMonitors and Dashboards is an awesome idea. Since it comes a bit late into the proposal process and we've already started work on this project, could you create an issue with your suggestions instead? This is the repository for the project: https://github.com/rhobs/monitoring-stack-operator

[nb-ohad]

@fpetkovski @sugarraysam
I think a higher-level abstraction is a nice idea on paper but the reality is that the SRE team or even the tenants' eng teams are not always the ones who define the Monitoring resources.

In the cases I have seen so far, including our own case - ODF MS, the underlying product (that is backing up the service) is the one responsible to deploy/manage/reconcile the monitoring resources (ServiceMonitors, PodMonitors, and PrometheusRuels).

If we go with the proposed abstraction as the only way to configure the scraping we will create a situation where these kinds of tenants will have to "jump through many hops", and introduce some ugly workarounds, just to configure scraping.

If we look at the big picture, taking into account all the deployed components (from infrastructure to the products themselves), we will see that all these workarounds are just creating a less stable solution.

[fpetkovski]

I think that we don't have to make a mutually-exclusive decision. We can still support all the existing APIs and add a few ones on top.

[sugarraysam]

@nb-ohad @fpetkovski I like the idea of exposing multiple interfaces, it makes it flexible and allows a transitioning period. I understand it might be some work to migrate the existing monitoring to an SLO-based approach, but in the long-run this is what we should aim for. Monitoring without SLO does not make much sense. Why are you alerting on X or Y? How does it relate to your users? CPU usage and memory consumption without context are not good measures. The whole industry is shifting towards SLO based monitoring.

Will follow up with a feature request (issue) -> https://github.com/rhobs/monitoring-stack-operator

(edit: this is what app-sre is doing in app-interface as well /schemas/app-sre/slo-document-1.yml)

[nb-ohad]

@sugarraysam I get your point, but your proposal is narrowing the solution to a single use case which is to facilitate service SLO tracking and visibility. There are other use cases for monitoring a managed service, for example, providing product-defined dashboard / OCP console plugins to the customer (as ODF is doing). The metrics that are needed to accomplish that are unrelated to SLO, the SLIs, or even the serviceability of the service.

Saying that I don't see the proposed interface as a replacement of the "older" interface, maybe as an enhancement. This means that we should not put any expectation for a transition to happen for some offerings.

[bwplotka]

I updated the enhancement with the following changes:

• Added Graduation criteria
• Added story for exposing metrics to Console and how to do it.
• Added mention about further steps around soft tenancy potential in other iterations
• Fixed typos

As per your comments @sugarraysam thanks for your idea. It is being discussed in rhobs/observability-operator#13 and I think it's solid thing to implement on top of ServiceMonitors. I don't think we can remove everything else which is not SLO based. Monitoring is more than SLO. It's also about troubleshooting, sometimes billing, data analysis of features etc. So I would not trim all down to SLO usage only, even as MTSRE you have right to ignore anything else. Does it make sense? (:

[simonpasquier]

"at lease" -> "at least"

[simonpasquier]

/lgtm

[bwplotka]

What's missing on this to get this merged? 🤗

[jeremyeder]

I scanned this whole thing again and since the SLO abstraction (which I also highly support) is scoped out, I would lgtm this as-is.

[simonpasquier]

/approve

I think that everybody had a chance to share their feedback and the current version reflects as much as we can where we want to/should go. Thank you all for the constructive comments and thanks @bwplotka and @fpetkovski for this significant proposal :)

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: simonpasquier

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• enhancements/monitoring/OWNERS [simonpasquier]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment