Mount rules and alerts with the helm chart

[dimitarvdimitrov]

There is an existing issue for creating Prometheus operator CRDs with rules and alerts - grafana/helm-charts#1240. This issue is slightly different.

This proposal is to somehow bundle the default recording and alerting rules with the helm chart so that Mimir starts up and starts evaluating the alerts without external dependencies like the Prometheus operator. This works better for smaller or test/PoC instances of Mimir where meta-monitoring (setting up something external to monitor Mimir) is too high overhead. It's also something we can enable by default as a fallback before someone has set up meta-monitoring.

It's also something that can be used for including arbitrary user rules that the Mimir from the chart can evaluate.

One way to do this would be to run an additional ruler pod which has the rules mounted on the filesystem and only evaluates these rules instead of the ones in the object store bucket.

[alexander-bauer]

This is something that Platform Federal is interested in at the moment, because of how we're deploying GEM: we're using the Helm chart mimir-distributed to deploy into a very restricted FedRAMP environment, and don't currently have Alloy or the Prometheus operator available to manage rules. We do have full control of our Helm chart values, though, and so are able to do our rule definitions up-front in Jsonnet.

A pattern very similar to this is how we're managing our Grafana dashboards: the grafana Helm chart allows us to include JSON dashboards directly in the chart values, where they're turned into ConfigMaps and mounted to the server pod, and then picked up by the dashboard provisioner. This lets us use the Monitoring Mixins pattern to take care of all our dashboard needs, and we'd love to do the same with Mimir and prometheusRules.