Added Mimir / Loki Rules Sync Support

[bentonam]

Resolves #564

[petewall]

I might resurrect this, but I think I might combine alloy-events and alloy-rules into a new alloy-singleton for this purpose.

[luong-komorebi]

I am curious about the progress of this and whether this will be supported in the near future. Right now there is almost no option to use prometheus rule crd for alerting with a k8s-monitoring and mimir setup. Plus, it is also hard to manage mimir rules inside helm chart, which makes this PR important

[stefanandres]

@luong-komorebi
As workaround we've deployed another dedicated alloy helm deployment without the k8s-monitoring chart for that:
(argocd appset)

        - repoURL: https://grafana.github.io/helm-charts
          chart: alloy
          targetRevision: 0.6.1
          helm:
            releaseName: alloy-prometheusrules
            ignoreMissingValueFiles: true
            valueFiles:
              - $values/argocd-apps/clusters/monitoring/k8s-monitoring/values/prometheusrules.yaml
              - $values/argocd-apps/clusters/monitoring/k8s-monitoring/values/prometheusrules-{{ metadata.labels.environment }}.yaml
            valuesObject:
              agent:
                extraEnv:
                  - name: CLUSTER
                    value: "{{ metadata.labels.environment }}"

prometheusrules.yaml

❯ cat prometheusrules.yaml
# Needed to change the selectorLabels
nameOverride: alloy-prometheusrules

crds:
  create: false

alloy:
  enableReporting: false

  resources:
    requests:
      cpu: 10m
      memory: 50Mi
    limits:
      memory: 100Mi

  configMap:
    content: |
        remote.kubernetes.secret "basic_auth_prometheus" {
          name = "basic-auth-prometheus"
          namespace = "monitoring"
        }

        mimir.rules.kubernetes "mimir_ruler" {
          address = nonsensitive(remote.kubernetes.secret.basic_auth_prometheus.data["host"])

          basic_auth {
            username = nonsensitive(remote.kubernetes.secret.basic_auth_prometheus.data["user"])
            password = remote.kubernetes.secret.basic_auth_prometheus.data["password"]
          }

          mimir_namespace_prefix = env("CLUSTER")
        }

controller:
  type: deployment
  updateStrategy:
    type: Recreate
  extraAnnotations:
    reloader.stakater.com/auto: "true"
  tolerations:
  - key: "spot-arm64"
    value: "true"
    operator: "Equal"
    effect: "NoSchedule"

serviceMonitor:
  enabled: true

That's working perfectly with mimir and prometheusrules from each cluster, but I appreciate having this integrated into k8s-monitoring directly 🤩

[luong-komorebi]

@stefanandres awesome to hear your suggested workaround. Thank you for that

While I would love to discuss more on different ways to avoid this blocker, this PR would become offtopic.
As a last message here, I will provide one of my workaround here as well, in order to prove the interest for this work
I have to resort to mimirtool with terraform using https://github.com/ovh/terraform-provider-mimirtool and providing the yamls inside terraform
Since I am already using terraform to manage helm release of mimir, that is one of the shortest way I can think of, besides running mimirtool in a local-exec.
However, this solution is not ideal due to extra component it requires, plus I manage multiple clusters and would love to have Mimir / Loki Rules as close to where they are needed as possible. Thus, this work when merged would open a more intuitive door for provision mimir and loki rules