Guidance for Setting Up Grafana in a High Availability (HA) Configuration in kube-prometheus-stack with grafana managed alerting

[vijaybiradar]

Is your feature request related to a problem ?

Title: Guidance for Setting Up Grafana in a High Availability (HA) Configuration in kube-prometheus-stack

Description:

We are seeking guidance or an example for setting up Grafana in a high availability (HA) configuration using the kube-prometheus-stack. Our setup involves deploying 2-3 replicas of Grafana, all utilizing a shared PostgreSQL database in AWS. The key objectives are to ensure fault tolerance, distribute the load effectively, and centralize alerting across Grafana instances. Additionally, we need support for Grafana’s managed alerting system.

Describe the solution you'd like.

Proposed Setup:

• Deploy kube-prometheus-stack with 2-3 Grafana replicas, leveraging Grafana’s managed Alertmanager.
• Use a shared PostgreSQL database in AWS for Grafana’s backend.
• Deploy Redis (or AWS ElastiCache) for high availability of the alerting system.
• Configure Grafana’s environment variables to connect to PostgreSQL and Redis (ElastiCache) for alerting.

Could you please provide any resources, documentation, or suggestions on how to optimize this setup? Are there any specific configurations or best practices we should follow for this setup using the kube-prometheus-stack?

Any insights or examples would be greatly appreciated!

Describe alternatives you've considered.

Please share any alternative setups or solutions that have worked with kube-prometheus-stack with Grafana's managed alerting in HA grafana configuration

Additional context.