Merge pull request #342 from grafana/remove-CortexQuerierCapacityFull…

…-alert Remove CortexQuerierCapacityFull alert
grafana · Jul 2, 2021 · b266d7b · b266d7b
2 parents 34b4d25 + af74202
commit b266d7b
Show file tree

Hide file tree

Showing 3 changed files with 1 addition and 19 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -17,6 +17,7 @@
 * [CHANGE] Renamed `CortexInconsistentConfig` alert to `CortexInconsistentRuntimeConfig` and increased severity to `critical`. #335
 * [CHANGE] Increased `CortexBadRuntimeConfig` alert severity to `critical` and removed support for `cortex_overrides_last_reload_successful` metric (was removed in Cortex 1.3.0). #335
 * [CHANGE] Grafana 'min step' changed to 15s so dashboard show better detail. #340
+* [CHANGE] Removed `CortexQuerierCapacityFull` alert. #342
 * [ENHANCEMENT] cortex-mixin: Make `cluster_namespace_deployment:kube_pod_container_resource_requests_{cpu_cores,memory_bytes}:sum` backwards compatible with `kube-state-metrics` v2.0.0. #317
 * [ENHANCEMENT] Added documentation text panels and descriptions to reads and writes dashboards. #324
 * [ENHANCEMENT] Dashboards: defined container functions for common resources panels: containerDiskWritesPanel, containerDiskReadsPanel, containerDiskSpaceUtilization. #331

diff --git a/cortex-mixin/alerts/alerts.libsonnet b/cortex-mixin/alerts/alerts.libsonnet
@@ -134,21 +134,6 @@
             |||,
           },
         },
-        {
-          alert: 'CortexQuerierCapacityFull',
-          expr: |||
-            prometheus_engine_queries_concurrent_max{job=~".+/(cortex|ruler|querier)"} - prometheus_engine_queries{job=~".+/(cortex|ruler|querier)"} == 0
-          |||,
-          'for': '5m',  // We don't want to block for longer.
-          labels: {
-            severity: 'critical',
-          },
-          annotations: {
-            message: |||
-              {{ $labels.job }} is at capacity processing queries.
-            |||,
-          },
-        },
         {
           alert: 'CortexFrontendQueriesStuck',
           expr: |||

diff --git a/cortex-mixin/docs/playbooks.md b/cortex-mixin/docs/playbooks.md
@@ -402,10 +402,6 @@ How to **investigate**:
 - Check the latest runtime config update (it's likely to be broken)
 - Check Cortex logs to get more details about what's wrong with the config
 
-### CortexQuerierCapacityFull
-
-_TODO: this playbook has not been written yet._
-
 ### CortexFrontendQueriesStuck
 
 This alert fires if Cortex is running without query-scheduler and queries are piling up in the query-frontend queue.