OCPBUGS-1803: Remove compliance_operator_compliance_scan_error_total …

CHANGELOG.md

@@ -25,7 +25,13 @@ Versioning](https://semver.org/spec/v2.0.0.html).
 
 ### Removals
 
--
+- The `compliance_scan_error_total` metric was designed to count individual
+  scan errors. As a result, one of the metric keys contained the scan error,
+  which is large. The length and uniqueness of the metric itself can cause
+  issues in Prometheus, as noted in [Metric and Label Naming best
+  practices](https://prometheus.io/docs/practices/naming/#labels). The error
+  in the metric has been removed to reduce cardinality. Please see the [bug
+  report](https://issues.redhat.com/browse/OCPBUGS-1803) for more details.
 
 ### Security
 

doc/usage.md

@@ -425,7 +425,7 @@ The compliance-operator exposes the following metrics to Prometheus when cluster
     compliance_operator_compliance_scan_status_total{name="scan-name",phase="AGGREGATING",result="NOT-AVAILABLE"} 1
 
     # HELP compliance_operator_compliance_scan_error_total A counter for the
-    # total number of encounters of error
+    # total number errors
     # TYPE compliance_operator_compliance_scan_error_total counter
     compliance_operator_compliance_scan_error_total{name="scan-name",error="some_error"} 1
 

pkg/controller/metrics/metrics.go

@@ -65,9 +65,9 @@ func DefaultControllerMetrics() *ControllerMetrics {
 			prometheus.CounterOpts{
 				Name:      metricNameComplianceScanError,
 				Namespace: metricNamespace,
-				Help:      "A counter for the total number of encounters of error",
+				Help:      "A counter for the total number of errors for a particular scan",
 			},
-			[]string{metricLabelScanName, metricLabelScanError},
+			[]string{metricLabelScanName},
 		),
 		metricComplianceScanStatus: prometheus.NewCounterVec(
 			prometheus.CounterOpts{
@@ -164,8 +164,7 @@ func (m *Metrics) IncComplianceScanStatus(name string, status v1alpha1.Complianc
 	}).Inc()
 	if len(status.ErrorMessage) > 0 {
 		m.metrics.metricComplianceScanError.With(prometheus.Labels{
-			metricLabelScanName:  name,
-			metricLabelScanError: status.ErrorMessage,
+			metricLabelScanName: name,
 		}).Inc()
 	}
 }