Metrics for configuration policy errors

[willkutler]

https://issues.redhat.com/browse/ACM-2216?filter=-1

Adds 2 metrics - user error and system error gauges that increment when certain errors are found. They are labeled with the parent policy and policy template that caused the error, as well as the reason for the error.

Also adds a check to ensure the spec of a config policy is not nil - when testing the metrics, I found that if a config policy did not have the spec field, the error message would be incorrect and the parent policy would be compliant

[gparvin]

Would you be able to add a test that creates a policy referencing a CRD that doesn't exist and then making sure the error shows up as expected? I don't have an idea for validating the system error path right now.

[dhaiducek]

If these metrics are only being added to, should they be Counters rather than Gauges?

https://prometheus.io/docs/concepts/metric_types/

[gparvin]

@dhaiducek Does the .Add(1) increment the value of the metric or does it add a new entry for the metric at the current time? To me it seems like the metric should be a gauge that reflects the number of errors in the current policy. If the policy gets fixed and there are no more errors, I think a gauge that indicates there are no errors makes sense instead of showing a counter that reflects errors in previous versions of the policy.

Edit: Thinking about this more... I guess after a fix there will just no longer be entries created for the metric. Right?

[gparvin]

Thanks for adding the test! This looks good even though I have some fuzziness on this being a counter. Look forward to trying it out.

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gparvin, willkutler

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [gparvin,willkutler]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment