Camel K operator monitoring

[astefanutti]

This PR:

• Exposes the metrics container port in the operator deployment
• Registers metrics for the relevant operator SLIs to the /metrics endpoint exposed by the operator:
  • Resources reconciliation duration histogram
  • Build duration histogram
  • Build queue duration histogram
  • Build recovery attempts histogram
  • Time to first integration readiness histogram
• Adds a monitoring option to the kamel install command, so that kamel install --monitoring=true:
  • Creates a default PodMonitor resource targeting the operator metrics endpoint
  • Creates a default PrometheusRule resource with alerting rules for:
    • Aggregated reconciliation request duration SLO
    • Aggregated reconciliation request failure SLO
    • Build duration SLOs (warning 1% > 2m, critical for 1% > 5m)
    • Build failure and error SLOs (warning for failures, critical for errors)
    • Build queue duration SLOs (warning for 1% > 1m, critical for 1% > 5m)
• Adds an option to configure the metrics endpoint port
• Exposes a health endpoint with a liveness probe
  • Note that no readiness probe is added, as it conflicts with leader election in case of a rolling update of the operator. Besides readiness isn't very relevant for the Camel K operator anyway.

Fixes #1267.

TODO:

• Document metrics (done in chore(doc): Add Camel K operator monitoring documentation #1812)
• Document securing the metrics endpoint using kube-rbac-proxy
  • It may depends on Add support for TLS and authentication to PodMonitors prometheus-operator/prometheus-operator#3370 that adds support for TLS configuration of PodMonitor resources and that has yet to be released been released in version 0.43.0

It should be enabling the discrimination of user vs. platform errors (#1633).

Release Note

feat: Camel K operator monitoring

[astefanutti]

Let's merge this and create a follow-up PR for the documentation.