Integrate Prometheus Exporter

[mpstefan]

As an operator of NKG in a production environment
I want Prometheus metrics to be exposed for NKG
So that I can observe and monitor the performance of NKG and respond to incidents.

Acceptance

• Prometheus Exporter is integrated with NKG
• NGINX and NKG Controller runtime metrics are exposed through the Prometheus endpoint
• Configurable port for the metrics endpoint
• Annotations are present in the deployment manifest for NKG to make the Prometheus endpoint discoverable
• Documentation is added that includes a list of metrics we provide
• Our architecture document is updated to include the traffic for Prometheus.

[sjberman]

Something to think about is how does a user configure the metrics port? CLI argument? Do we need a custom CRD for NKG-specific configuration? Thinking ahead to when we support tracing, there are quite a few configurables. Might be worth planning this out now.

I'll note that the CRD is likely for mutable runtime fields. Metrics port may not be something we can change at runtime, so CLI argument may make more sense for that.

[mpstefan]

Briefly talked about likely needing to control the metrics port via a CLI argument.