Add metrics to reflect controller health

[ahg-g]

Possible metrics:

• Reconciliation latency
• Reconciliation failures
• JobSets waiting to be processed (we need to look at controller-runtime if it exposes something)

[charles-chenzz]

does this issue urgent? might need some time to check how to implement metrics

[ahg-g]

#60 is more urgent, can you remove the CEL validation and just validate inside the webhook?

[charles-chenzz]

I need to check first, not sure if I know how to do it :(

[danielvegamyhre]

/assign

[danielvegamyhre]

Seems like Reconciliation Latency and Reconciliation Failures are already captured via controller-runtime prometheus metrics here: https://github.com/kubernetes-sigs/controller-runtime/blob/main/pkg/internal/controller/metrics/metrics.go#L37-L44

[ahg-g]

/re-open

We need to add docs explaining the couple of metrics you mentioned above, similar to https://kueue.sigs.k8s.io/docs/reference/metrics/