Update prometheus config map

[donatello]

• in case of secretkey, tls config or minio endpoints change.

• Change UpdatePrometheusConfigMap MinIO secret key change detection: Use jwt
  validation of current prometheus config bearer token to detect if MinIO secret
  key has changed.

• Use sidecar pattern to reload prometheus config on config file changes.

[harshavardhana]

Is this still draft ? @donatello

[harshavardhana]

Is this still draft ? @donatello

Is this still a draft ? @donatello - should this be part of the v2 change or no?

[donatello]

Is this still a draft ? @donatello - should this be part of the v2 change or no?

It is. There is a problem with reacting to changes in configmaps. We need to signal prometheus to reload config after updating the config. So far it is not working properly because there is a propagation delay in config map change to mounted volume change and racing with reload of the prom server. I am trying to do something like a config map informer to fix this issue and hope to get it done mid next week. It is needed for prometheus to be reconfigured on minio password, tls or pool changes.

[donatello]

@harshavardhana @dvaldivia This is working now. I took a sidecar pattern approach to reload prometheus config on config file changes. There doesn't seem to be a simple way to listen for configmap changes.