(can we) Enable mTLS between prometheus endpoints and prometheus

[draychev]

OSM controller should provision Envoy's prometheus endpoints w/ certs for mTLS.
Graphana pod querying from prometheus should also establish mTLS connection.

Questions to answer:

• can we provision special cert to encrypt and authenticate Envoy's prometheus endpoints?
• can we equip the grafana pod with a side car so that grafana -> prometheus is mTLS
• can we provision prometheus with a sidecar so that in talks mTLS to Envoys and grafana

These sidecars will be a service mesh, but that would not be part of SMI.

[snehachhabria]

Based on #966 we will not need to secure the Promethues and Grafana pods via mTLS.

The changes needed now would be :

• config of Prometheus needs to be updated to scrape the OSM pods using TLS
• provision a cert for Prometheus and update the Prometheus deployment to contain the root cert
• remove the http listener currently configured in OSM for Prometheus
• update the existing listener to contain a new filter match chain only when the feature flag is enabled (this can come in once the feature flag is implemented (Deploy Prometheus and Grafana to OSM via feature flag #1017)
• update the config map to get Prometheus server or service name to configuring the filter chain specifically for Prometheus

[snehachhabria]

@eduser25 let me know if you are still interested in working on this, incase you aren't I'd be happy to take it up in the next milestone. Further, this can be broken up into part as well so if you want to split it too that also works for me

[SanyaKochhar]

@snehachhabria @eduser25 is anybody working on this?

[draychev]

We researched this and it proves to be very difficult to achieve and unnecessary. Shelving this issue.