doc: Certificates and Rotation

[draychev]

This GitHub issue is for fleshing out the OSM documentation around Certificates and Rotation.

We already have documentation around certs. We should augment these docs and make sure they reflect latest v0.8 version of OSM.

• Document Certificate creation and rotation

  • Why are certificates needed? What are they used for?
  • How many types of certificates we have - this is already documented - ensure it is correct
  • How are certificates created?
  • What's in the cert's CN?
  • When are certs rotated?
  • Can this be changed?
  • Why rotate fast?
  • What are the downsides of rotating too soon? What are the upsides?
  • How are certificates sent to proxies?
  • Would a proxy know when a cert expires? Would the Envoy proxy ask the OSM Control plane for a new cert when the cert has expired?
  • Does OSM preemptively send certs to Envoys before they expire? Should it?
  • Are certificates cached?
  • Are these caches cleaned at some point? When?
  • Is there a particular order we need to follow when sending SDS responses? What happens if you send SDS first, before CDS, LDS etc. for a brand new Envoy w/ blank config state.
  • Where is the root cert?
  • How do you rotate the root cert? What would it take? Do you have to restart the OSM control plane?

• Create a small demo with sample apps and SMI showing how this feature works

  • Show how the Envoy is bootstrapped
  • Show what certs are sent to the Envoy

• List Common Issues

  • Expired certs?
  • Rotating too fast?

• Create Troubleshooting Guide

  • How do you know what Certs an envoy needs? (Check CDS, LDS etc)
  • How do you check whether an Envoy has a needed cert?
  • How do you know whether the cert has expired?
  • What tooling do we have to check certs (both OSM and other - openssl etc.)
  • Make a script available to go through all proxies, check their certs?
  • Are there proxies with expired certs? (Not renewed?)
  • Are there proxies with missing certs? (How do we know a cert is missing?)
  • Are there proxise with certs that are present in the Proxy but not linked to / required by any other *DS component?
  • Scan OSM Controller for cert related errors?
  • Check OSM for creating/rotating certs -- are these taking too long? (Timing should be in the logs -- add it if not) Is rotation happening too fast? Are these counters/metrics exposed via OSM's Prometheus endpoints - could be another GitHub issue if this is the case.
  • Check the certs for the Validating Webhook Configuration and the Mutating WC
    • Are they signed by the expected root?
    • Are they expired/expiring soon?
  • Check the certs on each proxy - do they have the expected CN - does it match the Service Account etc of the Pod they are on?

• Automate Troubleshooting Guide in pkg/troubleshooter (create appropriate functions) - alternatively create a GitHub Issue with the stub of the function that could be eventually created within pkg/troubleshooter package to automatically troubleshoot this feature.

[zr-msft]

@draychev @phillipgibson this issue needs a demo to be completed

[bridgetkromhout]

Closing in favor of specifically focused issues:
#434
#435
#436