Implement automatic certificate rotation in Activator + net-*

[ReToCode]

Context

Our cluster-local encryption should support the rotation of a CA without downtime. For this to work, we need to define how the rotation process looks like and how each component has to handle it. Then implement that accordingly including adding end to end tests for this. We probably also want to add a conformance test around it, to see if all ingress implementations handle that case properly.

📝 Note: For this to work, we need to support multiple CAs in a Secret, during the rotation of all certificates, all our components (and the user components at some point) need to trust more than one CA.

📝 Note: The following is a rough draft

Rotation process

• CA is rotated (either the built in Knative reconciler does this, or it is done by an external party like cert-manager or cluster-admin)

1. The new CA is added to the CA secret (assumption: the old one is still in there)
2. All Ingress controllers update their trust store adding the new CA
3. Activator updates its trust store adding the new CA

Now we have two cases:
4a. When Knative internal CA is used: all generated secrets are updated by Serving.Controller
4b. When external component is used: all generated secrets are either updated by something like cert-manager or cluster-admin

5. All components read their new secret and update their https server with the new certificate
6. The old CA is removed (either the built in Knative reconciler does this, or it is done by an external party like cert-manager or cluster-admin)
7. All Ingress controllers update their trust store adding the new CA
8. Activator updates its trust store adding the new CA

Necessary changes to Activator and net-* implementations

• Activator needs to trust a pool of CAs
• net-kourier needs to trust a pool of CAs
• net-contour needs to trust a pool of CAs
• net-istio needs to trust a pool of CAs
• net-gateway-api needs to trust a pool of CAs

Necessary changes to the Knative integrated CA

• Our integrated CA needs to rotate the CA and keep the old one
• Our integrated CA needs to use the newest CA to re-create all created certificates

Open question:

• How do we identify which CA to use? We could order it based on the issued date and remove it when expired?

[ReToCode]

Related to this #13877

[skonto]

I was looking at the Istio implementation here, https://istio.io/v1.5/pt-br/docs/tasks/security/citadel-config/auth-sds/ wrt case 4a. It seems that Istio uses its own process for certificate rotation and avoids secrets. Should we consider that to implement something similar?

[skonto]

/assign @skonto

[ReToCode]

According to the discussion we had yesterday (https://docs.google.com/document/d/1rpag5-zffHGxAT7V4Nv28C_xx5Ow6L4mZuHbe3ebOQ8/edit#heading=h.bnqlrml3g5zl), this is no longer necessary. We will rely on cert-manager mechanism to trust multiple CAs and do rotation outside of the scope of Knative.
Supporthing this form Knative side, will be handled in #14609.