KafkaConnect and KafkaMirrorMaker2 should roll when certificates or user credentials are updated

[opsocomusr]

Is your feature request related to a problem? Please describe.
When the clients CA certificate is renewed, the user certificates are also renewed but, the cluster operator does not trigger rolling update for the KafkaConnect and KafkaMirrormaker2 deployments and it continues to use the older truststore and keystore.

Describe the solution you'd like
Trigger a rolling update (similar to kafka brokers and entity operator) for the deployments when the mounted user secret is updated with a new certificate.

[scholzj]

This should ideally also apply to MM1 and Bridge.

[tombentley]

@sknot-rh is proposing to do this by computing a hash of the certs (and I guess sasl passwords/ oauth credentials) obtained from the Secrets consumed by the client pods via volume mounts, and annotating the pod template with it, so that the Deployment controller will rolls the pods when any of them.

I put a bit of thought into this, because although it's simple and should work, might not be ideal...

For TLS certificates and keys KIP-687 may apply (I think it's currently only intended for broker configs, but a case could be made for supporting this clients too). In one sense it would be desirable to leverage that since avoiding restarts is always a good thing. But there is no equivalent mechanism in Kafka for other credentials, such as SASLpassword / OAuth credentials, which aren't read from a separate file. So I think it would make sense to ignore KIP-687 for now and focus on a mechanism that requires a rolling restart.

Another consideration is related to #5079/#5277, i.e. should we assume that these sensitive credentials are always provided via a mount of some kubernetes resource that can be accessed by the operator (and thus the operator can determine when the values changed by reading them)? It is possible to imagine some agent or sidecar which observed the sensitive credentials from within the client pod and exposed to the operator only the fact that they'd changed, allowing the operator (via the deployment controller) to roll the Deployment. Although that's pretty complicated it does leave the door open to other secret stores than kube and I kinda like how the CO could operate a KafkaConnect without the ability to connect to the Kafka cluster.

There's also the point that really this is something that kube itself should provide support for. See kubernetes/kubernetes#22368.

On balance, I think if we are going to do this right now then @sknot-rh's suggested route is probably the best way. But TBH I'd prefer to not do this right now, since it seems like one or other of aforementioned things will change and we'll end up with something suboptimal.

[scholzj]

I added this to the agenda of the Community call tomorrow if we get to it.