[coord] New secret guarantees

[mkysel]

We have decided to move the creation of the secret into the environment controller in Kubernetes. When using the Kubernetes environment, all scripts have to create a new user-managed-secrets secret, if they desire to use the Kubernetes controller.

The KubernetesSecretsController now waits until the secret becomes visible on its local filesystem. When altering secrets, we do not guarantee that the new secret content has been propagated.

Motivation

This PR adds a known-desirable feature.

Testing

• This PR has adequate test coverage / QA involvement has been duly considered.

Tested manually. Here are the current execution times on a default Kubernetes installation:

root@psql:/# time psql postgres://materialize:materialize@materialized.default.svc.cluster.local:6875/materialize -c "create secret s as 'bar'";
CREATE SECRET

real	0m0.252s
user	0m0.027s
sys	0m0.009s
root@psql:/# time psql postgres://materialize:materialize@materialized.default.svc.cluster.local:6875/materialize -c "alter secret s as 'bar'";
ALTER SECRET

real	0m0.071s
user	0m0.026s
sys	0m0.010s
root@psql:/# time psql postgres://materialize:materialize@materialized.default.svc.cluster.local:6875/materialize -c "drop secret s";
DROP SECRET

real	0m0.083s
user	0m0.026s
sys	0m0.010s

[philip-stoev]

As discussed with @benesch , secrets will be the first thing to test once we have a kubernetes-enabled testing framework.

[mkysel]

thank you @philip-stoev. Looking forward to it.

[mkysel]

adding @alex-hunt-materialize for FYI

[benesch]

This looks about right to me, but would be good to get Alex's eyes on the secret pod update too!

[benesch]

Looks great!