Should HC activation be delayed until needed secrets are available?

[rgs1]

I did a cursory walk through the code in charge of activating HCs and I don't think it waits for secrets to be ready.

Furthermore, the initial_fetch_timeout broadly refers to the config gRPC stream becoming active (not necessarily to the actual payloads being received):

https://github.com/envoyproxy/envoy/blob/master/docs/root/intro/arch_overview/operations/init.rst

So for clusters where we don't set an initial_jitter we hit upstream_context_secrets_not_ready during hot restarts and we also see failed upstream health checks. I didn't test this for cold starts -- though it might be an issue there too.

I think we should either:

a) not activate HCs until the needed secrets are available

or

b) document the async nature of SDS and how setting initial_jitter might be necessary to workaround a race between the first HCs and secrets becoming available (we could add this to https://github.com/envoyproxy/envoy/blob/master/docs/root/intro/arch_overview/operations/init.rst
)

Thoughts?

cc: @mattklein123 @fishcakez

[mattklein123]

Yeah I think we should fix this somehow. cc @snowp for thoughts.

[mattklein123]

(Sorry I think we should do (a) somehow)

[snowp]

Yeah I agree, we should be sequencing the cluster warming so that we don't set/start the health checker for a cluster until its secrets have warmed.