Clarify graceful startup in the documentation

[ttn-nguyen42]

With consul-k8s 1.2.x and consul-dataplane 1.1.x, the default behaviour for consul.hashicorp.com/enable-sidecar-proxy-lifecycle: "true" is that (sidecar, application) used to start at the same time. This caused users like us to write a startup script for all of our mesh applications to wait for their proxy to work (which would allow external connections to databases, consul KV, etc.) before it would start itself.

With this PR: Dataplane - net 2898, graceful_startup and Consul K8s - Net 1784 inject sidecar first, included in consul-dataplane:1.3.0 and consul-k8s:1.3 the previous problem just disappear. Containers are still being started at the same time, but seems like the application container waited for the sidecar for a little bit before it would run the entrypoint. This seems like the default behaviour of a postStart hook though I couldn't find any reference to this hook anywhere in the repo and the Dataplane repo as well. I would love a quick explanation for this behaviour change.

It would be nice to add this behaviour change in the Consul documentation and explain it a little bit, in the lifecycle-related Helm configs as it might help users from this issue: Consul K8s - Adding support for lifecycle hooks and health probe for sidecars

[ttn-nguyen42]

My mistake, the issue never went away, it's probably because Kubernetes single threaded container startup sequence starts the proxy first before the service (due to the lifecycle annotation) and that causes a delay long enough for the proxy to start itself.

If I set the intention to "*" to make the sidecar starts longer, the issue persists.

Now that Kubernetes' sidecar is available, do we have any plan to support proper sidecar lifecycle?

[david-yu]

@ttn-nguyen42 This is the issue that is still outstanding: #1397 which would allow for the proxy to wait until the application finishes starting up.