Add helm chart post-hook option to clean up state secrets when kube agent run as deployment

[deusxanima]

What would you like Teleport to do?

Provide helm chart post-hook option to clean up state secrets of rotated teleport kube agents when kube agents are run as part of a deployment.

What problem does this solve?

Some customers run teleport kube-agents in deployment mode (vs. default sts mode). Currently teleport stores the state of each agent in a secret. When agents are managed manually as a deploy vs. sts, restarting the agents creates a new pod with a new secret, but the previous pod's state secret is left behind.

Teleport currently provides a helm post-hook to clean up stale state artifacts on helm delete, but not for kube pod restarts. This in turn leaves a large number of orphaned secrets behind which require manual cleanup.

If a workaround exists, please include it.

Manually track and remove old state secrets.

[tigrato]

Teleport kube agent chart running in deployment mode shouldn't use secrets. The lack of TELEPORT_REPLICA_NAME, KUBE_NAMESPACE and RELEASE_NAME env variables should prevent the pod from using secret storage.

Have you manually added this env vars?

[deusxanima]

@tigrato - getting clarity and deploy configs from customer env. will update with further info and details