Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cert generated by init container is not removed during uninstall #23

Open
f1ames opened this issue Nov 27, 2023 · 2 comments
Open

Cert generated by init container is not removed during uninstall #23

f1ames opened this issue Nov 27, 2023 · 2 comments
Labels
bug Something isn't working

Comments

@f1ames
Copy link
Contributor

f1ames commented Nov 27, 2023

There is an init container which generates certificate used by webhook server (see here). And since the cert is generated as part of initialization logic and not helm chart / install script itself it is not removed on helm install / kubectl delete.

It doesn't do much harm because the cert will be reused (or renewed when expired) when admission controller is again deployed to the same namespace. Still it will be better to have everything cleaned up. EDIT: see #23 (comment) below.

For helm we can use helm hooks (like post-delete) to fix this. For install manifest (or as more generic approach) we can maybe use container-lifecycle-hooks (see here too).

This is how leftover cert looks like:

NAME                                      TYPE                                  DATA   AGE
secret/monokle-admission-controller-tls   kubernetes.io/tls                     2      2m12s
@f1ames f1ames added the bug Something isn't working label Nov 27, 2023
@f1ames
Copy link
Contributor Author

f1ames commented Dec 21, 2023

It seems there is in fact negative impact there. Reinstalling to the same namespace, it seems to cause an issue with next deployment using previous cert and communication to webhook server is broken with:

Error from server (InternalError): Internal error occurred: failed calling webhook "monokle-admission-controller-server.monokle.svc": failed to call webhook: Post "https://monokle-admission-controller-server.monokle.svc:443/validate?timeout=10s": x509: certificate signed by unknown authority

I think this happens because there are two parts here:

  • Server cert resource is created under secret/monokle-admission-controller-tls.
  • Authority cert (with which server cert is signed) is added as webhook resource webhookConfig.clientConfig.caBundle.

And while server cert is reused, webhook caBundle is not propagated with authority cert on reinstall (because this cert is not there basically).

const certs = generateCertificates(NAMESPACE, 6);
const webhookPatched = patchWebhookCertificate(NAMESPACE, WEBHOOK_NAME, webhookConfig, certs.caCert, kc);
if (!webhookPatched) {
throw new Error('Failed to update webhook.');
}
logger.info('Webhook patched successfully.');
const certCreated = await applySecretCertificate(NAMESPACE, SECRET_NAME, certs.serverKey, certs.serverCert, kc);
if (!certCreated) {
throw new Error('Failed to create secret.');
}
logger.info('Secret created successfully.');
}

@f1ames
Copy link
Contributor Author

f1ames commented Jan 8, 2024

There are two parts here:

  • Making sure that uninstalling and installing again works (this is important).
  • Improving certs cleanup after uninstalling.

Those are related and can be fixed at once (by single solution), but it depends on the approach. The important part is fixing the issue with reinstalling.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant