kube-webhook-certgen create secret job not recreating existing CA secret if parameters change

[botchk]

What happened:

Installation using the kube-webhook-certgen for ca generation.

When updating the --host=<name> in the create secret job with the ingress-nginx/kube-webhook-certgen image, the existing CA secret is not updated. This results in the validating webhook configuration still having the old CA, which then fails all requests to the webhook with errors similar to:

Error: Failed to update Ingress ingress-nginx-hansi/ingress-nginx-hansi-synthetic because: Internal error occurred:
failed calling webhook "validate.nginx.ingress.kubernetes.io": failed to call webhook:
Post "https://ingress-nginx-hansi-controller-lb-admission.ingress-nginx-hansi.svc:443/networking/v1/ingresses?timeout=30s":
tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "x509: ECDSA verification failure" while trying to verify candidate authority certificate "nil1")

This situation can currently only be resolved manually by

# Delete the existing secret
kubectl delete secret -n ingress-nginx-hansi ingress-nginx-hansi-admission
# Rerun the create secret job, to create the secret with the new CA
# Rerun the patch secret job, to inject the new CA from the secret to the validating webhook configuration

Related issues have been reported here already: #5968

What you expected to happen:

When changing parameters of ingress-nginx/kube-webhook-certgen create secret job, that would result in a new CA, any existing secret should be updated with the new CA.

The responsible code can be found here: https://github.com/kubernetes/ingress-nginx/blob/main/images/kube-webhook-certgen/rootfs/cmd/create.go#L27-L34

Two possible approaches could be:

• Don't check if the secret already exists and always update the secret when the job runs
• Check if the secrets exists and uses the generated CA. If not, update the secret with the new CA

This would prevent the need for manual intervention and certification validation errors where it is not immediately clear on why it happens.

NGINX Ingress controller version (exec into the pod and run /nginx-ingress-controller --version): 1.9.6

Kubernetes version (use kubectl version): v1.29.12-eks-2d5f260

Environment:

• Cloud provider or hardware configuration: AWS

• OS (e.g. from /etc/os-release): AL2

• Kernel (e.g. uname -a): -

• Install tools:

  • AWS EKS with Terraform

• Basic cluster related info:

  • kubectl version
    Client Version: v1.32.0
    Kustomize Version: v5.5.0
    Server Version: v1.29.12-eks-2d5f260
    

• How was the ingress-nginx-controller installed:

  • If helm was used then please show output of helm ls -A | grep -i ingress
  • If helm was used then please show output of helm -n <ingresscontrollernamespace> get values <helmreleasename>
  • If helm was not used, then copy/paste the complete precise command used to install the controller, along with the flags and options used
  • if you have more than one instance of the ingress-nginx-controller installed in the same cluster, please provide details for all the instances

• Current State of the controller:

  • kubectl describe ingressclasses
  • kubectl -n <ingresscontrollernamespace> get all -A -o wide
  • kubectl -n <ingresscontrollernamespace> describe po <ingresscontrollerpodname>
  • kubectl -n <ingresscontrollernamespace> describe svc <ingresscontrollerservicename>

• Current state of ingress object, if applicable:

  • kubectl -n <appnamespace> get all,ing -o wide
  • kubectl -n <appnamespace> describe ing <ingressname>
  • If applicable, then, your complete and exact curl/grpcurl command (redacted if required) and the reponse to the curl/grpcurl command with the -v flag

• Others:

  • Any other related information like ;
    • copy/paste of the snippet (if applicable)
    • kubectl describe ... of any custom configmap(s) created and in use
    • Any other related information that may help

How to reproduce this issue:

1. Install ingress-nginx with eg. helm chart
2. Create an ingress resource. Validation works
3. Update the --host parameters of the kube-webhook-certgen https://github.com/kubernetes/ingress-nginx/tree/main/images/kube-webhook-certgen#create
   • In a helm chart this can be done by renaming the controller (.Values.controller.name)
4. Create an ingress resource. Validation fails with CA error

Anything else we need to know:
-

[k8s-ci-robot]

This issue is currently awaiting triage.

If Ingress contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[longwuyuan]

/remove-kind bug
The issues description you have provided is too cryptic and too succint. It does not provide the information needed by readers of a github issue to reproduce or even understand the problem to begin with.

The template of a new bug report template asks questions that are visible in the description post . Answer those questions. And then kindly add the exact install or upgrade command used, the version of the controller attempted to be installed etc etc.