Failing secret webhook causes unexpected behavior

[brandond]

K3s tracking issue for:

• Failing secret webhook causes unexpected behavior rancher/rke2#4266

To test:

1. Create a cluster (any number of servers/agents)
2. Create a bad webhook configuration (this one just calls out to httpbin.org for a 502 error response):

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: rancher.cattle.io
webhooks:
- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig:
    url: https://httpbin.org/status/502
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: rancher.cattle.io.secrets
  namespaceSelector: {}
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - secrets
    scope: Namespaced
  sideEffects: NoneOnDryRun
  timeoutSeconds: 5

3. Attempt to join a new agent
4. Note that the agent is unable to join the cluster - When node registration is blocked by the failing webhook, the error on the server isn't clear what's going on:

May 22 22:15:40 systemd-node-1 rke2[223]: time="2023-05-22T22:15:40Z" level=error msg="Internal error occurred: failed calling webhook \"rancher.cattle.io.secrets\": failed to call webhook: an error on the server (\"\") has prevented the request from succeeding"

The error log on the agent does indicate that the password was rejected, but not why:

May 22 22:17:15 systemd-node-2 rke2[222]: time="2023-05-22T22:17:15Z" level=info msg="Waiting to retrieve agent configuration; server is not ready: Node password rejected, duplicate hostname or contents of '/etc/rancher/node/password' may not match server node-passwd entry, try enabling a unique node name with the --with-node-id flag"

[bguzman-3pillar]

Validated on fe9604ca

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

Ubuntu

Cluster Configuration:

1 server 1 agent

Additional files

$ cat webhook.yaml 
apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: rancher.cattle.io
webhooks:
- admissionReviewVersions:
  - v1
  - v1beta1
  clientConfig:
    url: https://httpbin.org/status/502
  failurePolicy: Fail
  matchPolicy: Equivalent
  name: rancher.cattle.io.secrets
  namespaceSelector: {}
  objectSelector: {}
  reinvocationPolicy: Never
  rules:
  - apiGroups:
    - ""
    apiVersions:
    - v1
    operations:
    - CREATE
    resources:
    - secrets
    scope: Namespaced
  sideEffects: NoneOnDryRun
  timeoutSeconds: 5

Testing Steps

1. Create a cluster (any number of servers/agents)
2. Create a bad webhook configuration (this one just calls out to httpbin.org for a 502 error response):

Validation Results:

$ kubectl get node -A 
NAME              STATUS   ROLES                  AGE     VERSION
ip-172-31-17-79   Ready    control-plane,master   50m     v1.27.3-rc1+k3s1
ip-172-31-29-4    Ready    <none>                 6m36s   v1.27.3-rc1+k3s1

• Logs from server node

$ journalctl -xe | grep "NodePasswordValidationFailed" -i
Jun 16 17:21:34 ip-172-31-17-79 k3s[54471]: I0616 17:21:34.638484   54471 event.go:307] "Event occurred" object="ip-172-31-29-4" fieldPath="" kind="Node" apiVersion="" type="Warning" reason="NodePasswordValidationFailed" message="Deferred node password secret validation failed: Internal error occurred: failed calling webhook \"rancher.cattle.io.secrets\": failed to call webhook: an error on the server (\"\") has prevented the request from succeeding"