debug startup issues: Some problems occurred during startup

[googs1025]

➜  jobset git:(main) ✗ kubectl apply --server-side -k  /Users/zhenyu.jiang/go/src/golanglearning/new_project/jobset/config/default
namespace/jobset-system serverside-applied
customresourcedefinition.apiextensions.k8s.io/jobsets.jobset.x-k8s.io serverside-applied
serviceaccount/jobset-controller-manager serverside-applied
role.rbac.authorization.k8s.io/jobset-leader-election-role serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-manager-role serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-metrics-reader serverside-applied
clusterrole.rbac.authorization.k8s.io/jobset-proxy-role serverside-applied
rolebinding.rbac.authorization.k8s.io/jobset-leader-election-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/jobset-manager-rolebinding serverside-applied
clusterrolebinding.rbac.authorization.k8s.io/jobset-proxy-rolebinding serverside-applied
secret/jobset-jobset-webhook-server-cert serverside-applied
service/jobset-controller-manager-metrics-service serverside-applied
service/jobset-webhook-service serverside-applied
deployment.apps/jobset-controller-manager serverside-applied
mutatingwebhookconfiguration.admissionregistration.k8s.io/jobset-mutating-webhook-configuration serverside-applied
validatingwebhookconfiguration.admissionregistration.k8s.io/jobset-validating-webhook-configuration serverside-applied

When I use go run main.go, the service starts normally.

➜  jobset git:(main) ✗ go run main.go
2024-02-17T13:03:26+08:00       INFO    setup   both healthz and readyz check are finished and configured
2024-02-17T13:03:26+08:00       INFO    setup   starting manager
2024-02-17T13:03:26+08:00       INFO    setup   waiting for the cert generation to complete
2024-02-17T13:03:26+08:00       INFO    starting server {"kind": "health probe", "addr": "[::]:8081"}
2024-02-17T13:03:26+08:00       INFO    controller-runtime.metrics      Starting metrics server
2024-02-17T13:03:26+08:00       INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8080", "secure": false}
2024-02-17T13:03:26+08:00       INFO    cert-rotation   starting cert rotator controller
2024-02-17T13:03:26+08:00       INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *v1.Secret"}
2024-02-17T13:03:26+08:00       INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2024-02-17T13:03:26+08:00       INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2024-02-17T13:03:26+08:00       INFO    Starting Controller     {"controller": "cert-rotator"}
2024-02-17T13:03:26+08:00       INFO    cert-rotation   no cert refresh needed
2024-02-17T13:03:26+08:00       INFO    Starting workers        {"controller": "cert-rotator", "worker count": 1}
2024-02-17T13:03:26+08:00       INFO    cert-rotation   no cert refresh needed
2024-02-17T13:03:26+08:00       INFO    cert-rotation   Ensuring CA cert        {"name": "jobset-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "jobset-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2024-02-17T13:03:26+08:00       INFO    cert-rotation   Ensuring CA cert        {"name": "jobset-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration", "name": "jobset-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration"}

When I create an example, the error that occurs with the webhook is as follows.

➜  jobset git:(main) ✗ kubectl apply -f examples/simple/max-restarts.yaml
Error from server (InternalError): error when creating "examples/simple/max-restarts.yaml": Internal error occurred: failed calling webhook "mjobset.kb.io": failed to call webhook: Post "https://jobset-webhook-service.jobset-system.svc:443/mutate-jobset-x-k8s-io-v1alpha2-jobset?timeout=10s": dial tcp 10.108.45.68:443: connect: connection refused

Is there anything I should have done less?

[kannon92]

Maybe the deployment had some issues.

https://github.com/kubernetes-sigs/jobset/blob/main/docs/faq/README.md#1-webhook-not-available-error-when-attempting-to-create-a-jobset

[googs1025]

thanks a lot!

[googs1025]

Sorry to ask a basic question. I want to use program debugging, but I find that there are always webhook related errors. I would like to ask how you debug your own programs, such as goland.

I use minikube for testing. But this error occurs when I use.

➜  jobset git:(main) ✗ kubectl apply -f examples/simple/max-restarts.yaml
Error from server (InternalError): error when creating "examples/simple/max-restarts.yaml": Internal error occurred: failed calling webhook "mjobset.kb.io": failed to call webhook: Post "https://jobset-webhook-service.jobset-system.svc:443/mutate-jobset-x-k8s-io-v1alpha2-jobset?timeout=10s": dial tcp 10.108.45.68:443: connect: connection refused

apiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: jobset-mutating-webhook-configuration
webhooks:
  - admissionReviewVersions:
      - v1
    clientConfig:
      url: "https://127.0.0.1:9443/mutate-jobset-x-k8s-io-v1alpha2-jobset"
#      service:
#        name: jobset-webhook-service
#        namespace: jobset-system
#        path: /mutate-jobset-x-k8s-io-v1alpha2-jobset

➜  jobset git:(main) ✗ kubectl apply -f examples/simple/paralleljobs.yaml
Error from server (InternalError): error when creating "examples/simple/paralleljobs.yaml": Internal error occurred: failed calling webhook "mjobset.kb.io": failed to call webhook: Post "https://127.0.0.1:9443/mutate-jobset-x-k8s-io-v1alpha2-jobset?timeout=10s": dial tcp 127.0.0.1:9443: connect: connection refused

I found that when I started the program, I could not access port 9443 using 127.0.0.1

➜  ~ curl --location 'https://127.0.0.1:9443/mutate-jobset-x-k8s-io-v1alpha2-jobset'
curl: (7) Failed to connect to 127.0.0.1 port 9443 after 6 ms: Connection refused

[danielvegamyhre]

The JobSet controller manager deployment resource requirements are 2CPU and 512Mi of memory. If your minikube instance does not have at least that much (plus enough for kube-system pods), the JobSet controller manager pod will not be scheduled and you will see this error.

As explained in the troubleshooting guide, you need to describe the jobset controller manager pod and determine why it is not being scheduled. https://github.com/kubernetes-sigs/jobset/blob/main/docs/faq/README.md#1-webhook-not-available-error-when-attempting-to-create-a-jobset