Traefik return 404 page not found - Kubernetes on Cloudstack 4.17.2.0

[tuanhoangth1603]

Hello,
I created a Kubernetes cluster on Cloudstack and used this guide (https://docs.cloudstack.apache.org/en/latest/plugins/cloudstack-kubernetes-provider.html) to deploy Traefik as an ingress controller.

1. According to the guide:
   On successfully deploying the yaml file, a new Public IP Address in the same network as the cluster will be created. It will automatically have the firewall and port forwarding rules configured to distribute any traffic amongst the cluster worker nodes

However, upon inspecting the LB rules, I noticed that the control node was also added as a backend server ?

2. Furthermore, I created and deployed a file that defines a basic deployment, service, and ingress. However, when I curl http://k8s.lab.com.vn/, the response is "404 page not found". Is there a mistake in my YAML file?
   Thanks for your help!

---
apiVersion: v1
kind: Service
metadata:
  name: app
spec:
  selector:
    app: app
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
  labels:
    app: app
spec:
  replicas: 1
  selector:
    matchLabels:
      app: app
  template:
    metadata:
      labels:
        app: app
    spec:
      containers:
      - name: app
        image: nginx:latest
        ports:
        - containerPort: 80
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: traefik-app-ingress
spec:
  rules:
    - host: k8s.lab.com.vn
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name:  app
                port:
                  number: 80

[weizhouapache]

@tuanhoangth1603
have you solved the issue ?

[tuanhoangth1603]

I follow this guide and the issue was solved.
https://doc.traefik.io/traefik/getting-started/install-traefik/

[tuanhoangth1603]

Hello,

Today I following this procedure to deploy k8s with traefik ingress.
I am facing an error with traefik ingress: "Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value"

• Cloudstack version: 4.19.0.1
• K8S: Version: v1.26.0
• traefik: v3.1.2

Here is full describe of service:

Name:                     traefik
Namespace:                default
Labels:                   app.kubernetes.io/instance=traefik-default
                          app.kubernetes.io/managed-by=Helm
                          app.kubernetes.io/name=traefik
                          helm.sh/chart=traefik-30.1.0
Annotations:              meta.helm.sh/release-name: traefik
                          meta.helm.sh/release-namespace: default
                          service.beta.kubernetes.io/cloudstack-load-balancer-proxy-protocol: enabled
Selector:                 app.kubernetes.io/instance=traefik-default,app.kubernetes.io/name=traefik
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.107.132.3
IPs:                      10.107.132.3
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  30893/TCP
Endpoints:                <none>
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  32256/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type     Reason                  Age                  From                Message
  ----     ------                  ----                 ----                -------
  Normal   EnsuringLoadBalancer    58s (x6 over 2m14s)  service-controller  Ensuring load balancer
  Warning  SyncLoadBalancerFailed  58s (x6 over 2m14s)  service-controller  Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value

[weizhouapache]

any logs on cloudstack management server and in the pod cloud-controller-manager ?

[tuanhoangth1603]

any logs on cloudstack management server and in the pod cloud-controller-manager ?

Thanks for your reply. Here is logs of CCM

W0821 15:11:05.221926       1 client_config.go:552] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0821 15:11:05.420413       1 controllermanager.go:120] Version: v0.0.0-master+$Format:%h$
I0821 15:11:05.422112       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0821 15:11:05.422169       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0821 15:11:05.422223       1 configmap_cafile_content.go:202] Starting client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0821 15:11:05.422240       1 shared_informer.go:223] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0821 15:11:05.618715       1 secure_serving.go:178] Serving securely on [::]:10258
I0821 15:11:05.618813       1 leaderelection.go:242] attempting to acquire leader lease  kube-system/cloud-controller-manager...
I0821 15:11:05.619351       1 tlsconfig.go:240] Starting DynamicServingCertificateController
I0821 15:11:05.918307       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0821 15:11:05.918307       1 shared_informer.go:230] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0821 15:11:06.018794       1 event.go:278] Event(v1.ObjectReference{Kind:"Endpoints", Namespace:"kube-system", Name:"cloud-controller-manager", UID:"79917607-9b73-462e-bf49-76a5cb88c24b", APIVersion:"v1", ResourceVersion:"785", FieldPath:""}): type: 'Normal' reason: 'LeaderElection' cloud-controller-manager-574bcb86c-fn5r5_957b1aed-2d27-4dac-b6af-418d983f7572 became leader
E0821 15:11:06.018661       1 event.go:316] Could not construct reference to: '&v1.Lease{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"cloud-controller-manager", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"33584655-e620-4127-9ce5-e1ac08e95df8", ResourceVersion:"786", Generation:0, CreationTimestamp:v1.Time{Time:time.Time{wall:0x0, ext:63859849865, loc:(*time.Location)(0x30c2040)}}, DeletionTimestamp:(*v1.Time)(nil), DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry{v1.ManagedFieldsEntry{Manager:"cloudstack-ccm", Operation:"Update", APIVersion:"coordination.k8s.io/v1", Time:(*v1.Time)(0xc0001a6340), FieldsType:"FieldsV1", FieldsV1:(*v1.FieldsV1)(0xc0001a6380)}}}, Spec:v1.LeaseSpec{HolderIdentity:(*string)(nil), LeaseDurationSeconds:(*int32)(nil), AcquireTime:(*v1.MicroTime)(nil), RenewTime:(*v1.MicroTime)(nil), LeaseTransitions:(*int32)(nil)}}' due to: 'no kind is registered for the type v1.Lease in scheme "pkg/runtime/scheme.go:101"'. Will not report event: 'Normal' 'LeaderElection' 'cloud-controller-manager-574bcb86c-fn5r5_957b1aed-2d27-4dac-b6af-418d983f7572 became leader'
I0821 15:11:06.118337       1 leaderelection.go:252] successfully acquired lease kube-system/cloud-controller-manager
I0821 15:11:06.120413       1 node_controller.go:110] Sending events to api server.
I0821 15:11:06.120516       1 controllermanager.go:247] Started "cloud-node"
I0821 15:11:06.220667       1 node_lifecycle_controller.go:78] Sending events to api server
I0821 15:11:06.220758       1 controllermanager.go:247] Started "cloud-node-lifecycle"
I0821 15:11:06.319991       1 controllermanager.go:247] Started "service"
I0821 15:11:06.320047       1 core.go:101] Will not configure cloud provider routes for allocate-node-cidrs: false, configure-cloud-routes: true.
W0821 15:11:06.320064       1 controllermanager.go:244] Skipping "route"
I0821 15:11:06.418280       1 controller.go:208] Starting service controller
I0821 15:11:06.418313       1 shared_informer.go:223] Waiting for caches to sync for service
I0821 15:11:06.918866       1 shared_informer.go:230] Caches are synced for service
E0821 15:11:08.719857       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:11:08.721984       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
E0821 15:11:08.818466       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:11:08.821324       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
E0821 15:11:08.823046       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:11:08.920077       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
E0821 15:16:09.221094       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:16:09.319975       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
E0821 15:16:09.321029       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:16:09.323385       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
E0821 15:16:09.324513       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:16:09.327037       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
E0821 15:21:09.922209       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:21:10.019894       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
E0821 15:21:10.021178       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:21:10.118357       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
E0821 15:21:10.120241       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:21:10.122821       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
I0821 15:25:38.304728       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3023", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:25:38.519534       1 controller.go:244] error processing service default/traefik (will retry): failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
I0821 15:25:38.519646       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3023", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
I0821 15:25:43.520304       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3026", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:25:43.521805       1 controller.go:244] error processing service default/traefik (will retry): failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
I0821 15:25:43.521878       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3026", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
I0821 15:25:53.522501       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3026", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:25:53.524087       1 controller.go:244] error processing service default/traefik (will retry): failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
I0821 15:25:53.524171       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3026", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
E0821 15:26:10.619873       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:26:10.622562       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
E0821 15:26:10.718288       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:26:10.721020       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
E0821 15:26:10.722174       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:26:10.724422       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
I0821 15:26:11.190439       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:26:11.191841       1 controller.go:244] error processing service default/traefik (will retry): failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
I0821 15:26:11.191884       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
I0821 15:26:13.524803       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:26:13.526348       1 controller.go:244] error processing service default/traefik (will retry): failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
I0821 15:26:13.526489       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
I0821 15:27:33.527261       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:27:33.721428       1 controller.go:244] error processing service default/traefik (will retry): failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
I0821 15:27:33.818172       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
I0821 15:30:13.723275       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Normal' reason: 'EnsuringLoadBalancer' Ensuring load balancer
E0821 15:30:13.921406       1 controller.go:244] error processing service default/traefik (will retry): failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
I0821 15:30:14.018213       1 event.go:278] Event(v1.ObjectReference{Kind:"Service", Namespace:"default", Name:"traefik", UID:"9b64629a-10b0-4dd7-a7c1-d768d6f502bd", APIVersion:"v1", ResourceVersion:"3141", FieldPath:""}): type: 'Warning' reason: 'SyncLoadBalancerFailed' Error syncing load balancer: failed to ensure load balancer: error retrieving load balancer rules: invalid character '<' looking for beginning of value
E0821 15:31:10.921484       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:31:11.119953       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-control-1917579d77d": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
E0821 15:31:11.121277       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:31:11.123845       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a2e0b": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value
E0821 15:31:11.125066       1 node_controller.go:237] error retrieving instance ID: invalid character '<' looking for beginning of value
E0821 15:31:11.127270       1 node_controller.go:245] Error getting node addresses for node "tuanlab-k8s-node-191757a899c": error fetching node by provider ID: error retrieving node addresses: invalid character '<' looking for beginning of value, and error by node name: error retrieving node addresses: invalid character '<' looking for beginning of value

[tuanhoangth1603]

and I also found this log in mgmt to be helpful

2024-08-21 22:09:17,133 WARN  [c.c.k.c.u.KubernetesClusterUtil] (API-Job-Executor-47:ctx-4b9b2287 job-16540 ctx-d327ad2e) (logid:584f4248) API endpoint for Kubernetes cluster : tuanlab-k8s not available
javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
	at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1701)
	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1519)
	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1421)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:456)
	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:427)
	at java.base/sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:580)
	at java.base/sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:201)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1613)
	at java.base/sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1541)
	at java.base/sun.net.www.protocol.https.HttpsURLConnectionImpl.getInputStream(HttpsURLConnectionImpl.java:250)
	at com.cloud.kubernetes.cluster.utils.KubernetesClusterUtil.isKubernetesClusterServerRunning(KubernetesClusterUtil.java:239)
	at com.cloud.kubernetes.cluster.actionworkers.KubernetesClusterStartWorker.startKubernetesClusterOnCreate(KubernetesClusterStartWorker.java:546)
	at com.cloud.kubernetes.cluster.KubernetesClusterManagerImpl.startKubernetesCluster(KubernetesClusterManagerImpl.java:1321)
	at org.apache.cloudstack.api.command.user.kubernetes.cluster.CreateKubernetesClusterCmd.execute(CreateKubernetesClusterCmd.java:294)
	at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
	at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:112)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:654)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
	at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
	at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
	at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:602)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.io.EOFException: SSL peer shut down incorrectly
	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:489)
	at java.base/sun.security.ssl.SSLSocketInputRecord.readHeader(SSLSocketInputRecord.java:478)
	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:160)
	... 28 more

[tuanhoangth1603]

Hi @weizhouapache,

Do you have any suggestion for this issue ?

[weizhouapache]

any logs on cloudstack management server and in the pod cloud-controller-manager ?

@tuanhoangth1603
it looks like failed to get the cks nodes/vms "tuanlab-k8s-control-1917579d77d" and "tuanlab-k8s-node-191757a899c"

can you try with cloudstack-kubernetes-provider v1.1.0 ?

1. delete cloudstack-kubernetes-provider 1.0.0
   kubectl delete -f
   https://raw.githubusercontent.com/apache/cloudstack-kubernetes-provider/main/deployment.yaml

2. install 1.1.0
   kubectl apply -f
   https://github.com/apache/cloudstack-kubernetes-provider/releases/download/v1.1.0/deployment.yaml

[tuanhoangth1603]

Thanks for you idea.
Sorry, but are you mistaken? Deleting the main release and installing version 1.1.0 are the same. I have checked the two manifest deployment.yaml files, and they are exactly the same.

[weizhouapache]

Thanks for you idea.
Sorry, but are you mistaken? Deleting the main release and installing version 1.1.0 are the same. I have checked the two manifest deployment.yaml files, and they are exactly the same.

It is used to recreate ccm with latest 1.1.0

[weizhouapache]

Thanks for you idea.
Sorry, but are you mistaken? Deleting the main release and installing version 1.1.0 are the same. I have checked the two manifest deployment.yaml files, and they are exactly the same.

It is used to recreate ccm with latest 1.1.0
If you have already installed 1.1.0, ignore it