openebs-jiva-csi-node pod gets in CrashLoopBackOff

[pckroon]

Hi all,

I can't get my jiva volumes working, its openebs-jiva-csi-node pod gets stuck in a CrashLoopBackOff.

What steps did you take and what happened:
[A clear and concise description of what the bug is, and what commands you ran.)

1. Clean up old openebs attempts according to the uninstall guide: https://openebs.io/docs/user-guides/uninstall
2. Install jiva as per the quickstart guide: https://github.com/openebs/jiva-operator/blob/develop/docs/quickstart.md

kubectl apply -f https://openebs.github.io/charts/hostpath-operator.yaml
kubectl apply -f https://openebs.github.io/charts/jiva-operator.yaml

3. Edit the openebs-hostpath SC to have a different BasePath (snipped config)

metadata:
  annotations:
    cas.openebs.io/config: |
      - name: StorageType
        value: "hostpath"
      - name: BasePath
        value: "/local-fs/virtuals/kubernetes/openebs"

4. Check status of the jiva components (as per quickguide)

$ kubectl get -n openebs all
NAME                                               READY   STATUS             RESTARTS        AGE
pod/jiva-operator-57b879cfc8-hxnhc                 1/1     Running            0               10m
pod/openebs-jiva-csi-controller-0                  5/5     Running            0               10m
pod/openebs-jiva-csi-node-bd96p                    2/3     CrashLoopBackOff   6 (4m48s ago)   10m
pod/openebs-localpv-provisioner-778d48fff8-9wmbr   1/1     Running            0               14m

NAME                                   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/openebs-jiva-csi-node   1         1         0       1            0           <none>          10m

NAME                                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jiva-operator                 1/1     1            1           10m
deployment.apps/openebs-localpv-provisioner   1/1     1            1           14m

NAME                                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/jiva-operator-57b879cfc8                 1         1         1       10m
replicaset.apps/openebs-localpv-provisioner-778d48fff8   1         1         1       14m

NAME                                           READY   AGE
statefulset.apps/openebs-jiva-csi-controller   1/1     10m

What did you expect to happen:
Working Jiva

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other pastebin is fine.)

• kubectl describe -n openebs pod openebs-jiva-node-bd96p

Name:                 openebs-jiva-csi-node-bd96p
Namespace:            openebs
Priority:             900001000
Priority Class Name:  openebs-jiva-csi-node-critical
Node:                 rohan2013/192.168.1.2
Start Time:           Mon, 13 Jun 2022 13:22:02 +0200
Labels:               app=openebs-jiva-csi-node
                      controller-revision-hash=6799874754
                      name=openebs-jiva-csi-node
                      openebs.io/component-name=openebs-jiva-csi-node
                      openebs.io/version=3.2.0
                      pod-template-generation=1
                      role=openebs-jiva-csi
Annotations:          <none>
Status:               Running
IP:                   192.168.1.2
IPs:
  IP:           192.168.1.2
Controlled By:  DaemonSet/openebs-jiva-csi-node
Containers:
  csi-node-driver-registrar:
    Container ID:  containerd://e08df422db1a7e4e94e033a10f71b74cf53fe83d6ad5e13eedbecf42891d11e0
    Image:         k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0
    Image ID:      k8s.gcr.io/sig-storage/csi-node-driver-registrar@sha256:f9bcee63734b7b01555ee8fc8fb01ac2922478b2c8934bf8d468dd2916edc405
    Port:          <none>
    Host Port:     <none>
    Args:
      --v=5
      --csi-address=$(ADDRESS)
      --kubelet-registration-path=$(DRIVER_REG_SOCK_PATH)
    State:          Running
      Started:      Mon, 13 Jun 2022 13:22:03 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      ADDRESS:               /plugin/csi.sock
      DRIVER_REG_SOCK_PATH:  /var/lib/kubelet/plugins/jiva.csi.openebs.io/csi.sock
      KUBE_NODE_NAME:         (v1:spec.nodeName)
      NODE_DRIVER:           openebs-jiva-csi
    Mounts:
      /plugin from plugin-dir (rw)
      /registration from registration-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9dzsn (ro)
  jiva-csi-plugin:
    Container ID:  containerd://bba1da5f70845cd77697a2e8f2f7523b589a298fd9de0ed904c2e5d8c803a036
    Image:         openebs/jiva-csi:3.2.0
    Image ID:      docker.io/openebs/jiva-csi@sha256:506a4d9ca03a956fe28e5fa8d3bc2960e69accd128e5f93c6024f7cfe0650151
    Port:          <none>
    Host Port:     <none>
    Args:
      --name=jiva.csi.openebs.io
      --nodeid=$(OPENEBS_NODE_ID)
      --endpoint=$(OPENEBS_CSI_ENDPOINT)
      --plugin=$(OPENEBS_NODE_DRIVER)
      --retrycount=20
      --metricsBindAddress=:9505
    State:          Running
      Started:      Mon, 13 Jun 2022 13:22:03 +0200
    Ready:          True
    Restart Count:  0
    Environment:
      OPENEBS_NODE_ID:        (v1:spec.nodeName)
      OPENEBS_CSI_ENDPOINT:  unix:///plugin/csi.sock
      OPENEBS_NODE_DRIVER:   node
      OPENEBS_NAMESPACE:     openebs (v1:metadata.namespace)
      REMOUNT:               True
    Mounts:
      /dev from device-dir (rw)
      /host from host-root (rw)
      /plugin from plugin-dir (rw)
      /sbin/iscsiadm from chroot-iscsiadm (rw,path="iscsiadm")
      /var/lib/kubelet/ from pods-mount-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9dzsn (ro)
  liveness-probe:
    Container ID:  containerd://098a2ff188207d6b34f10a3524d1024762e46d86f7409e6e17c6b08e9b76b6d0
    Image:         k8s.gcr.io/sig-storage/livenessprobe:v2.3.0
    Image ID:      k8s.gcr.io/sig-storage/livenessprobe@sha256:1b7c978a792a8fa4e96244e8059bd71bb49b07e2e5a897fb0c867bdc6db20d5d
    Port:          <none>
    Host Port:     <none>
    Args:
      --csi-address=/plugin/csi.sock
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    255
      Started:      Mon, 13 Jun 2022 13:33:04 +0200
      Finished:     Mon, 13 Jun 2022 13:33:04 +0200
    Ready:          False
    Restart Count:  7
    Environment:    <none>
    Mounts:
      /plugin from plugin-dir (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-9dzsn (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  device-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /dev
    HostPathType:  Directory
  registration-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins_registry/
    HostPathType:  DirectoryOrCreate
  plugin-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/plugins/jiva.csi.openebs.io/
    HostPathType:  DirectoryOrCreate
  pods-mount-dir:
    Type:          HostPath (bare host directory volume)
    Path:          /var/lib/kubelet/
    HostPathType:  Directory
  chroot-iscsiadm:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      openebs-jiva-csi-iscsiadm
    Optional:  false
  host-root:
    Type:          HostPath (bare host directory volume)
    Path:          /
    HostPathType:  Directory
  kube-api-access-9dzsn:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/disk-pressure:NoSchedule op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/network-unavailable:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists
                             node.kubernetes.io/pid-pressure:NoSchedule op=Exists
                             node.kubernetes.io/unreachable:NoExecute op=Exists
                             node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  12m                   default-scheduler  Successfully assigned openebs/openebs-jiva-csi-node-bd96p to rohan2013
  Normal   Pulled     12m                   kubelet            Container image "k8s.gcr.io/sig-storage/csi-node-driver-registrar:v2.3.0" already present on machine
  Normal   Created    12m                   kubelet            Created container csi-node-driver-registrar
  Normal   Started    12m                   kubelet            Started container csi-node-driver-registrar
  Normal   Pulled     12m                   kubelet            Container image "openebs/jiva-csi:3.2.0" already present on machine
  Normal   Created    12m                   kubelet            Created container jiva-csi-plugin
  Normal   Started    12m                   kubelet            Started container jiva-csi-plugin
  Normal   Pulled     11m (x4 over 12m)     kubelet            Container image "k8s.gcr.io/sig-storage/livenessprobe:v2.3.0" already present on machine
  Normal   Created    11m (x4 over 12m)     kubelet            Created container liveness-probe
  Normal   Started    11m (x4 over 12m)     kubelet            Started container liveness-probe
  Warning  BackOff    2m34s (x48 over 12m)  kubelet            Back-off restarting failed container

As you can see, the liveness-probe is failing.

• kubectl logs -n openebs openebs-jiva-csi-node-bd96p -c liveness-probe

I0613 11:33:04.652474       1 main.go:149] calling CSI driver to discover driver name
I0613 11:33:04.653452       1 main.go:155] CSI driver name: "jiva.csi.openebs.io"
I0613 11:33:04.653466       1 main.go:183] ServeMux listening at "0.0.0.0:9808"
F0613 11:33:04.653653       1 main.go:186] failed to start http server with error: listen tcp 0.0.0.0:9808: bind: address already in use
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc000102001, 0xc0003fa000, 0x89, 0xaa)
        /workspace/vendor/k8s.io/klog/v2/klog.go:1021 +0xb9
k8s.io/klog/v2.(*loggingT).output(0xe78700, 0xc000000003, 0x0, 0x0, 0xc0000d6f50, 0xbef6a8, 0x7, 0xba, 0x0)
        /workspace/vendor/k8s.io/klog/v2/klog.go:970 +0x191
k8s.io/klog/v2.(*loggingT).printf(0xe78700, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0xa708f9, 0x2a, 0xc00045cd50, 0x1, ...)
        /workspace/vendor/k8s.io/klog/v2/klog.go:751 +0x191
k8s.io/klog/v2.Fatalf(...)
        /workspace/vendor/k8s.io/klog/v2/klog.go:1509
main.main()
        /workspace/cmd/livenessprobe/main.go:186 +0x86c

goroutine 18 [chan receive]:
k8s.io/klog/v2.(*loggingT).flushDaemon(0xe78700)
        /workspace/vendor/k8s.io/klog/v2/klog.go:1164 +0x8b
created by k8s.io/klog/v2.init.0
        /workspace/vendor/k8s.io/klog/v2/klog.go:418 +0xdf

Anything else you would like to add:
[Miscellaneous information that will assist in solving the issue.]

Environment:

• Jiva version: 3.2.0
• OpenEBS version: 3.2.0
• Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:18:48Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}

• Kubernetes installer & version: kubeadm

kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:24:38Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration: On prem
• OS (e.g. from /etc/os-release): Debian bullseye

Any advice or help is much appreciated :)