You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
linyuting@MacbookDeYuting beegfs-csi-driver % kubectl logs csi-beegfs-controller-0 -n beegfs-csi
I1219 21:45:53.140374 1 beegfs_util.go:436] "msg"="Executing command" "command"=["lsmod"]
I1219 21:45:53.147503 1 beegfs_util.go:444] "msg"="Found the BeeGFS client module with lsmod"
I1219 21:45:53.147526 1 beegfs_util.go:487] "msg"="The BeeGFS client module is loaded"
E1219 21:45:53.148296 1 main.go:70] "msg"="Fatal: Failed to initialize driver" "error"="failed to handle configuration file: likely missing quotes around an integer or boolean beegfsClientConf value: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal bool into Go struct field BeegfsConfig.config.beegfsClientConf of type string" "fullError"="error unmarshaling JSON: while decoding JSON: json: cannot unmarshal bool into Go struct field BeegfsConfig.config.beegfsClientConf of type string\nlikely missing quotes around an integer or boolean beegfsClientConf value\ngithub.com/netapp/beegfs-csi-driver/pkg/beegfs.parseConfigFromFile\n\t/home/runner/work/beegfs-csi-driver/beegfs-csi-driver/pkg/beegfs/config.go:59\ngithub.com/netapp/beegfs-csi-driver/pkg/beegfs.newBeegfsDriver\n\t/home/runner/work/beegfs-csi-driver/beegfs-csi-driver/pkg/beegfs/beegfs.go:245\ngithub.com/netapp/beegfs-csi-driver/pkg/beegfs.NewBeegfsDriver\n\t/home/runner/work/beegfs-csi-driver/beegfs-csi-driver/pkg/beegfs/beegfs.go:179\nmain.handle\n\t/home/runner/work/beegfs-csi-driver/beegfs-csi-driver/cmd/beegfs-csi-driver/main.go:67\nmain.main\n\t/home/runner/work/beegfs-csi-driver/beegfs-csi-driver/cmd/beegfs-csi-driver/main.go:62\nruntime.main\n\t/opt/hostedtoolcache/go/1.23.2/x64/src/runtime/proc.go:272\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.23.2/x64/src/runtime/asm_amd64.s:1700\nfailed to handle configuration file" "goroutine"="main"
Based on the error message, it seems that there is some misconfiguration on /etc/beegfs/beegfs-client.conf. However, as I am able to execute beegfs-ctl command on node 2. I do not understand where could be error in /etc/beegfs/beegfs-client.conf.
For better describe my testing environment, I also put some detail about beegfs-controller-0 and node here.
linyuting@MacbookDeYuting beegfs-csi-driver % kubectl describe pod csi-beegfs-controller-0 -n beegfs-csi
Name: csi-beegfs-controller-0
Namespace: beegfs-csi
Priority: 0
Service Account: csi-beegfs-controller-sa
Node: ip-172-31-21-135.ec2.internal/172.31.21.135
Start Time: Fri, 20 Dec 2024 05:43:01 +0800
Labels: app=csi-beegfs-controller
apps.kubernetes.io/pod-index=0
controller-revision-hash=csi-beegfs-controller-79499f5978
statefulset.kubernetes.io/pod-name=csi-beegfs-controller-0
Annotations: kubectl.kubernetes.io/default-container: beegfs
Status: Running
IP: 172.31.21.135
IPs:
IP: 172.31.21.135
Controlled By: StatefulSet/csi-beegfs-controller
Containers:
csi-provisioner:
Container ID: containerd://0c59cf7941859f9bc5cb78241a4565f47d766951c57faddc52662977ba5d0d12
Image: registry.k8s.io/sig-storage/csi-provisioner:v3.5.0
Image ID: registry.k8s.io/sig-storage/csi-provisioner@sha256:d078dc174323407e8cc6f0f9abd4efaac5db27838f1564d0253d5e3233e3f17f
Port: <none>
Host Port: <none>
Args:
--csi-address=/csi/csi.sock
--volume-name-uuid-length=8
-v=$(LOG_LEVEL)
State: Running
Started: Fri, 20 Dec 2024 05:43:01 +0800
Ready: True
Restart Count: 0
Limits:
memory: 256Mi
Requests:
cpu: 80m
memory: 24Mi
Environment:
LOG_LEVEL: 3
Mounts:
/csi from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6sm97 (ro)
csi-resizer:
Container ID: containerd://b5adc7a4df62b547702c22f974fda21eec7fe8fc8256e2b560acb2337d98bbca
Image: registry.k8s.io/sig-storage/csi-resizer:v1.11.1
Image ID: registry.k8s.io/sig-storage/csi-resizer@sha256:a541e6cc2d8b011bb21b1d4ffec6b090e85270cce6276ee302d86153eec0af43
Port: <none>
Host Port: <none>
Args:
--csi-address=/csi/csi.sock
-v=$(LOG_LEVEL)
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Fri, 20 Dec 2024 05:57:20 +0800
Finished: Fri, 20 Dec 2024 05:57:50 +0800
Ready: False
Restart Count: 7
Limits:
memory: 500Mi
Requests:
cpu: 10m
memory: 20Mi
Environment:
LOG_LEVEL: 3
Mounts:
/csi from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6sm97 (ro)
beegfs:
Container ID: containerd://0d8000e5dac2eb888b7ecadb823d949c9372d081a0e64bf5b8f11838b81b5b1d
Image: ghcr.io/thinkparq/beegfs-csi-driver:v1.7.0
Image ID: ghcr.io/thinkparq/beegfs-csi-driver@sha256:eb7ff01259e6770d0103f219e11b7ff334c9d9102a29ea0af95a3c0d7aec1ae3
Port: <none>
Host Port: <none>
Args:
--driver-name=beegfs.csi.netapp.com
--node-id=$(KUBE_NODE_NAME)
--endpoint=unix://csi/csi.sock
--cs-data-dir=/var/lib/kubelet/plugins/beegfs.csi.netapp.com
--config-path=/csi/config/csi-beegfs-config.yaml
--connauth-path=/csi/connauth/csi-beegfs-connauth.yaml
--node-unstage-timeout=60
-v=$(LOG_LEVEL)
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Fri, 20 Dec 2024 05:58:57 +0800
Finished: Fri, 20 Dec 2024 05:58:57 +0800
Ready: False
Restart Count: 8
Limits:
memory: 256Mi
Requests:
cpu: 100m
memory: 16Mi
Environment:
KUBE_NODE_NAME: (v1:spec.nodeName)
LOG_LEVEL: 3
Mounts:
/csi from socket-dir (rw)
/csi/config from config-dir (rw)
/csi/connauth from connauth-dir (rw)
/host from host-dir (ro)
/var/lib/kubelet/plugins/beegfs.csi.netapp.com from plugin-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-6sm97 (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
host-dir:
Type: HostPath (bare host directory volume)
Path: /
HostPathType: Directory
plugin-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/beegfs.csi.netapp.com
HostPathType: DirectoryOrCreate
socket-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
config-dir:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: csi-beegfs-config-gbc9dg4htd
Optional: false
connauth-dir:
Type: Secret (a volume populated by a Secret)
SecretName: csi-beegfs-connauth-b596k67656
Optional: false
kube-api-access-6sm97:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node-role.kubernetes.io/master:NoSchedule
node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 19m default-scheduler Successfully assigned beegfs-csi/csi-beegfs-controller-0 to ip-172-31-21-135.ec2.internal
Normal Pulled 19m kubelet Container image "registry.k8s.io/sig-storage/csi-provisioner:v3.5.0" already present on machine
Normal Created 19m kubelet Created container csi-provisioner
Normal Started 19m kubelet Started container csi-provisioner
Normal Started 19m (x3 over 19m) kubelet Started container beegfs
Normal Pulled 19m (x2 over 19m) kubelet Container image "registry.k8s.io/sig-storage/csi-resizer:v1.11.1" already present on machine
Normal Created 19m (x2 over 19m) kubelet Created container csi-resizer
Normal Started 19m (x2 over 19m) kubelet Started container csi-resizer
Warning BackOff 19m (x5 over 19m) kubelet Back-off restarting failed container beegfs in pod csi-beegfs-controller-0_beegfs-csi(039e8293-9158-4edb-9558-4d0d84e39fb8)
Normal Pulled 19m (x4 over 19m) kubelet Container image "ghcr.io/thinkparq/beegfs-csi-driver:v1.7.0" already present on machine
Normal Created 19m (x4 over 19m) kubelet Created container beegfs
Warning BackOff 4m44s (x55 over 18m) kubelet Back-off restarting failed container csi-resizer in pod csi-beegfs-controller-0_beegfs-csi(039e8293-9158-4edb-9558-4d0d84e39fb8)
About logs of csi-beegfs-node, it is the same as the error message of controller.
linyuting@MacbookDeYuting beegfs-csi-driver % kubectl logs csi-beegfs-node-z55hq -n beegfs-csi
I1219 22:06:39.129310 1 beegfs_util.go:436] "msg"="Executing command" "command"=["lsmod"]
I1219 22:06:39.135415 1 beegfs_util.go:444] "msg"="Found the BeeGFS client module with lsmod"
I1219 22:06:39.135437 1 beegfs_util.go:487] "msg"="The BeeGFS client module is loaded"
E1219 22:06:39.136126 1 main.go:70] "msg"="Fatal: Failed to initialize driver" "error"="failed to handle configuration file: likely missing quotes around an integer or boolean beegfsClientConf value: error unmarshaling JSON: while decoding JSON: json: cannot unmarshal bool into Go struct field BeegfsConfig.config.beegfsClientConf of type string" "fullError"="error unmarshaling JSON: while decoding JSON: json: cannot unmarshal bool into Go struct field BeegfsConfig.config.beegfsClientConf of type string\nlikely missing quotes around an integer or boolean beegfsClientConf value\ngithub.com/netapp/beegfs-csi-driver/pkg/beegfs.parseConfigFromFile\n\t/home/runner/work/beegfs-csi-driver/beegfs-csi-driver/pkg/beegfs/config.go:59\ngithub.com/netapp/beegfs-csi-driver/pkg/beegfs.newBeegfsDriver\n\t/home/runner/work/beegfs-csi-driver/beegfs-csi-driver/pkg/beegfs/beegfs.go:245\ngithub.com/netapp/beegfs-csi-driver/pkg/beegfs.NewBeegfsDriver\n\t/home/runner/work/beegfs-csi-driver/beegfs-csi-driver/pkg/beegfs/beegfs.go:179\nmain.handle\n\t/home/runner/work/beegfs-csi-driver/beegfs-csi-driver/cmd/beegfs-csi-driver/main.go:67\nmain.main\n\t/home/runner/work/beegfs-csi-driver/beegfs-csi-driver/cmd/beegfs-csi-driver/main.go:62\nruntime.main\n\t/opt/hostedtoolcache/go/1.23.2/x64/src/runtime/proc.go:272\nruntime.goexit\n\t/opt/hostedtoolcache/go/1.23.2/x64/src/runtime/asm_amd64.s:1700\nfailed to handle configuration file" "goroutine"="main"
linyuting@MacbookDeYuting beegfs-csi-driver % kubectl describe pod csi-beegfs-node-s2x6j -n beegfs-csi
Name: csi-beegfs-node-s2x6j
Namespace: beegfs-csi
Priority: 0
Service Account: csi-beegfs-node-sa
Node: ip-172-31-21-135.ec2.internal/172.31.21.135
Start Time: Fri, 20 Dec 2024 04:54:52 +0800
Labels: app=csi-beegfs-node
controller-revision-hash=68484ffd7b
pod-template-generation=1
Annotations: kubectl.kubernetes.io/default-container: beegfs
Status: Running
IP: 172.31.21.135
IPs:
IP: 172.31.21.135
Controlled By: DaemonSet/csi-beegfs-node
Containers:
node-driver-registrar:
Container ID: containerd://856cf960850bd58bf9c4f945748789459ca59a19855f99d13e1c527281b7d720
Image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.8.0
Image ID: registry.k8s.io/sig-storage/csi-node-driver-registrar@sha256:f6717ce72a2615c7fbc746b4068f788e78579c54c43b8716e5ce650d97af2df1
Port: <none>
Host Port: <none>
Args:
--csi-address=/csi/csi.sock
--kubelet-registration-path=/var/lib/kubelet/plugins/beegfs.csi.netapp.com/csi.sock
-v=$(LOG_LEVEL)
State: Running
Started: Fri, 20 Dec 2024 04:54:54 +0800
Ready: True
Restart Count: 0
Limits:
memory: 128Mi
Requests:
cpu: 80m
memory: 10Mi
Environment:
LOG_LEVEL: 3
Mounts:
/csi from socket-dir (rw)
/registration from registration-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5bj55 (ro)
beegfs:
Container ID: containerd://61588c801bfbd569b7e5b3e9c204e57c5c853b2715360da1ca5c5e6a67a2a6ba
Image: ghcr.io/thinkparq/beegfs-csi-driver:v1.7.0
Image ID: ghcr.io/thinkparq/beegfs-csi-driver@sha256:eb7ff01259e6770d0103f219e11b7ff334c9d9102a29ea0af95a3c0d7aec1ae3
Port: 9898/TCP
Host Port: 9898/TCP
Args:
--driver-name=beegfs.csi.netapp.com
--node-id=$(KUBE_NODE_NAME)
--endpoint=unix://csi/csi.sock
--config-path=/csi/config/csi-beegfs-config.yaml
--connauth-path=/csi/connauth/csi-beegfs-connauth.yaml
-v=$(LOG_LEVEL)
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Fri, 20 Dec 2024 04:57:48 +0800
Finished: Fri, 20 Dec 2024 04:57:48 +0800
Ready: False
Restart Count: 5
Limits:
memory: 128Mi
Requests:
cpu: 100m
memory: 20Mi
Liveness: http-get http://:healthz/healthz delay=10s timeout=3s period=2s #success=1 #failure=5
Environment:
KUBE_NODE_NAME: (v1:spec.nodeName)
LOG_LEVEL: 3
Mounts:
/csi from socket-dir (rw)
/csi/config from config-dir (rw)
/csi/connauth from connauth-dir (rw)
/host from host-dir (ro)
/var/lib/kubelet/plugins/kubernetes.io/csi from plugins-mount-dir (rw)
/var/lib/kubelet/pods from pods-mount-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5bj55 (ro)
liveness-probe:
Container ID: containerd://e5865815c801983808e943b4c85fed4d186c354e77378225275636a5094a0ceb
Image: registry.k8s.io/sig-storage/livenessprobe:v2.10.0
Image ID: registry.k8s.io/sig-storage/livenessprobe@sha256:4dc0b87ccd69f9865b89234d8555d3a614ab0a16ed94a3016ffd27f8106132ce
Port: <none>
Host Port: <none>
Args:
--csi-address=/csi/csi.sock
--health-port=9898
State: Running
Started: Fri, 20 Dec 2024 04:54:57 +0800
Ready: True
Restart Count: 0
Limits:
memory: 128Mi
Requests:
cpu: 60m
memory: 20Mi
Environment: <none>
Mounts:
/csi from socket-dir (rw)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-5bj55 (ro)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
host-dir:
Type: HostPath (bare host directory volume)
Path: /
HostPathType: Directory
pods-mount-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/pods
HostPathType: DirectoryOrCreate
plugins-mount-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/kubernetes.io/csi
HostPathType: DirectoryOrCreate
registration-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins_registry
HostPathType: Directory
socket-dir:
Type: HostPath (bare host directory volume)
Path: /var/lib/kubelet/plugins/beegfs.csi.netapp.com
HostPathType: DirectoryOrCreate
config-dir:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: csi-beegfs-config-gbc9dg4htd
Optional: false
connauth-dir:
Type: Secret (a volume populated by a Secret)
SecretName: csi-beegfs-connauth-b596k67656
Optional: false
kube-api-access-5bj55:
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/disk-pressure:NoSchedule op=Exists
node.kubernetes.io/memory-pressure:NoSchedule op=Exists
node.kubernetes.io/network-unavailable:NoSchedule op=Exists
node.kubernetes.io/not-ready:NoExecute op=Exists
node.kubernetes.io/pid-pressure:NoSchedule op=Exists
node.kubernetes.io/unreachable:NoExecute op=Exists
node.kubernetes.io/unschedulable:NoSchedule op=Exists
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 3m45s default-scheduler Successfully assigned beegfs-csi/csi-beegfs-node-s2x6j to ip-172-31-21-135.ec2.internal
Normal Pulling 3m44s kubelet Pulling image "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.8.0"
Normal Pulled 3m43s kubelet Successfully pulled image "registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.8.0" in 1.01s (1.01s including waiting). Image size: 10536524 bytes.
Normal Created 3m43s kubelet Created container node-driver-registrar
Normal Started 3m43s kubelet Started container node-driver-registrar
Normal Pulling 3m43s kubelet Pulling image "ghcr.io/thinkparq/beegfs-csi-driver:v1.7.0"
Normal Pulled 3m42s kubelet Successfully pulled image "ghcr.io/thinkparq/beegfs-csi-driver:v1.7.0" in 1.448s (1.448s including waiting). Image size: 15838558 bytes.
Normal Pulling 3m42s kubelet Pulling image "registry.k8s.io/sig-storage/livenessprobe:v2.10.0"
Normal Pulled 3m41s kubelet Successfully pulled image "registry.k8s.io/sig-storage/livenessprobe:v2.10.0" in 896ms (896ms including waiting). Image size: 9579908 bytes.
Normal Created 3m41s kubelet Created container liveness-probe
Normal Started 3m40s kubelet Started container liveness-probe
Normal Created 3m27s (x3 over 3m42s) kubelet Created container beegfs
Normal Started 3m26s (x3 over 3m42s) kubelet Started container beegfs
Warning BackOff 3m10s (x6 over 3m39s) kubelet Back-off restarting failed container beegfs in pod csi-beegfs-node-s2x6j_beegfs-csi(0244cae4-a997-4d22-81e6-cb5e55821c69)
Normal Pulled 2m58s (x3 over 3m40s) kubelet Container image "ghcr.io/thinkparq/beegfs-csi-driver:v1.7.0" already present on machine
Do you have any idea what can be the reason that I can not launch my csi driver in my k8s cluster.
The text was updated successfully, but these errors were encountered:
I finally solved this issue by changing deploy/k8s/overlays/my-overlay/csi-beegfs-config.yaml as following:
# Copyright 2021 NetApp, Inc. All Rights Reserved.
# Licensed under the Apache License, Version 2.0.
# Use this file as instructed in the General Configuration section of /docs/deployment.md. See
# /deploy/k8s/examples/csi-beegfs-config.yaml for a worst-case example of what to put in this file. Kustomize will
# automatically transform this file into a correct ConfigMap readable by the deployed driver. If this file is left
# unmodified, the driver will deploy correctly with no custom configuration.
config:
beegfsClientConf:
connDisableAuthentication: "false"
It seems that I need to add quote to property connDisableAuthentication.
Thanks for pointing that out. I checked the documentation and there are a few places where it is not clear quotes are required around this parameter. I've submitted a PR to clarify this is required (#38).
I followed the tutorial mentioned in https://www.youtube.com/watch?v=uu7q_PcHUXA&ab_channel=NetAppTechCommTV. However, I am not able to launch csi-beegfs-controller and csi-beegfs-node on my kubernetes cluster as following kubectl command shown.
I have two nodes:
I am able to execute beegfs-ctl on the second node:
The second node coexist with kubernetes.
Based on the error message, it seems that there is some misconfiguration on /etc/beegfs/beegfs-client.conf. However, as I am able to execute beegfs-ctl command on node 2. I do not understand where could be error in /etc/beegfs/beegfs-client.conf.
For better describe my testing environment, I also put some detail about beegfs-controller-0 and node here.
About logs of csi-beegfs-node, it is the same as the error message of controller.
Do you have any idea what can be the reason that I can not launch my csi driver in my k8s cluster.
The text was updated successfully, but these errors were encountered: