CSIProxy isn't igniting SMBGlobalMapping command

[marciogmorales]

What happened:

CSIProxy isn't igniting SMBGLobalMapping command

As the mount isn't happening on the host, it throws a timeout; this sounds to be the root cause. CSIProxy isn't igniting the command. However, I can successfully run the SMB-GlobalMapping command manually.

Also, CSIProxy isn't detecting if an existing SMBGlobalMapping is already mounted. I did the test having the SMBGlobalMapping manually mounted.

CSIProxy Log:

Log file created at: 2022/04/08 13:16:45
Running on machine: EC2AMAZ-5C7TE01
Binary: Built with gc go1.18 for windows/amd64
Log line format: [IWEF]mmdd hh:mm:ss.uuuuuu threadid file:line] msg
I0408 13:16:45.457061    2776 main.go:142] Running as a Windows service.
I0408 13:16:45.457061    2776 main.go:152] Windows Service initialized through SCM
I0408 13:16:45.516779    2776 main.go:64] Starting CSI-Proxy Server ...
I0408 13:16:45.516779    2776 main.go:65] Version: 
I0408 13:16:45.519451    2776 main.go:84] Working directories: %v[C:\var\lib\kubelet]
I0408 13:18:10.381196    2776 server.go:114] Request: PathExists with path="c:\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pv-smb\\globalmount"
I0408 13:18:10.381196    2776 server.go:231] Request: IsSymlink with path="c:\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pv-smb\\globalmount"
I0408 13:18:10.381739    2776 server.go:114] Request: PathExists with path="c:\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pv-smb\\globalmount"
I0408 13:18:10.384371    2776 server.go:153] Request: Rmdir with path="c:\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pv-smb\\globalmount"
I0408 13:18:10.384901    2776 server.go:114] Request: PathExists with path="c:\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pv-smb"
I0408 13:18:10.387008    2776 server.go:36] calling NewSmbGlobalMapping with remote path "\\\\fs-002d771bfd375aac6.marciomorales.local\\share\\windowscontainer"
I0408 13:18:11.415292    2776 server.go:69] Remote \\fs-002d771bfd375aac6.marciomorales.local\share\windowscontainer not mapped. Mapping now!
I0408 13:25:33.641010    2776 server.go:114] Request: PathExists with path="c:\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pv-smb\\globalmount"
I0408 13:25:33.641544    2776 server.go:231] Request: IsSymlink with path="c:\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pv-smb\\globalmount"
I0408 13:25:33.642078    2776 server.go:114] Request: PathExists with path="c:\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pv-smb\\globalmount"
I0408 13:25:33.642078    2776 server.go:153] Request: Rmdir with path="c:\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pv-smb\\globalmount"
I0408 13:25:33.642609    2776 server.go:114] Request: PathExists with path="c:\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pv-smb"
I0408 13:25:33.642609    2776 server.go:36] calling NewSmbGlobalMapping with remote path "\\\\fs-002d771bfd375aac6.marciomorales.local\\share\\"
I0408 13:25:34.404591    2776 server.go:69] Remote \\fs-002d771bfd375aac6.marciomorales.local\share\ not mapped. Mapping now!

kubelet log:

Events:
  Type     Reason             Age              From                     Message
  ----     ------             ----             ----                     -------
  Normal   Scheduled          2m8s             default-scheduler        Successfully assigned default/busybox-smb-76db684959-qlksd to ip-172-31-6-224.ec2.internal
  Normal   ResourceAllocated  2m8s             vpc-resource-controller  Allocated Resource vpc.amazonaws.com/PrivateIPv4Address: 172.31.27.96/19 to the pod
  Warning  FailedMount        8s               kubelet                  MountVolume.MountDevice failed for volume "pv-smb" : rpc error: code = DeadlineExceeded desc = context deadline exceeded
  Warning  FailedMount        5s               kubelet                  Unable to attach or mount volumes: unmounted volumes=[smb], unattached volumes=[smb kube-api-access-pzq67]: timed out waiting for the condition
  Warning  FailedMount        0s (x4 over 8s)  kubelet                  MountVolume.MountDevice failed for volume "pv-smb" : rpc error: code = Aborted desc = An operation with the given Volume ID eks-test-08 already exists

What you expected to happen:

CSIProxy to ignite SMBGlobalMapping command

How to reproduce it:

Using CSI-SMB-Driver.

Anything else we need to know?:

Environment:

• CSI Driver version: 1.1.1
• Kubernetes version (use kubectl version): 1.22
• OS (e.g. from /etc/os-release): Windows Server 2019
• Kernel (e.g. uname -a): 10.0.17763 N/A Build 17763
• Install tools:
• Others: containerd runtime

[mauriciopoppe]

From the logs I'm not sure if this is the part where it hangs because that's the last statement that's printed in the method, there's actually additional code that runs and we don't have any confirmation about the other parts running or not.

To narrow down the issue I'd add a klog.V(2).Infof("done calling NewSmbGlobalMapping") at the end to see if this method is done, I'm not sure why we aren't doing that yet but I've added #207

[marciogmorales]

I did some progress and fix the problem. It was a block on my Active Directory account.

However, now I'm receiving the following:

I0411 16:35:21.281672    7436 utils.go:76] GRPC call: /csi.v1.Node/NodeStageVolume
I0411 16:35:21.281672    7436 utils.go:77] GRPC request: {"secrets":"***stripped***","staging_target_path":"\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pv-smb\\globalmount","volume_capability":{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777"]}},"access_mode":{"mode":5}},"volume_context":{"source":"//fs-002d771bfd375aac6.marciomorales.local/share/"},"volume_id":"eks-test-21"}
I0411 16:35:21.281672    7436 nodeserver.go:187] NodeStageVolume: targetPath(\var\lib\kubelet\plugins\kubernetes.io\csi\pv\pv-smb\globalmount) volumeID(eks-test-21) context(map[source://fs-002d771bfd375aac6.marciomorales.local/share/]) mountflags([dir_mode=0777 file_mode=0777]) mountOptions([marciomorales\admin])
I0411 16:35:21.281672    7436 safe_mounter_windows.go:181] IsLikelyNotMountPoint: \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pv-smb\globalmount
I0411 16:35:21.281672    7436 safe_mounter_windows.go:238] Exists path: \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pv-smb\globalmount
I0411 16:35:21.282739    7436 safe_mounter_windows.go:238] Exists path: \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pv-smb\globalmount
I0411 16:35:21.282739    7436 smb_common_windows.go:74] Removing path: \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pv-smb\globalmount
I0411 16:35:21.282739    7436 safe_mounter_windows.go:151] Remove directory: \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pv-smb\globalmount
I0411 16:35:21.283281    7436 safe_mounter_windows.go:71] SMBMount: remote path: //fs-002d771bfd375aac6.marciomorales.local/share/ local path: \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pv-smb\globalmount
I0411 16:35:21.283281    7436 safe_mounter_windows.go:238] Exists path: \var\lib\kubelet\plugins\kubernetes.io\csi\pv\pv-smb
I0411 16:35:21.283281    7436 safe_mounter_windows.go:111] begin to mount \\fs-002d771bfd375aac6.marciomorales.local\share\ on c:\var\lib\kubelet\plugins\kubernetes.io\csi\pv\pv-smb\globalmount
E0411 16:35:22.768588    7436 utils.go:81] GRPC error: rpc error: code = Internal desc = volume(eks-test-21) mount "//fs-002d771bfd375aac6.marciomorales.local/share/" on "\\var\\lib\\kubelet\\plugins\\kubernetes.io\\csi\\pv\\pv-smb\\globalmount" failed with smb mapping failed with error: rpc error: code = Unknown desc = NewSmbGlobalMapping failed. output: "New-SmbGlobalMapping : The parameter is incorrect. \r\nAt line:1 char:190\r\n+ ... ser, $PWord;New-SmbGlobalMapping -RemotePath $Env:smbremotepath -Cred ...\r\n+                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n    + CategoryInfo          : InvalidArgument: (MSFT_SmbGlobalMapping:ROOT/Microsoft/...mbGlobalMapping) [New-SmbGloba \r\n   lMapping], CimException\r\n    + FullyQualifiedErrorId : Windows System Error 87,New-SmbGlobalMapping\r\n \r\n", err: exit status 1

Appears to be encapsulation the SMB-GlobalMapping command wrong. What should the exactly kubectl create secret command for a domain join?

This is what I'm running:

kubectl create secret generic smbcreds --from-literal username=DOMAIN\USER --from-literal password="MYPASSWORD"

Also tried:

kubectl create secret generic smbcreds --from-literal domain=MYDOMAIN --from-literal username=USER --from-literal password="MYPASSWORD"

[marciogmorales]

Ok, it worked.
Somehow on the CSI-SMB-Driver when the target SMB Share is a Windows Server, it requires the Worker node to be domain joined. However I successfully manually mounted it on a Workgroup using Powershell.

This is the error: Windows System Error 87

[andyzhangx]

Ok, it worked. Somehow on the CSI-SMB-Driver when the target SMB Share is a Windows Server, it requires the Worker node to be domain joined. However I successfully manually mounted it on a Workgroup using Powershell.

This is the error: Windows System Error 87

@mauriciopoppe so the issue solved after worker node joined the domain? seems that's a manual step.

[marciogmorales]

At first I just joined the worker node in the domain due to the Windows System Error 87 and it worked, however my next Worker node which is Workgroup worked as well.

Today I’ll investigate two things:
1 - It sounds like the kubectl create secret command, on the password string, it didn’t work when I use ” between the password. (I’ll investigate this path).

2 - It worked when using the latest 1.6.0 CSI-Driver.