feat: add smb file restore from snapshot #1466

umagnus · 2023-09-26T09:01:35Z

What type of PR is this?

/kind feature

What this PR does / why we need it:

feat: add smb file restore from snapshot by using azcopy

Which issue(s) this PR fixes:

Fixes #136

Requirements:

uses conventional commit messages
includes documentation
adds unit tests
tested upgrade from previous version

Special notes for your reviewer:

enable different storageclass smb file restore, but only be enabled in the same sub because csi cannot get different subs id in sourceVolumeContent snapshotId, but it can get account name in it. Also add e2e test case should create a pod, write to its pv, take a volume snapshot, overwrite data in original pv, create another pod from the snapshot, use another storage class, and read unaltered original data from original pv[file.csi.azure.com] to test this case.

NAME                              STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS                                         AGE
pvc-azurefile                     Bound    pvc-38a798d8-2a95-439f-98b6-340963369a33   100Gi      RWX            azurefile-csi                                        47h
pvc-azurefile-snapshot-restored   Bound    pvc-21d8fdf8-d9c8-4dfc-a0d1-3883b8489cca   100Gi      RWX            azurefile-6843-file.csi.azure.com-dynamic-sc-zdnfp   43h

csi logs

I0202 07:38:14.720539       1 utils.go:100] GRPC call: /csi.v1.Controller/CreateVolume
I0202 07:38:14.720559       1 utils.go:101] GRPC request: {"capacity_range":{"required_bytes":1073741824000},"name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","parameters":{"csi.storage.k8s.io/pv/name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","csi.storage.k8s.io/pvc/name":"pvc-azurefile-snapshot-restored","csi.storage.k8s.io/pvc/namespace":"default","skuName":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","mfsymlinks","cache=strict","nosharesock","actimeo=30"]}},"access_mode":{"mode":5}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0###default#2024-02-02T07:04:56.0000000Z"}}}}
I0202 07:38:14.981775       1 azure_storageaccount.go:322] found a matching account f958d93650b164b0d93c052 type Premium_LRS location eastus
I0202 07:38:15.347448       1 controllerserver.go:561] begin to create file share(pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03) on account(f958d93650b164b0d93c052) type(Premium_LRS) subID() rg(MC_aks-smbtest_group_aks-smbtest_eastus) location() size(1000) protocol(SMB)
I0202 07:38:15.783626       1 controllerserver.go:1376] generate sas token for account(f958d93650b164b0d93c052)
I0202 07:38:15.794328       1 controllerserver.go:1040] azcopy job status: NotFound, copy percent: %, error: <nil>
I0202 07:38:15.794347       1 controllerserver.go:1047] copy fileshare pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0(snapshot: 2024-02-02T07:04:56.0000000Z) to pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03
W0202 07:56:15.821338       1 controllerserver.go:1059] CopyFileShare(MC_aks-smbtest_group_aks-smbtest_eastus, f958d93650b164b0d93c052, pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03) failed with error(timeout waiting for copy fileshare pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0(snapshot: 2024-02-02T07:04:56.0000000Z) to pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03 succeed, copy percent: 20.0%): 
I0202 07:56:15.821495       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=1081.100546864 request="azurefile_csi_driver_controller_create_volume_from_snapshot" resource_group="mc_aks-smbtest_group_aks-smbtest_eastus" subscription_id="" source="file.csi.azure.com" volumeid="" result_code="failed_csi_driver_controller_create_volume_from_snapshot"
E0202 07:56:15.821519       1 utils.go:105] GRPC error: timeout waiting for copy fileshare pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0(snapshot: 2024-02-02T07:04:56.0000000Z) to pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03 succeed, copy percent: 20.0%
I0202 07:56:15.835425       1 utils.go:100] GRPC call: /csi.v1.Controller/CreateVolume
I0202 07:56:15.835444       1 utils.go:101] GRPC request: {"capacity_range":{"required_bytes":1073741824000},"name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","parameters":{"csi.storage.k8s.io/pv/name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","csi.storage.k8s.io/pvc/name":"pvc-azurefile-snapshot-restored","csi.storage.k8s.io/pvc/namespace":"default","skuName":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","mfsymlinks","cache=strict","nosharesock","actimeo=30"]}},"access_mode":{"mode":5}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0###default#2024-02-02T07:04:56.0000000Z"}}}}
I0202 07:56:16.040436       1 controllerserver.go:561] begin to create file share(pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03) on account(f958d93650b164b0d93c052) type(Premium_LRS) subID() rg(MC_aks-smbtest_group_aks-smbtest_eastus) location() size(1000) protocol(SMB)
I0202 07:56:16.353439       1 controllerserver.go:1376] generate sas token for account(f958d93650b164b0d93c052)
I0202 07:56:16.383047       1 controllerserver.go:1040] azcopy job status: Running, copy percent: 20.0%, error: <nil>
I0202 07:56:16.383189       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.54713538 request="azurefile_csi_driver_controller_create_volume_from_snapshot" resource_group="mc_aks-smbtest_group_aks-smbtest_eastus" subscription_id="" source="file.csi.azure.com" volumeid="" result_code="failed_csi_driver_controller_create_volume_from_snapshot"
E0202 07:56:16.383213       1 utils.go:105] GRPC error: An existed azcopy job is running, please wait for it to complete
I0202 07:56:16.837452       1 utils.go:100] GRPC call: /csi.v1.Controller/CreateVolume
I0202 07:56:16.837515       1 utils.go:101] GRPC request: {"capacity_range":{"required_bytes":1073741824000},"name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","parameters":{"csi.storage.k8s.io/pv/name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","csi.storage.k8s.io/pvc/name":"pvc-azurefile-snapshot-restored","csi.storage.k8s.io/pvc/namespace":"default","skuName":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","mfsymlinks","cache=strict","nosharesock","actimeo=30"]}},"access_mode":{"mode":5}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0###default#2024-02-02T07:04:56.0000000Z"}}}}
I0202 07:56:16.940987       1 controllerserver.go:561] begin to create file share(pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03) on account(f958d93650b164b0d93c052) type(Premium_LRS) subID() rg(MC_aks-smbtest_group_aks-smbtest_eastus) location() size(1000) protocol(SMB)
I0202 07:56:17.132965       1 controllerserver.go:1376] generate sas token for account(f958d93650b164b0d93c052)
I0202 07:56:17.165058       1 controllerserver.go:1040] azcopy job status: Running, copy percent: 20.0%, error: <nil>
I0202 07:56:17.165244       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.327512003 request="azurefile_csi_driver_controller_create_volume_from_snapshot" resource_group="mc_aks-smbtest_group_aks-smbtest_eastus" subscription_id="" source="file.csi.azure.com" volumeid="" result_code="failed_csi_driver_controller_create_volume_from_snapshot"
E0202 07:56:17.165267       1 utils.go:105] GRPC error: An existed azcopy job is running, please wait for it to complete
I0202 07:56:21.179437       1 utils.go:100] GRPC call: /csi.v1.Controller/CreateVolume
I0202 07:56:21.179584       1 utils.go:101] GRPC request: {"capacity_range":{"required_bytes":1073741824000},"name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","parameters":{"csi.storage.k8s.io/pv/name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","csi.storage.k8s.io/pvc/name":"pvc-azurefile-snapshot-restored","csi.storage.k8s.io/pvc/namespace":"default","skuName":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","mfsymlinks","cache=strict","nosharesock","actimeo=30"]}},"access_mode":{"mode":5}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0###default#2024-02-02T07:04:56.0000000Z"}}}}
I0202 07:56:21.305654       1 controllerserver.go:561] begin to create file share(pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03) on account(f958d93650b164b0d93c052) type(Premium_LRS) subID() rg(MC_aks-smbtest_group_aks-smbtest_eastus) location() size(1000) protocol(SMB)
I0202 07:56:21.932902       1 controllerserver.go:1376] generate sas token for account(f958d93650b164b0d93c052)
I0202 07:56:21.962335       1 controllerserver.go:1040] azcopy job status: Running, copy percent: 20.0%, error: <nil>
I0202 07:56:21.962391       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.782584073 request="azurefile_csi_driver_controller_create_volume_from_snapshot" resource_group="mc_aks-smbtest_group_aks-smbtest_eastus" subscription_id="" source="file.csi.azure.com" volumeid="" result_code="failed_csi_driver_controller_create_volume_from_snapshot"
E0202 07:56:21.962412       1 utils.go:105] GRPC error: An existed azcopy job is running, please wait for it to complete
I0202 07:56:29.979872       1 utils.go:100] GRPC call: /csi.v1.Controller/CreateVolume
I0202 07:56:29.979896       1 utils.go:101] GRPC request: {"capacity_range":{"required_bytes":1073741824000},"name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","parameters":{"csi.storage.k8s.io/pv/name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","csi.storage.k8s.io/pvc/name":"pvc-azurefile-snapshot-restored","csi.storage.k8s.io/pvc/namespace":"default","skuName":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","mfsymlinks","cache=strict","nosharesock","actimeo=30"]}},"access_mode":{"mode":5}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0###default#2024-02-02T07:04:56.0000000Z"}}}}
I0202 07:56:30.036367       1 controllerserver.go:561] begin to create file share(pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03) on account(f958d93650b164b0d93c052) type(Premium_LRS) subID() rg(MC_aks-smbtest_group_aks-smbtest_eastus) location() size(1000) protocol(SMB)
I0202 07:56:30.288995       1 controllerserver.go:1376] generate sas token for account(f958d93650b164b0d93c052)
I0202 07:56:30.347651       1 controllerserver.go:1040] azcopy job status: Running, copy percent: 20.0%, error: <nil>
I0202 07:56:30.347725       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.367587463 request="azurefile_csi_driver_controller_create_volume_from_snapshot" resource_group="mc_aks-smbtest_group_aks-smbtest_eastus" subscription_id="" source="file.csi.azure.com" volumeid="" result_code="failed_csi_driver_controller_create_volume_from_snapshot"
E0202 07:56:30.347747       1 utils.go:105] GRPC error: An existed azcopy job is running, please wait for it to complete
I0202 07:56:46.362968       1 utils.go:100] GRPC call: /csi.v1.Controller/CreateVolume
I0202 07:56:46.362990       1 utils.go:101] GRPC request: {"capacity_range":{"required_bytes":1073741824000},"name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","parameters":{"csi.storage.k8s.io/pv/name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","csi.storage.k8s.io/pvc/name":"pvc-azurefile-snapshot-restored","csi.storage.k8s.io/pvc/namespace":"default","skuName":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","mfsymlinks","cache=strict","nosharesock","actimeo=30"]}},"access_mode":{"mode":5}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0###default#2024-02-02T07:04:56.0000000Z"}}}}
I0202 07:56:46.529748       1 controllerserver.go:561] begin to create file share(pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03) on account(f958d93650b164b0d93c052) type(Premium_LRS) subID() rg(MC_aks-smbtest_group_aks-smbtest_eastus) location() size(1000) protocol(SMB)
I0202 07:56:46.820895       1 controllerserver.go:1376] generate sas token for account(f958d93650b164b0d93c052)
I0202 07:56:46.868274       1 controllerserver.go:1040] azcopy job status: Running, copy percent: 20.0%, error: <nil>
I0202 07:56:46.868934       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.50510625 request="azurefile_csi_driver_controller_create_volume_from_snapshot" resource_group="mc_aks-smbtest_group_aks-smbtest_eastus" subscription_id="" source="file.csi.azure.com" volumeid="" result_code="failed_csi_driver_controller_create_volume_from_snapshot"
E0202 07:56:46.868955       1 utils.go:105] GRPC error: An existed azcopy job is running, please wait for it to complete
I0202 07:57:18.884280       1 utils.go:100] GRPC call: /csi.v1.Controller/CreateVolume
I0202 07:57:18.884447       1 utils.go:101] GRPC request: {"capacity_range":{"required_bytes":1073741824000},"name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","parameters":{"csi.storage.k8s.io/pv/name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","csi.storage.k8s.io/pvc/name":"pvc-azurefile-snapshot-restored","csi.storage.k8s.io/pvc/namespace":"default","skuName":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","mfsymlinks","cache=strict","nosharesock","actimeo=30"]}},"access_mode":{"mode":5}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0###default#2024-02-02T07:04:56.0000000Z"}}}}
I0202 07:57:18.958625       1 controllerserver.go:561] begin to create file share(pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03) on account(f958d93650b164b0d93c052) type(Premium_LRS) subID() rg(MC_aks-smbtest_group_aks-smbtest_eastus) location() size(1000) protocol(SMB)
I0202 07:57:19.165210       1 controllerserver.go:1376] generate sas token for account(f958d93650b164b0d93c052)
I0202 07:57:19.193582       1 controllerserver.go:1040] azcopy job status: Running, copy percent: 20.0%, error: <nil>
I0202 07:57:19.193639       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.308890875 request="azurefile_csi_driver_controller_create_volume_from_snapshot" resource_group="mc_aks-smbtest_group_aks-smbtest_eastus" subscription_id="" source="file.csi.azure.com" volumeid="" result_code="failed_csi_driver_controller_create_volume_from_snapshot"
E0202 07:57:19.193655       1 utils.go:105] GRPC error: An existed azcopy job is running, please wait for it to complete
I0202 07:58:23.207230       1 utils.go:100] GRPC call: /csi.v1.Controller/CreateVolume
I0202 07:58:23.207380       1 utils.go:101] GRPC request: {"capacity_range":{"required_bytes":1073741824000},"name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","parameters":{"csi.storage.k8s.io/pv/name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","csi.storage.k8s.io/pvc/name":"pvc-azurefile-snapshot-restored","csi.storage.k8s.io/pvc/namespace":"default","skuName":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","mfsymlinks","cache=strict","nosharesock","actimeo=30"]}},"access_mode":{"mode":5}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0###default#2024-02-02T07:04:56.0000000Z"}}}}
I0202 07:58:23.332958       1 controllerserver.go:561] begin to create file share(pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03) on account(f958d93650b164b0d93c052) type(Premium_LRS) subID() rg(MC_aks-smbtest_group_aks-smbtest_eastus) location() size(1000) protocol(SMB)
I0202 07:58:23.557224       1 controllerserver.go:1376] generate sas token for account(f958d93650b164b0d93c052)
I0202 07:58:23.589042       1 controllerserver.go:1040] azcopy job status: Running, copy percent: 20.0%, error: <nil>
I0202 07:58:23.589099       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.381526988 request="azurefile_csi_driver_controller_create_volume_from_snapshot" resource_group="mc_aks-smbtest_group_aks-smbtest_eastus" subscription_id="" source="file.csi.azure.com" volumeid="" result_code="failed_csi_driver_controller_create_volume_from_snapshot"
E0202 07:58:23.589149       1 utils.go:105] GRPC error: An existed azcopy job is running, please wait for it to complete
I0202 08:00:31.612168       1 utils.go:100] GRPC call: /csi.v1.Controller/CreateVolume
I0202 08:00:31.612192       1 utils.go:101] GRPC request: {"capacity_range":{"required_bytes":1073741824000},"name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","parameters":{"csi.storage.k8s.io/pv/name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","csi.storage.k8s.io/pvc/name":"pvc-azurefile-snapshot-restored","csi.storage.k8s.io/pvc/namespace":"default","skuName":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","mfsymlinks","cache=strict","nosharesock","actimeo=30"]}},"access_mode":{"mode":5}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0###default#2024-02-02T07:04:56.0000000Z"}}}}
I0202 08:00:31.746105       1 controllerserver.go:561] begin to create file share(pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03) on account(f958d93650b164b0d93c052) type(Premium_LRS) subID() rg(MC_aks-smbtest_group_aks-smbtest_eastus) location() size(1000) protocol(SMB)
I0202 08:00:32.069130       1 controllerserver.go:1376] generate sas token for account(f958d93650b164b0d93c052)
I0202 08:00:32.106192       1 controllerserver.go:1040] azcopy job status: Running, copy percent: 20.0%, error: <nil>
I0202 08:00:32.106256       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.493872805 request="azurefile_csi_driver_controller_create_volume_from_snapshot" resource_group="mc_aks-smbtest_group_aks-smbtest_eastus" subscription_id="" source="file.csi.azure.com" volumeid="" result_code="failed_csi_driver_controller_create_volume_from_snapshot"
E0202 08:00:32.106285       1 utils.go:105] GRPC error: An existed azcopy job is running, please wait for it to complete
I0202 08:04:48.123723       1 utils.go:100] GRPC call: /csi.v1.Controller/CreateVolume
I0202 08:04:48.123745       1 utils.go:101] GRPC request: {"capacity_range":{"required_bytes":1073741824000},"name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","parameters":{"csi.storage.k8s.io/pv/name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","csi.storage.k8s.io/pvc/name":"pvc-azurefile-snapshot-restored","csi.storage.k8s.io/pvc/namespace":"default","skuName":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","mfsymlinks","cache=strict","nosharesock","actimeo=30"]}},"access_mode":{"mode":5}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0###default#2024-02-02T07:04:56.0000000Z"}}}}
I0202 08:04:48.403043       1 controllerserver.go:561] begin to create file share(pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03) on account(f958d93650b164b0d93c052) type(Premium_LRS) subID() rg(MC_aks-smbtest_group_aks-smbtest_eastus) location() size(1000) protocol(SMB)
I0202 08:04:49.023654       1 controllerserver.go:1376] generate sas token for account(f958d93650b164b0d93c052)
I0202 08:04:49.067010       1 controllerserver.go:1040] azcopy job status: Running, copy percent: 20.0%, error: <nil>
I0202 08:04:49.067069       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.94280594 request="azurefile_csi_driver_controller_create_volume_from_snapshot" resource_group="mc_aks-smbtest_group_aks-smbtest_eastus" subscription_id="" source="file.csi.azure.com" volumeid="" result_code="failed_csi_driver_controller_create_volume_from_snapshot"
E0202 08:04:49.067134       1 utils.go:105] GRPC error: An existed azcopy job is running, please wait for it to complete
I0202 08:08:14.539375       1 utils.go:100] GRPC call: /csi.v1.Controller/CreateVolume
I0202 08:08:14.539398       1 utils.go:101] GRPC request: {"capacity_range":{"required_bytes":1073741824000},"name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","parameters":{"csi.storage.k8s.io/pv/name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","csi.storage.k8s.io/pvc/name":"pvc-azurefile-snapshot-restored","csi.storage.k8s.io/pvc/namespace":"default","skuName":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","mfsymlinks","cache=strict","nosharesock","actimeo=30"]}},"access_mode":{"mode":5}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0###default#2024-02-02T07:04:56.0000000Z"}}}}
I0202 08:08:14.754147       1 controllerserver.go:561] begin to create file share(pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03) on account(f958d93650b164b0d93c052) type(Premium_LRS) subID() rg(MC_aks-smbtest_group_aks-smbtest_eastus) location() size(1000) protocol(SMB)
I0202 08:08:15.022329       1 controllerserver.go:1376] generate sas token for account(f958d93650b164b0d93c052)
I0202 08:08:15.065648       1 controllerserver.go:1040] azcopy job status: Running, copy percent: 40.0%, error: <nil>
I0202 08:08:15.065698       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.526114391 request="azurefile_csi_driver_controller_create_volume_from_snapshot" resource_group="mc_aks-smbtest_group_aks-smbtest_eastus" subscription_id="" source="file.csi.azure.com" volumeid="" result_code="failed_csi_driver_controller_create_volume_from_snapshot"
E0202 08:08:15.065715       1 utils.go:105] GRPC error: An existed azcopy job is running, please wait for it to complete
I0202 08:09:49.084252       1 utils.go:100] GRPC call: /csi.v1.Controller/CreateVolume
I0202 08:09:49.084273       1 utils.go:101] GRPC request: {"capacity_range":{"required_bytes":1073741824000},"name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","parameters":{"csi.storage.k8s.io/pv/name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","csi.storage.k8s.io/pvc/name":"pvc-azurefile-snapshot-restored","csi.storage.k8s.io/pvc/namespace":"default","skuName":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","mfsymlinks","cache=strict","nosharesock","actimeo=30"]}},"access_mode":{"mode":5}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0###default#2024-02-02T07:04:56.0000000Z"}}}}
I0202 08:09:49.304469       1 controllerserver.go:561] begin to create file share(pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03) on account(f958d93650b164b0d93c052) type(Premium_LRS) subID() rg(MC_aks-smbtest_group_aks-smbtest_eastus) location() size(1000) protocol(SMB)
I0202 08:09:49.623094       1 controllerserver.go:1376] generate sas token for account(f958d93650b164b0d93c052)
I0202 08:09:49.652297       1 controllerserver.go:1040] azcopy job status: Running, copy percent: 40.0%, error: <nil>
I0202 08:09:49.652351       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.567849873 request="azurefile_csi_driver_controller_create_volume_from_snapshot" resource_group="mc_aks-smbtest_group_aks-smbtest_eastus" subscription_id="" source="file.csi.azure.com" volumeid="" result_code="failed_csi_driver_controller_create_volume_from_snapshot"
E0202 08:09:49.652368       1 utils.go:105] GRPC error: An existed azcopy job is running, please wait for it to complete
I0202 08:14:49.668920       1 utils.go:100] GRPC call: /csi.v1.Controller/CreateVolume
I0202 08:14:49.668943       1 utils.go:101] GRPC request: {"capacity_range":{"required_bytes":1073741824000},"name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","parameters":{"csi.storage.k8s.io/pv/name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","csi.storage.k8s.io/pvc/name":"pvc-azurefile-snapshot-restored","csi.storage.k8s.io/pvc/namespace":"default","skuName":"Premium_LRS"},"volume_capabilities":[{"AccessType":{"Mount":{"mount_flags":["dir_mode=0777","file_mode=0777","mfsymlinks","cache=strict","nosharesock","actimeo=30"]}},"access_mode":{"mode":5}}],"volume_content_source":{"Type":{"Snapshot":{"snapshot_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0###default#2024-02-02T07:04:56.0000000Z"}}}}
I0202 08:14:49.863733       1 controllerserver.go:561] begin to create file share(pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03) on account(f958d93650b164b0d93c052) type(Premium_LRS) subID() rg(MC_aks-smbtest_group_aks-smbtest_eastus) location() size(1000) protocol(SMB)
I0202 08:14:50.313310       1 controllerserver.go:1376] generate sas token for account(f958d93650b164b0d93c052)
I0202 08:14:50.323557       1 controllerserver.go:1040] azcopy job status: Completed, copy percent: 100.0%, error: <nil>
I0202 08:14:50.323605       1 controllerserver.go:593] create file share pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03 on storage account f958d93650b164b0d93c052 successfully
I0202 08:14:50.323650       1 azure_metrics.go:115] "Observed Request Latency" latency_seconds=0.654503239 request="azurefile_csi_driver_controller_create_volume_from_snapshot" resource_group="mc_aks-smbtest_group_aks-smbtest_eastus" subscription_id="" source="file.csi.azure.com" volumeid="MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03###default" result_code="succeeded"
I0202 08:14:50.323740       1 utils.go:107] GRPC response: {"volume":{"capacity_bytes":1073741824000,"content_source":{"Type":{"Snapshot":{"snapshot_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-f27cf6c3-236d-4b46-95c8-eb7e60019be0###default#2024-02-02T07:04:56.0000000Z"}}},"volume_context":{"csi.storage.k8s.io/pv/name":"pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03","csi.storage.k8s.io/pvc/name":"pvc-azurefile-snapshot-restored","csi.storage.k8s.io/pvc/namespace":"default","secretnamespace":"default","skuName":"Premium_LRS"},"volume_id":"MC_aks-smbtest_group_aks-smbtest_eastus#f958d93650b164b0d93c052#pvc-adb84e31-5c2d-49eb-9a2a-deaa614e2c03###default"}}

Release note:

none

k8s-ci-robot · 2023-09-26T09:01:44Z

Hi @umagnus. Thanks for your PR.

I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

cvvz · 2023-09-27T07:51:55Z

/ok-to-test

andyzhangx · 2023-10-09T02:19:19Z

pkg/azurefile/controllerserver.go

+	for {
+		select {
+		case <-timeTick:
+			jobState, percent, err := getAzcopyJob(dstFileShareName)


can you combine the logic with copy volume, the select ... case logic is quite similar

I think you could use same function for both restoreFromSansphot and copyVolume

It's because srcPath is different from copy volume and snapshot has other logic for different storage account, should we combine the logic just for azcopy?

combine the logic in a copyFileShareByAzcopy func

test/e2e/dynamic_provisioning_test.go

test/e2e/testsuites/dynamically_provisioned_volume_snapshot_tester.go

umagnus · 2023-10-11T01:24:43Z

/retest

umagnus · 2023-10-11T07:50:59Z

/retest

umagnus · 2023-10-12T06:15:41Z

/retest

andyzhangx · 2024-04-06T02:14:04Z

/test pull-azurefile-csi-driver-e2e-capz

umagnus · 2024-04-07T00:45:57Z

/retest

umagnus · 2024-04-08T01:43:13Z

/retest

umagnus · 2024-04-08T01:43:23Z

/test pull-azurefile-csi-driver-e2e-capz

umagnus · 2024-04-09T01:54:14Z

/retest

umagnus · 2024-04-09T06:03:55Z

/test pull-azurefile-csi-driver-e2e-capz

umagnus · 2024-04-09T07:44:24Z

/test pull-azurefile-csi-driver-e2e-capz

andyzhangx · 2024-04-15T01:49:01Z

test/e2e/dynamic_provisioning_test.go

+		test.Run(ctx, cs, snapshotrcs, ns)
+	})
+
+	ginkgo.It("should create a pod, write to its pv, take a volume snapshot, overwrite data in original pv, create another pod from the snapshot, and read unaltered original data from original pv[file.csi.azure.com]", func(ctx ginkgo.SpecContext) {


I think we could remove this test case to make test pass first

done, remove it

andyzhangx · 2024-04-15T03:54:36Z

/test pull-azurefile-csi-driver-e2e-capz

k8s-ci-robot · 2024-04-15T04:00:38Z

@umagnus: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-azurefile-csi-driver-e2e-vmss	`94f81d2`	link	true	`/test pull-azurefile-csi-driver-e2e-vmss`
pull-azurefile-csi-driver-e2e	`4e36304`	link	true	`/test pull-azurefile-csi-driver-e2e`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

andyzhangx · 2024-04-15T04:03:23Z

/retest

andyzhangx

/lgtm

k8s-ci-robot · 2024-04-15T06:00:22Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: andyzhangx, umagnus

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [andyzhangx]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

andyzhangx · 2024-04-15T06:04:44Z

pls cherrypick to release-1.30, it may depend on #1657 merge first

andyzhangx · 2024-04-15T06:14:38Z

/cherrypick release-1.30

k8s-infra-cherrypick-robot · 2024-04-15T06:15:15Z

@andyzhangx: #1466 failed to apply on top of branch "release-1.30":

Applying: smb restore from snapshot
Using index info to reconstruct a base tree...
A	charts/latest/azurefile-csi-driver-v0.0.0.tgz
M	deploy/csi-azurefile-controller.yaml
Falling back to patching base and 3-way merge...
Auto-merging deploy/csi-azurefile-controller.yaml
CONFLICT (modify/delete): charts/latest/azurefile-csi-driver-v0.0.0.tgz deleted in HEAD and modified in smb restore from snapshot. Version smb restore from snapshot of charts/latest/azurefile-csi-driver-v0.0.0.tgz left in tree.
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0001 smb restore from snapshot
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherrypick release-1.30

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

kaovilai · 2024-08-26T14:47:07Z

Would this gets cherry-picked to 1.29 as well?

andyzhangx · 2024-08-26T14:51:21Z

Would this gets cherry-picked to 1.29 as well?

@kaovilai no, we don't have plan to support this feature on csi driver v1.29

k8s-ci-robot added kind/feature Categorizes issue or PR as related to a new feature. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 26, 2023

k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Sep 26, 2023

k8s-ci-robot requested review from cvvz and gnufied September 26, 2023 09:01

k8s-ci-robot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Sep 27, 2023

umagnus force-pushed the smb_snapshot_restore branch from f41df5e to 60e9312 Compare September 27, 2023 08:13

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Sep 27, 2023

umagnus force-pushed the smb_snapshot_restore branch 2 times, most recently from 7f6efab to 236d63a Compare September 28, 2023 01:38

andyzhangx mentioned this pull request Oct 4, 2023

support azure file snapshot vmware-tanzu/velero#3151

Open

umagnus force-pushed the smb_snapshot_restore branch from 8a26a08 to 6ba13ec Compare October 8, 2023 02:27

andyzhangx reviewed Oct 9, 2023

View reviewed changes

test/e2e/testsuites/dynamically_provisioned_volume_snapshot_tester.go Outdated Show resolved Hide resolved

umagnus force-pushed the smb_snapshot_restore branch from bc90501 to 39033de Compare October 10, 2023 04:27

umagnus force-pushed the smb_snapshot_restore branch from 2ced8e0 to 3374e63 Compare October 12, 2023 02:13

k8s-ci-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Oct 12, 2023

umagnus force-pushed the smb_snapshot_restore branch 4 times, most recently from 63b1b7d to 931c2bd Compare October 13, 2023 03:29

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 6, 2024

umagnus force-pushed the smb_snapshot_restore branch 2 times, most recently from d68ca98 to 11f9a5c Compare April 6, 2024 11:47

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 6, 2024

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 10, 2024

andyzhangx reviewed Apr 15, 2024

View reviewed changes

smb restore from snapshot

7049605

umagnus force-pushed the smb_snapshot_restore branch from ccad31e to 7049605 Compare April 15, 2024 02:12

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Apr 15, 2024

andyzhangx approved these changes Apr 15, 2024

View reviewed changes

k8s-ci-robot assigned andyzhangx Apr 15, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 15, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 15, 2024

k8s-ci-robot merged commit 1ce6146 into kubernetes-sigs:master Apr 15, 2024
24 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add smb file restore from snapshot #1466

feat: add smb file restore from snapshot #1466

umagnus commented Sep 26, 2023 •

edited

Loading

k8s-ci-robot commented Sep 26, 2023

cvvz commented Sep 27, 2023

andyzhangx Oct 9, 2023

andyzhangx Oct 9, 2023

umagnus Oct 10, 2023

umagnus Oct 12, 2023

umagnus commented Oct 11, 2023

umagnus commented Oct 11, 2023

umagnus commented Oct 12, 2023

andyzhangx commented Apr 6, 2024

umagnus commented Apr 7, 2024

umagnus commented Apr 8, 2024

umagnus commented Apr 8, 2024

umagnus commented Apr 9, 2024

umagnus commented Apr 9, 2024

umagnus commented Apr 9, 2024

andyzhangx Apr 15, 2024

umagnus Apr 15, 2024

andyzhangx commented Apr 15, 2024

k8s-ci-robot commented Apr 15, 2024 •

edited

Loading

andyzhangx commented Apr 15, 2024

andyzhangx left a comment

k8s-ci-robot commented Apr 15, 2024

andyzhangx commented Apr 15, 2024

andyzhangx commented Apr 15, 2024

k8s-infra-cherrypick-robot commented Apr 15, 2024

kaovilai commented Aug 26, 2024

andyzhangx commented Aug 26, 2024

feat: add smb file restore from snapshot #1466

feat: add smb file restore from snapshot #1466

Conversation

umagnus commented Sep 26, 2023 • edited Loading

k8s-ci-robot commented Sep 26, 2023

cvvz commented Sep 27, 2023

andyzhangx Oct 9, 2023

Choose a reason for hiding this comment

andyzhangx Oct 9, 2023

Choose a reason for hiding this comment

umagnus Oct 10, 2023

Choose a reason for hiding this comment

umagnus Oct 12, 2023

Choose a reason for hiding this comment

umagnus commented Oct 11, 2023

umagnus commented Oct 11, 2023

umagnus commented Oct 12, 2023

andyzhangx commented Apr 6, 2024

umagnus commented Apr 7, 2024

umagnus commented Apr 8, 2024

umagnus commented Apr 8, 2024

umagnus commented Apr 9, 2024

umagnus commented Apr 9, 2024

umagnus commented Apr 9, 2024

andyzhangx Apr 15, 2024

Choose a reason for hiding this comment

umagnus Apr 15, 2024

Choose a reason for hiding this comment

andyzhangx commented Apr 15, 2024

k8s-ci-robot commented Apr 15, 2024 • edited Loading

andyzhangx commented Apr 15, 2024

andyzhangx left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Apr 15, 2024

andyzhangx commented Apr 15, 2024

andyzhangx commented Apr 15, 2024

k8s-infra-cherrypick-robot commented Apr 15, 2024

kaovilai commented Aug 26, 2024

andyzhangx commented Aug 26, 2024

umagnus commented Sep 26, 2023 •

edited

Loading

k8s-ci-robot commented Apr 15, 2024 •

edited

Loading