-
Notifications
You must be signed in to change notification settings - Fork 62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docker EE 2.0 with UCP : Error in creating snapshot. #390
Comments
'size' parameter is not allowed for creating a snapshot. However, size parameter seems to be a requirement by K8S. |
Same behavior is seen while importing the volume. |
@dockerciuser could you tell what is the size with which the snapshot gets created when size is not specified and marked validate==false? |
@nilangekarss , The size of snapshot is same as the size of parent volume with flag validate=false |
@nilangekarss |
@imran-ansari @wdurairaj @prablr79 We should allow the size parameter to be passed along with the virtualCopyOf and Import in our plugin but this parameter is ignored inside the code. The PVC is associated with a StorageClass and PVC has a required parameter spec.resources[storage]. Clone accepts size which is specified during PVC creation hence we do not see any error while creating a clone of a volume. Hence due to the dependency on K8S, we should remove the restriction on size while creating snap, schedule, and import functionality. |
This bug was raised while executing the tests on kubernetes 1.8.11. As a workaround we were able to create snapshot / importVol with flag --validate=false When tried to import volume got following error - [root@kuber-master STORAGE-CLASS]# kubectl create -f pvc-importvol.yaml --validate=false Used following YAMLS - [root@kuber-master STORAGE-CLASS]# cat sc-importvol.yamlkind: StorageClass
|
@dockerciuser , @imran-ansari , i believe we can allow size in -o virtualCopyOf parameter in the request validator and in inside the code we have to ignore processing for size, since the snapshot size is same as that of volume size. Will require a code fix. |
Yes. Code fix + Documentation for Docker users. |
I think this fix will be required for snapshot-schedule as well. |
Even for -o importVol size should be supported. |
Ideally this needed to be fixed in Dory/Doryd |
I totally forgot that this option will be used in a storageClass,
So, we should'nt ideally use this in a storageclass. |
@wdurairaj @imran-ansari , I am able to create the snapshot with latest plugin. When tried to create the snapshot schedule got error. 05:44:53 dockervol.go:262: unable to create docker volume using sc-snapschedule-566bb67b-ebe7-11e8-917a-6cc21739c360 & map[virtualCopyOf:FC-VOLUME mountConflictDelay:30 scheduleFrequency:10 * * * * scheduleName:schedule-tushar snapshotPrefix:tushar] - Post http://unix/VolumeDriver.Create: http: ContentLength=217 with Body length 0 response - &{{ map[]} } The storage class YAML which I used is -
|
Could you please check if you are able to create snapshot schedule directly using Docker with the same parameters? |
@imran-ansari , yes with docker command able to create the snapshot schedule. /var/log/messages snippet - |
I somewhat suspect the |
Hi Tushar, Can you check how this volume got created ?
Is the above volume is created using a storage class ? |
Actually as per my analysis, even scheduleName can't used in a storage class for the same reason, we can't have
You can see the 'doc_schedule' already exists. |
you can confirm the above theory, my creatig a new storageclass with schedulename as different from the already existing ones in 3par, but you will see only the first schedule based on a SC class will work correctly. So, it's an expected behaviour. |
@wdurairaj , Did more experimentation. My observation is as - -> You are right, the snapshot is created on 3PAR
But the PVC status remains "pending"(Expecting as Bound) this bit confusing. Also saw following error in dory logs - 08:50:08 provisioner.go:237: statusLogger: provision chains=1, delete chains=0, parked chains=0, ids tracked=1, connection=valid That means functionality is working as expected but it is giving the unwanted error and the " Pending " status. |
I do agree with your statements @dockerciuser , But if we say our guideline would be to just use the snapshot scheduling only as part of static provisioning (as part of PV,POD only) and not relevant for dynamic provisioning (using SC) would that suffice ? |
@dockerciuser @wdurairaj are we thinking of fix here ? shall we live with the document for this scenario, how customer can handle this issue. I believe other scenarios we are able to create the snapshot scheduling without any issues.. kindly confirm. |
@dockerciuser @wdurairaj While creating a snapshot schedule we use ssh connection, if the ssh response from the array is slow the docker http client will timeout in 30 seconds. If the plugin is able to response in a proper manner in a limited time it is observed that snapshot schedule gets created properly(client doesn't time out and proper success response will be returned from plugin side). When we enable the plugin the default client http timeout is set to 30 seconds. @dockerciuser , Tushar can you try creating a schedule on 3PAR array which has a good ssh response time? Also please try to create a schedule on a problematic array which has bad ssh response time. The response time of docker http client can be increased up to 2 min while enabling the plugin(But IT LOOKS like to be limited to only plugin enable). I believe, when dory/plugin sends an error response while creating a pvc, it is expected for a pvc to go in a pending state. Please correct me if this is not a correct statement. Few of the moby links: |
Fix duplicate schedule name when -o scheduleName is passed in SC (issue #390)
Created the snapshot schedule successfully on Kubernetes setup with both the arrays 10.50.3.9 and 10.50.3.24.
YAMLS used – [root@kuber-master tushar]# cat sc_sch_tushar.ymlkind: StorageClass
|
Based on Tushar's previous update, I'm closing this bug. |
* Fix Issue hpe-storage#390, Allow 'size' in snapshot options * Updated usage doc * Fix issue hpe-storage#534 - invalid config entry creates session leak
* Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Added space between words * Ability to create a regular volume from a replicated backend * Fixed pep8 issue regarding redundant back slashes * Pep8 Fix attempt 2: Indentation * Update hpe_storage_api.py * Fixed an issue when a regular volume is mounted while the backend is replication enabled * Fixed pep8 line too long issues * Not treating as a replication backend if the volume created is not replicated volume * Allowing the non replicated volume from a replicated backend to be successfully imported to docker volume plugin * Added back some required checks * Removed unwanted space * Update .travis.yml * Fix for issue 518-encrypted password decrytion fails when the passphrase length is of 16,24,32 characters * Fix for issue 502 (#555) * Added needed check to see mgr object is available or not (#559) * Feature: Add logging to file -- Include changes for Pull Request #563 in v300 branch (#569) * Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Feature: Add support for RotatingFileHandler in logging * Fix Issue #534 (#576) * Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Fix issue #534 - invalid config entry creates session leak * Fix issue #513 on v300 branch (#583) * Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Fixed issue #513 -Added rollback to mount flow for any cleanup in case of any failure -Added validation for fsOwner * Pep8 fixed * Use deferred thread for processing REST calls in twistd * Fixed msg creation * Retry on lock exception * Another attempt on processing lock failed exception * Changes in mount_volume to avoid lock during mount conflict check * Fix _is_vol_mounted_on_this_node definition * Minor change * Backport pull request #650 and related changes * Implemented blocking Etcd lock + Eviction fixes merged * Fix problem with mount entry check * Returning multiple enums from _is_vol_mounted_on_this_node + inspect output to have volume id * Expect node_mount_info to be absent for the first mount -Also removed dead code * path_info to be handled as JSON object + handled stale mount_id in reboot case * Fix for UTs * Replaced path.path with path * Fixed snap related TC * PEP8 errors fixed * Added more information to the logs * For UT 3pardcv.log location changed * Added check for manager-list initialization * Removed redundant code * Removed duplicate functions from fileutil As part of merge process, fileutil ended up having two duplicate functions. Fixed it. Also UT needed to use un-deferred thread code to avoid handling multi-threaded UTs. * Fixed UTs for File * Added exception handling for mount_dir() * Adopted 3.2 async initialization fix required for UT * Reintroduced sleep of 3 secs * Corrected usage of sleep() call * Disabled detailed logging due to Travis CI log size restriction * Pep8 fix * Fix for issue #735 * Fixed removal of redundant old_path_info entries * Added missing argument to rollback call * Removed code that was added to look for iscsi devices Ideally, we should remove this file altogether... to be taken up later
* Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Added space between words * Ability to create a regular volume from a replicated backend * Fixed pep8 issue regarding redundant back slashes * Pep8 Fix attempt 2: Indentation * Update hpe_storage_api.py * Fixed an issue when a regular volume is mounted while the backend is replication enabled * Fixed pep8 line too long issues * Not treating as a replication backend if the volume created is not replicated volume * Allowing the non replicated volume from a replicated backend to be successfully imported to docker volume plugin * Added back some required checks * Removed unwanted space * Update .travis.yml * Fix for issue 518-encrypted password decrytion fails when the passphrase length is of 16,24,32 characters * Fix for issue 502 (#555) * Added needed check to see mgr object is available or not (#559) * Feature: Add logging to file -- Include changes for Pull Request #563 in v300 branch (#569) * Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Feature: Add support for RotatingFileHandler in logging * Fix Issue #534 (#576) * Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Fix issue #534 - invalid config entry creates session leak * Fix issue #513 on v300 branch (#583) * Fix Issue #390, Allow 'size' in snapshot options * Updated usage doc * Fixed issue #513 -Added rollback to mount flow for any cleanup in case of any failure -Added validation for fsOwner * Pep8 fixed * Use deferred thread for processing REST calls in twistd * Fixed msg creation * Retry on lock exception * Another attempt on processing lock failed exception * Changes in mount_volume to avoid lock during mount conflict check * Fix _is_vol_mounted_on_this_node definition * Minor change * Backport pull request #650 and related changes * Implemented blocking Etcd lock + Eviction fixes merged * Fix problem with mount entry check * Returning multiple enums from _is_vol_mounted_on_this_node + inspect output to have volume id * Expect node_mount_info to be absent for the first mount -Also removed dead code * path_info to be handled as JSON object + handled stale mount_id in reboot case * Fix for UTs * Replaced path.path with path * Fixed snap related TC * PEP8 errors fixed * Added more information to the logs * For UT 3pardcv.log location changed * Added check for manager-list initialization * Removed redundant code * Removed duplicate functions from fileutil As part of merge process, fileutil ended up having two duplicate functions. Fixed it. Also UT needed to use un-deferred thread code to avoid handling multi-threaded UTs. * Fixed UTs for File * Added exception handling for mount_dir() * Adopted 3.2 async initialization fix required for UT * Reintroduced sleep of 3 secs * Corrected usage of sleep() call * Disabled detailed logging due to Travis CI log size restriction * Pep8 fix * Fix for issue #735 * Fixed removal of redundant old_path_info entries * Added missing argument to rollback call * Removed code that was added to look for iscsi devices Ideally, we should remove this file altogether... to be taken up later * pyparsing ImportError fix * Changed setuptools version to 41.0.0 41.0.0 was used by v3.1 of plugin and was * Device remapping fix On reboot, the volumes that were mapped to the multipath devices earlier are remapped to different devices. This fix handles that case. * Fixed lock name * Fixed PEP8 issues * Missed a PEP8 conformance * Fixed log statement
While creating the snapshot got an error due to size parameter.
Below are the detailed steps which I followed -
Created storage class -
[root@CSSOSBE03-B11 new_yaml]# cat sc.yml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: test-sc
provisioner: hpe.com/hpe
parameters:
size: "17"
[root@CSSOSBE03-B11 new_yaml]# kubectl create -f sc.yml
storageclass "test-sc" created
[root@CSSOSBE03-B11 new_yaml]# kubectl get sc
NAME PROVISIONER
test-sc hpe.com/hpe
Created PVC -
[root@CSSOSBE03-B11 new_yaml]# cat pvc.yml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-test
spec:
accessModes:
resources:
requests:
storage: 20Gi
storageClassName: test-sc
[root@CSSOSBE03-B11 new_yaml]# kubectl create -f pvc.yml
persistentvolumeclaim "pvc-test" created
[root@CSSOSBE03-B11 new_yaml]# docker volume ls
DRIVER VOLUME NAME
hpe FP-VOLUME
hpe test-sc-9c1413bb-db39-11e8-a0ae-0242ac110010
Created new storage class required for snapshot -
[root@CSSOSBE03-B11 new_yaml]# cat sc_2.yml
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
name: test-sc-snp
provisioner: hpe.com/hpe
parameters:
virtualCopyOf: test-sc-9c1413bb-db39-11e8-a0ae-0242ac110010
[root@CSSOSBE03-B11 new_yaml]# kubectl create -f sc_2.yml
storageclass "test-sc-snp" created
[root@CSSOSBE03-B11 new_yaml]# cat pvc_2.yml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-test-snp
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 20Gi
storageClassName: test-sc-snp
ERROR -
01:27:51 dockervol.go:267: unable to create docker volume using test-sc-snp-d2300f9a-db3a-11e8-a0ae-0242ac110010 & map[virtualCopyOf:test-sc-9c1413bb-db39-11e8-a0ae-0242ac110010 size:20 mountConflictDelay:30] - Invalid input received: Invalid option(s) ['size'] specified for operation create snapshot. Please check help for usage.
Removed the size parameter as -
[root@CSSOSBE03-B11 new_yaml]# cat pvc_2.yml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-test-snp
spec:
accessModes:
- ReadWriteOnce
storageClassName: test-sc-snp
ERROR -
[root@CSSOSBE03-B11 new_yaml]# kubectl create -f pvc_2.yml
The PersistentVolumeClaim "pvc-test-snp" is invalid: spec.resources[storage]: Required value
The value of size parameter kept as blank (null)
[root@CSSOSBE03-B11 new_yaml]# cat pvc_2.yml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: pvc-test-snp
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage:
ERROR -
[root@CSSOSBE03-B11 new_yaml]# kubectl create -f pvc_2.yml
error: error validating "pvc_2.yml": error validating data: unknown object type "nil" in PersistentVolumeClaim.spec.resources.requests.storage; if you choose to ignore these errors, turn validation off with --validate=false
With the flag "validate=false" able to create the snapshot.
[root@CSSOSBE03-B11 new_yaml]# kubectl create -f pvc_2.yml --validate=false
persistentvolumeclaim "pvc-test-snp" created
[root@CSSOSBE03-B11 new_yaml]# docker volume ls
DRIVER VOLUME NAME
hpe test-sc-9c1413bb-db39-11e8-a0ae-0242ac110010
hpe test-sc-snp-58a9c097-db3e-11e8-a0ae-0242ac110010
The text was updated successfully, but these errors were encountered: