cannot update snapshot metadata

[gman0]

My attempt of creating a new VolumeSnapshot from a PVC source resulted in following error message in external-snapshotter:

I0717 18:09:11.377904       1 connection.go:180] GRPC call: /csi.v1.Controller/CreateSnapshot
I0717 18:09:11.377925       1 connection.go:181] GRPC request: {"name":"snapshot-bd0540a0-1912-44f8-a04c-2e69ab1c21c6","secrets":"***stripped***","source_volume_id":"b44ffa38-5d65-4346-9265-807d9c966d6f"}
I0717 18:09:11.439795       1 reflector.go:235] github.com/kubernetes-csi/external-snapshotter/pkg/client/informers/externalversions/factory.go:117: forcing resync
I0717 18:09:11.577934       1 connection.go:183] GRPC response: {"snapshot":{"creation_time":{"seconds":1560416046},"ready_to_use":true,"size_bytes":1073741824,"snapshot_id":"bfe22f76-c3af-48a8-a326-6fdc3e5a747d","source_volume_id":"b44ffa38-5d65-4346-9265-807d9c966d6f"}}
I0717 18:09:11.579576       1 connection.go:184] GRPC error: <nil>
I0717 18:09:11.584450       1 snapshotter.go:81] CSI CreateSnapshot: snapshot-bd0540a0-1912-44f8-a04c-2e69ab1c21c6 driver name [nfs.manila.csi.openstack.org] snapshot ID [bfe22f76-c3af-48a8-a326-6fdc3e5a747d] time stamp [&{1560416046 0 {} [] 0}] size [1073741824] readyToUse [true]
I0717 18:09:11.584530       1 snapshot_controller.go:640] Created snapshot: driver nfs.manila.csi.openstack.org, snapshotId bfe22f76-c3af-48a8-a326-6fdc3e5a747d, creationTime 2019-06-13 08:54:06 +0000 UTC, size 1073741824, readyToUse true
I0717 18:09:11.584563       1 snapshot_controller.go:645] createSnapshot [default/new-nfs-share-snap]: trying to update snapshot creation timestamp
I0717 18:09:11.584604       1 snapshot_controller.go:825] updating VolumeSnapshot[]default/new-nfs-share-snap, readyToUse true, timestamp 2019-06-13 08:54:06 +0000 UTC
I0717 18:09:11.588727       1 snapshot_controller.go:650] failed to update snapshot default/new-nfs-share-snap creation timestamp: snapshot controller failed to update default/new-nfs-share-snap on API server: the server could not find the requested resource (put volumesnapshots.snapshot.storage.k8s.io new-nfs-share-snap)

The snapshot is successfully created by the driver, but external-snapshotter is having trouble updating the snapshot object metadata.

I'm using external-snapshotter v1.2.0-0-gb3f591d8 in k8s 1.15.0 running with VolumeSnapshotDataSource=true feature gate. The previous version of external-snapshotter, 1.1.0, works just fine though. Is this a regression or rather mis-configuration on my part? Always happy to debug more or provide more logs! Thanks!

[gman0]

/kind bug

[xing-yang]

@gman0 Thanks for reporting the issue! Do you see this "failed to update snapshot default/new-nfs-share-snap creation timestamp" appear multiple times in the log? By default it should retry 5 times.

[xing-yang]

/assign @zhucan

[zhucan]

@gman0 can paste your rbac yaml file for snapshot?

[gman0]

@xing-yang here's the whole log https://pastebin.com/8TqMZqwt
@zhucan i've just copy-pasted RBAC rules from this repo

apiVersion: v1
kind: ServiceAccount
metadata:
  name: openstack-manila-csi-controllerplugin
  labels:
    app: openstack-manila-csi
    component: controllerplugin
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: openstack-manila-csi-controllerplugin
  labels:
    app: openstack-manila-csi
    component: controllerplugin
aggregationRule:
  clusterRoleSelectors:
    - matchLabels:
        rbac.manila.csi.openstack.org/aggregate-to-openstack-manila-csi-controllerplugin: "true"
rules: []
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: openstack-manila-csi-controllerplugin-rules
  labels:
    app: openstack-manila-csi
    component: controllerplugin
    rbac.manila.csi.openstack.org/aggregate-to-openstack-manila-csi-controllerplugin: "true"
rules:
  - apiGroups: [""]
    resources: ["nodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["secrets"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["list", "watch", "create", "update", "patch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["csinodes"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshotcontents"]
    verbs: ["create", "get", "list", "watch", "update", "delete"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["snapshot.storage.k8s.io"]
    resources: ["volumesnapshots/status"]
    verbs: ["update"]
  - apiGroups: ["apiextensions.k8s.io"]
    resources: ["customresourcedefinitions"]
    verbs: ["create", "list", "watch", "delete", "get", "update"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: openstack-manila-csi-controllerplugin
  labels:
    app: openstack-manila-csi
    component: controllerplugin
subjects:
  - kind: ServiceAccount
    name: openstack-manila-csi-controllerplugin
    namespace: default
roleRef:
  kind: ClusterRole
  name: openstack-manila-csi-controllerplugin
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: openstack-manila-csi-controllerplugin
  labels:
    app: openstack-manila-csi
    component: controllerplugin
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "watch", "list", "delete", "update", "create"]
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list", "watch", "create", "delete"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: openstack-manila-csi-controllerplugin
  labels:
    app: openstack-manila-csi
    component: controllerplugin
subjects:
  - kind: ServiceAccount
    name: openstack-manila-csi-controllerplugin
    namespace: default
roleRef:
  kind: Role
  name: openstack-manila-csi-controllerplugin
  apiGroup: rbac.authorization.k8s.io

[xing-yang]

@gman0 In v1.2.0, we added a new feature to support status subresource. In order for that to work, you'll have to deploy the newer version of the snapshot CRD's. I think since you have already installed snapshot CRD v1.1.0 in your k8s cluster, it will not be installed again when you deploy external-snapshotter v1.2.0. Therefore the snapshot controller and the CRD's are out of sync.

Is it possible for you to restart your k8s cluster and deploy external-snapshotter 1.2.0+ and test again?

We'll fix this so it is backward compatible.

[zhucan]

@gman0 @xing-yang I have tested it in v1.15.0, I have already installed snapshot( image's version is v1.0.1) CRD v1.0.1 in my k8s cluster and successful snapshot creation, but when I upgrade the image's version to v1.2.0, the CRD will not be upgraded, it used old CRD, so I delete the snapshot pod and old CRD and rebuild snapshot pod, It can works.

It's no needed to restart k8s cluster, only delete the crd.

@xing-yang Maybe we should delete the old CRD when creating snapshot pod that if the CRD version is different with the image's version?

[gman0]

@xing-yang @zhucan ah I see, I'll try it out today, thanks!

[xing-yang]

@zhucan Sure. Please work on a fix. Thanks.

[gman0]

@xing-yang @zhucan removed the volumesnapshot CRDs, all is working now! Thank you both! :) A fix would be nice though :p Closing.

[xing-yang]

@gman0 I'm going to re-open this issue to keep track of the bug fix. Once it is fixed, we can close this again.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Sathishkunisai]

/reopen
ceph/ceph:v14.2.11
quay.io/cephcsi/cephcsi:v2.1.2
quay.io/k8scsi/csi-attacher:v2.1.0
quay.io/k8scsi/csi-node-driver-registrar:v1.2.0
quay.io/k8scsi/csi-provisioner:v1.6.0
quay.io/k8scsi/csi-resizer:v0.4.0
quay.io/k8scsi/csi-snapshotter:v2.1.0
rook/ceph:v1.3.9

kubectl volumensnapshots shows "ReadyToUse false". Please suggest whether the version of snapshotter I am using is correct,

[k8s-ci-robot]

@Sathishkunisai: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen
ceph/ceph:v14.2.11
quay.io/cephcsi/cephcsi:v2.1.2
quay.io/k8scsi/csi-attacher:v2.1.0
quay.io/k8scsi/csi-node-driver-registrar:v1.2.0
quay.io/k8scsi/csi-provisioner:v1.6.0
quay.io/k8scsi/csi-resizer:v0.4.0
quay.io/k8scsi/csi-snapshotter:v2.1.0
rook/ceph:v1.3.9

kubectl volumensnapshots shows "ReadyToUse false". Please suggest whether the version of snapshotter I am using is correct,

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[xing-yang]

I don't think what you had is the same problem reported by this very old issue.
Can you try v2.1.1? We are actually going to cut a 2.2.0 release very soon.

[Sathishkunisai]

Thanks @xing-yang . To give more insight, my cluster is running in Azure Kubernetes (AKS), using Azure Disk to mount rook volumes (OSD) with feature gate is enabled in API Server [ volumesnapshot=true ] . Followed the procedure here in the link as well.
https://github.com/kubernetes-sigs/azuredisk-csi-driver/blob/master/docs/install-csi-driver-master.md

apart from that CSI_ENABLE_SNAPSHOTTER: "true" is enabled in operator.yaml

what am i missing ? Please advise

[Sathishkunisai]

@xing-yang upgraded to 2.1.1 result is same

NAME READYTOUSE SOURCEPVC SOURCESNAPSHOTCONTENT RESTORESIZE SNAPSHOTCLASS SNAPSHOTCONTENT CREATIONTIME AGE

datadir-mongo-pvc-snapshot-15 false datadir-mongo-replica-mongodb-replicaset-0 csi-rbdplugin-snapclass 3h7m
datadir-mongo-pvc-snapshot-17 false datadir-mongo-replica-mongodb-replicaset-1 csi-rbdplugin-snapclass 28s

[Sathishkunisai]

apiVersion: snapshot.storage.k8s.io/v1beta1
deletionPolicy: Delete
driver: rook-ceph.rbd.csi.ceph.com
kind: VolumeSnapshotClass
metadata
name: csi-rbdplugin-snapclass
parameters:
clusterID: rook-ceph
csi.storage.k8s.io/snapshotter-secret-name: rook-ceph-csi
csi.storage.k8s.io/snapshotter-secret-namespace: rook-ceph

[Madhu-1]

@Sathishkunisai as this is more specific to ceph I would suggest you open issue in a rook and try to get help in rook slack