Support VolumeSnapshot feature from Kubernetes

[tamalsaha]

Here is the scenario:

Cloud provider supports volume snapshotting. The CSI driver for that cloud provider implements this feature. https://kubernetes.io/docs/concepts/storage/volume-snapshots/

Now, instead of using restic, Stash uses the volume snapshotting crds to take backups. How will the restore process work?

Limitations of this approach:

• This requires CSI driver. None of the CSI drivers are production ready for major cloud providers. https://github.com/kubernetes-sigs?utf8=%E2%9C%93&q=csi&type=&language=
• VolumeSnapshot CRDs are all v1alpha1. We can't use v1alpha1 resources in production set ups. This will take 6+ months to get GA (at least, probably much longer).
• Only limited to supported cloud providers.

Having said all these, it will be worth thinking about the design for this, since we don't want to break Stash CRD compatibility between v1beta1 and v1.

[tamalsaha]

• https://kubernetes.io/blog/2018/10/09/introducing-volume-snapshot-alpha-for-kubernetes/
• https://kubernetes.io/blog/2019/01/17/update-on-volume-snapshot-alpha-for-kubernetes/

[tamalsaha]

Example: https://github.com/kubernetes-sigs/gcp-compute-persistent-disk-csi-driver

[hossainemruz]

Requirements:

• CSI driver must support VolumeSnpahot. List of CSI drivers that support VolumeSnapshot:https://kubernetes.io/blog/2019/01/17/update-on-volume-snapshot-alpha-for-kubernetes/
• It is alpha feature. Must be enabled using --feature-gates=VolumeSnapshotDataSource=true flag in Kubernetes API server.

Limitation:

• Restore can be done only in new PVC. Currently not possible to restore in existing PVC.
• Does not support in-place restore.
• No snapshot consistency guarantees beyond any guarantees provided by storage system (e.g. crash consistency)

[tamalsaha]

• Application-centric backup/restore
• Snapshot scheduling
• Retention policy
• Clarify that VolumeSnapshots live in project silos
• Restore without application downtime.
• Clone stateful application
• Take backup for one replica and recover into 3 replica (MongoDB fs backup)

Video Demo: https://www.youtube.com/watch?v=SG_Yy-73q0g

[tamalsaha]

xref: kubernetes/enhancements#989

@hossainemruz , K8s is adding support to clone a PVC. Alpha support planned for 1.15 release.

[vijay-dcrust]

Hi, i am getting below error while stashing postgres db:-
Error: exit status 1, reason: Fatal: unable to save snapshot: snapshot is empty

Any ideas or if someone can redirect me to correct thread.

[tamalsaha]

@vijay-dcrust , which CSI driver are you using? And what is your Kubernetes version?

[vijay-dcrust]

@tamalsaha , I am using kubernetes volumes(local backend) with kubernetes server version 1.14.3 and client version 1.16.3 to store the backup. I use rook.ceph.io/block for postgres db storage volume.

[tamalsaha]

@vijay-dcrust , we are upgrading Stash to support the new v1beta1 VolumeSnapshot api. I see that rook currently supports v1alpha1 VolumeSnapshpot api. https://github.com/rook/rook/blob/master/Documentation/ceph-csi-drivers.md

We intend to support v1beta1 api going forward. So, once rook adds support for v1beta1 api, we are going to circle back and see what the issue is.

[vijay-dcrust]

@tamalsaha , I think this error is not due to CSI driver since i am taking the postgres backup using stash functions and tasks like postgres-backup-11.2. I am facing this error with certain installation of postgres not with all. I think stash performs logical backup using functions and tasks. I am not able to figure out the source of error that i am getting from the logs generated by the sidecar pod logs.

[hossainemruz]

Already added in #787