volume leak after CreateVolume started and before PV creation

[pohly]

In the following sequence of events a volume can leak:

• external-provisioner issues a CreateVolume call for a PVC
• external-provisioner is stopped before it creates the PV
• the PVC is deleted
• external-provisioner is restarted

That volume provisioning started is not recorded anywhere, so the restarted external-provisioner doesn't know that it needs to continue with it.

[pohly]

One possible solution is to set a finalizer for the PVC before issuing the CreateVolume call and removing it after creating the PV.

[pohly]

/kind bug

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[pohly]

/remove-lifecycle stale
/lifecycle frozen

[bertinatto]

/assign

[bertinatto]

/remove-lifecycle stale

[BroYongVeryBusy]

One possible solution is to set a finalizer for the PVC before issuing the CreateVolume call and removing it after creating the PV.

I have a question:
If user delete finalizer by "kubelet edit pvc ”, this method is useless.

[pohly]

There's no perfect protection against user mistakes. If a user force-deletes or removes finalizers, they need to be aware that they are working against the system and accept the consequences.

[BroYongVeryBusy]

There's no perfect protection against user mistakes. If a user force-deletes or removes finalizers, they need to be aware that they are working against the system and accept the consequences.

however，if user and storage provisioner are on different side.
user can use this，to make storage provisioner to get volume leak.

[pohly]

Isn't the storage provisioner charging the user for such a leaked volume?

If they can't do that, then they shouldn't grant their untrusted users the permission to create or modify PVCs.

[koct9i]

I've just found monstrous kludge in openebs lvm-localpv for this issue.

For more robust protection against race provisioner could create stub PV before calling CreateVolume. There is PV state for that: Phase=Pending. But pv_controller does not respect this phase and will try to bind PVC to unready PV. Or probably I've read state machine in wrong way.

Also, provisioner could report leaked volumes in some way. Maybe even create PV for them.
For example, "external-health-monitoring" could easily detect unknown volumes in output of ListVolumes.

[Madhu-1]

Do we have any fix for this issue?

I have hit this problem with clones. if the PVC is deleted when the clone is in progress and the volume snapshot is also deleted, if there is any provisioner restart DeleteVolume will never get called, and also snapshot cannot be deleted if a driver doesn't support snapshot deletion when the clone is created from the snapshot.

@msau42 @xing-yang @pohly ^^

[msau42]

I think it would be some significant amount of API complexity to support a stub PV. Maybe we can explore something with CRDs. @jsafrane wdyt?