-
Notifications
You must be signed in to change notification settings - Fork 49
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature] Allow druid to delete and trigger recreation of PVCs on-demand #481
Comments
API-wise this approach seems procedural which we probably want to avoid. Do we need to expose a function-style operation or isn't it possible to stick to Kubernetes's well known desired/actual state paradigm, i.e. API changes trigger changes that Druid performs in order to transform the actual to the desired one? |
That was my first thought as well. The issue here is that any change to Of course we can safeguard this operation with something similar to Also, Etcd resource does not currently store any state for the volumes it uses (unless we plan to do that using the |
Also, other use-cases of this functionality of being able to programatically delete a PVC on-demand are:
/cc @abdasgupta |
Pro:
Con:
So, whether we automate the volume replacement (scripts have risks themselves, so a well-tested and safe-guarded druid implementation may still be the better option, even if we are all fearful of volume deletion code), I would still like to raise the question, whether we can offer a way to do that w/o a downtime as initially hoped/planned (HA became the prerequisite for volume replacement for critical clusters and scaling down the stateful set, would defy this goal). |
This would lead to inconsistency in the |
Yes, exactly. I was commenting on the sentence "Scale down the statefulset (if it isn't already scaled down)" and if you have the capacity to automate gardener/gardener-extension-provider-aws#646 (comment) that would be most welcome. |
/assign |
/assign @seshachalam-yv |
Blocked until #588 is implemented |
Feature (What you would like to be added):
I would like to leverage the data restoration functionality of the etcd-backup-restore sidecar (in both single- and multi-node etcd clusters) to allow druid to delete PVC(s) on-demand and trigger recreation of the PVC(s) later.
Motivation (Why is this needed?):
There are use-cases where user wants to switch storage class of the PVC to a better one, or to change the volume size of the etcd disk, or maybe to switch to encrypted disks. Today, any changes to the volume configuration in the
Etcd
CR leads to reconciliation errors, as the existing statefulset forbids any updates to thevolumeClaimTemplate
, as can be seen from gardener/gardener-extension-provider-aws#646 (comment). I would like druid to catch such errors, and possibly even gracefully handle them in a certain way, if I would specify it via maybe annotation(s) on theEtcd
CR.Approach/Hint to the implement solution (optional):
Introduce annotations in the
Etcd
CR, something likedruid.gardener.cloud/operation: "recreate-pvc/etcd-main-0"
anddruid.gardener.cloud/operation: recreate-sts
(if necessary), that druid sees and does the following:recreate-sts
annotationAlso enhance the
immutableFieldUpdate
function introduced in #408 and see how this can be leveraged for the newdruid.gardener.cloud/operation: recreate-sts
annotation, if necessary.cc @unmarshall @timuthy @vlerenc
The text was updated successfully, but these errors were encountered: