ECK: Document recovery from failed volume upsize #3459
+11
−0
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Michael Montgomery <mmontg1@gmail.com>
naemono
requested review from
barkbay,
Copilot,
kvalliyurnatt,
pebrc and
rhr323
There was a problem hiding this comment.
Pull Request Overview
This PR adds documentation to help users recover from failed Elasticsearch volume expansion operations in ECK (Elastic Cloud on Kubernetes). The change addresses user issues where volume expansion failures can leave deployments in an unrecoverable state.
- Adds a new troubleshooting section for volume expansion failures
- Documents the recommended recovery approach using nodeSet renaming
- Provides specific error message and solution context
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Signed-off-by: Michael Montgomery <mmontg1@gmail.com>
🔍 Preview links for changed docs |
kilfoyle
reviewed
Oct 14, 2025
|
|
||
| ## If a volume expansion failed [k8s-common-problems-volume-failed-expansion] | ||
|
|
||
| If you attempted an expansion of an Elasticsearch volume via its [volume claim template](/deploy-manage/deploy/cloud-on-k8s/volume-claim-templates.md#k8s-volume-claim-templates-update), you may have encountered scenarios where the operation failed such as Azure not allowing volume expansion without shutting down the Virtual Machine to which it is attached. If you try to adjust the volume claim template back to the original size you will encounter a failure: |
There was a problem hiding this comment.
Suggested change
| If you attempted an expansion of an Elasticsearch volume via its [volume claim template](/deploy-manage/deploy/cloud-on-k8s/volume-claim-templates.md#k8s-volume-claim-templates-update), you may have encountered scenarios where the operation failed such as Azure not allowing volume expansion without shutting down the Virtual Machine to which it is attached. If you try to adjust the volume claim template back to the original size you will encounter a failure: | |
| If you attempted an expansion of an {{es}} volume via its [volume claim template](/deploy-manage/deploy/cloud-on-k8s/volume-claim-templates.md#k8s-volume-claim-templates-update), you may have encountered scenarios where the operation failed such as Azure not allowing volume expansion without shutting down the Virtual Machine to which it is attached. If you try to adjust the volume claim template back to the original size you will encounter a failure: |
kilfoyle
reviewed
Oct 14, 2025
| Failed to apply spec change: handle volume expansion: decreasing storage size is not supported: an attempt was made to decrease storage size for claim elasticsearch-data | ||
| ``` | ||
| In this scenario the best course of action is to rename the existing `nodeSet` to a new name while simultaneously updating the volume claim template to the original size. This operation will bring a new `StatefulSet` online while moving all existing indices to the new volumes and will delete the old `StatefulSet` and its volumes once the operation is complete. |
There was a problem hiding this comment.
Suggested change
| In this scenario the best course of action is to rename the existing `nodeSet` to a new name while simultaneously updating the volume claim template to the original size. This operation will bring a new `StatefulSet` online while moving all existing indices to the new volumes and will delete the old `StatefulSet` and its volumes once the operation is complete. | |
| In this scenario the best course of action is to rename the existing `nodeSet` to a new name while simultaneously updating the volume claim template to the original size. This operation will bring a new `StatefulSet` online while moving all existing indices to the new volumes, and will delete the old `StatefulSet` and its volumes once the operation is complete. |
kilfoyle
approved these changes
Oct 14, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In elastic/cloud-on-k8s#4467 it's noted that some users are dealing with volume expansion failure issues, and documenting how to recover from this situation would be helpful. This is the attempt to update that documentation.
After merge