-
Notifications
You must be signed in to change notification settings - Fork 25k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Bulk Delete API to Blob Store Interface #40250
Labels
:Distributed Coordination/Snapshot/Restore
Anything directly related to the `_snapshot/*` APIs
>enhancement
Comments
original-brownbear
added
>enhancement
:Distributed Coordination/Snapshot/Restore
Anything directly related to the `_snapshot/*` APIs
labels
Mar 20, 2019
Pinging @elastic/es-distributed |
This was referenced Mar 21, 2019
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this issue
Apr 1, 2019
* Adds Bulk delete API to blob container * Implement bulk delete API for S3 * Adjust S3Fixture to accept both path styles for bulk deletes since the S3 SDK uses both during our ITs * Closes elastic#40250
original-brownbear
added a commit
that referenced
this issue
Apr 2, 2019
* Adds Bulk delete API to blob container * Implement bulk delete API for S3 * Adjust S3Fixture to accept both path styles for bulk deletes since the S3 SDK uses both during our ITs * Closes #40250
original-brownbear
added a commit
to original-brownbear/elasticsearch
that referenced
this issue
Apr 16, 2019
* Adds Bulk delete API to blob container * Implement bulk delete API for S3 * Adjust S3Fixture to accept both path styles for bulk deletes since the S3 SDK uses both during our ITs * Closes elastic#40250
original-brownbear
added a commit
that referenced
this issue
Apr 16, 2019
gurkankaymak
pushed a commit
to gurkankaymak/elasticsearch
that referenced
this issue
May 27, 2019
* Adds Bulk delete API to blob container * Implement bulk delete API for S3 * Adjust S3Fixture to accept both path styles for bulk deletes since the S3 SDK uses both during our ITs * Closes elastic#40250
37 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
:Distributed Coordination/Snapshot/Restore
Anything directly related to the `_snapshot/*` APIs
>enhancement
Continuing from #39656 here:
Currently deleting snapshots can be a very slow process if the delete entails removing a large number of blobs from the repository (i.e. when one or more indices that consistent of many files becomes unreferenced and has to be deleted).
The current implementation of the blob store only offers synchronous and sequential deletes of these blobs, which means that one needs at least as many network calls as the number of to be deleted blobs.
In addition to being potentially very slow, this also might incur higher than necessary cost for blob stores cloud solutions that bill for each API request.
There are at least two possible improvements to be made to performance:
I think we should do the latter here and adjust
org.elasticsearch.common.blobstore.BlobContainer#deleteBlob
to accept a list of blobs to delete and leverage bulk delete APIs in implementations where possible.Note: Improving the performance of deletes will also lower the likelihood of running into stale snapshot blobs that aren't referenced by any meta data (see #13159 ).
The text was updated successfully, but these errors were encountered: