Include wal_name in list_bucket prefix to improve performance of download_wal #876
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The current implementation of download_wal lists all files in the wal directory and filters client-side for the actual wal being requested. However since we're querying blob storage which filters by prefix we can include the wal_name in the prefix to execute this filtering on the cloud provider while still retrieving both compressed and non-compressed files.
The
list_bucket
api in thecloud_interface
currently specifies prefix for filtering without any restrictions on the prefix pointing to afolder
, and I believe all cloud providers supported by barman support filtering by any prefix as well.This change should provide a noticeable performance improvement on wal directories containing a large amount of archived wal segments.