Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

barman-cloud list & recover failed if there are glacier storage class in s3 endpoint url #332

Closed
bdoublet91 opened this issue Mar 29, 2021 · 1 comment · Fixed by #363
Closed
Assignees
Labels
Milestone

Comments

@bdoublet91
Copy link

Hi,

I do daily backup to aws with barman-cloud and I wanted to restore the data as a test but barman-cloud-list failed with error below.
I could understand that storage class of previous backup make it failed because there is glacier backup in the bucket. To test, I have removed all glacier storage and it works.
Could you add an option like --ignore-storage-class or something like this ?

2021-03-29 13:55:36,158 [9244] ERROR: Barman cloud backup list exception: An error occurred (InvalidObjectState) when calling the GetObject operation: The operation is not valid for the object's storage class
2021-03-29 13:55:36,159 [9244] DEBUG: Exception details:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/barman/clients/cloud_backup_list.py", line 66, in main
    backup_list = catalog.get_backup_list()
  File "/usr/lib/python3/dist-packages/barman/cloud.py", line 1346, in get_backup_list
    backup_info = self.get_backup_info(backup_id)
  File "/usr/lib/python3/dist-packages/barman/cloud.py", line 1361, in get_backup_info
    backup_info_file = self.cloud_interface.remote_open(backup_info_path)
  File "/usr/lib/python3/dist-packages/barman/cloud.py", line 861, in remote_open
    return StreamingBodyIO(obj.get()['Body'])
  File "/usr/local/lib/python3.6/dist-packages/boto3/resources/factory.py", line 520, in do_action
    response = action(self, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/boto3/resources/action.py", line 83, in __call__
    response = getattr(parent.meta.client, operation_name)(*args, **params)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/usr/local/lib/python3.6/dist-packages/botocore/client.py", line 676, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.InvalidObjectState: An error occurred (InvalidObjectState) when calling the GetObject operation: The operation is not valid for the object's storage class

Thanks

@mikewallace1979 mikewallace1979 added this to the 2.13 milestone Jul 13, 2021
mikewallace1979 added a commit that referenced this issue Jul 23, 2021
Catches exceptions encountered when reading backup.info files
during backup list operations. A warning is logged and we continue
iterating through backup files.

This fixes a problem where a single unreadable backup.info file
would break barman-cloud-backup-list entirely, preventing users
from finding any backups in cloud storage. The specific example
reported was when a backup.info file has the AWS S3 Glacier
storage class however there are other reasons why a backup.info
file might not be readable. In these scenarios we would still
want to list the backup.info files that were readable, so we
therefore catch `Exception`s rather than anything more specific.

Closes #332
amenonsen pushed a commit that referenced this issue Jul 23, 2021
Catches exceptions encountered when reading backup.info files
during backup list operations. A warning is logged and we continue
iterating through backup files.

This fixes a problem where a single unreadable backup.info file
would break barman-cloud-backup-list entirely, preventing users
from finding any backups in cloud storage. The specific example
reported was when a backup.info file has the AWS S3 Glacier
storage class however there are other reasons why a backup.info
file might not be readable. In these scenarios we would still
want to list the backup.info files that were readable, so we
therefore catch `Exception`s rather than anything more specific.

Closes #332
@mikewallace1979
Copy link
Contributor

Rather than handling storage classes explicitly I've proposed a fix which would catch any exception on reading the backup.info file and continue walking the bucket looking for backup.info files. This is on the basis that there are other possible reasons why a backup.info file might not be readable and in all cases we'd still want to list the backups that we could read successfully.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants