Avoid pickling boto3 client in S3CloudInterface #361
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Excludes the boto3 client from the S3CloudInterface state so that
it is not pickled by multiprocessing.
This fixes barman-cloud-backup with Python >= 3.8. Previously this
would fail with the following error:
This is because boto3 cannot be pickled using the default pickle
protocol in Python >= 3.8. See the following boto3 issue:
The workaround of forcing pickle to use an older version of the
pickle protocol is not available because it is multiprocessing
which invokes pickle and it does not allow us to specify the
protocol version.
We therefore exclude the boto3 client from the pickle operation by
implementing custom
__getstate__
and__setstate__
methods asdocumented here:
This works because the worker processes create their own boto3
session anyway due to race conditions around re-using the boto3
session from the parent process.
It is also necessary to defer the assignment of the
worker_processes
list until after all worker processes have beenspawned as the references to those worker processes also cannot
be pickled with the default pickle protocol in Python >= 3.8. As
with the boto3 client, the
worker_processes
list was not beingused by the worker processes anyway.