Refactor cloudpickle support in Python operators/decorators #39270
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Some refactor cloudpickle support which added into the #38531
Use new
serializer
parameter for specify serialiser, this one for avoid boolean trap. Deprecateuse_dill
parameter in favor ofserializer="dill"
and removeuse_cloudpickle
in favor ofserializer="cloudpickle"
, this one safe to remove because change not released yet.Fix situation if
cloudpickle
installed thendill
serialisation not available and implicitly usecloudpickle
instead.airflow/airflow/operators/python.py
Lines 61 to 66 in adeb7f7
In addition lazy load serialisers as late as possible, for avoid the situation that we load all serialiser even if users do not use any of them
cc @VladaZakharova
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rst
or{issue_number}.significant.rst
, in newsfragments.