-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Make the policy functions pluggable #28558
Conversation
Nice. Super useful for multiple policies and to be able to write multiple smaller policies. I wondered initially if the args passed by pluggy are mutable but the example has it explained https://pluggy.readthedocs.io/en/stable/#a-complete-example Indeed steak sauce does not goo well with eggs, but I wish they also deleted mint sauce and picked walnuts (?). I love mushy peas though. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good, haven't tested it out -- but code looks sensible :)
382bb34
to
9728b82
Compare
docs/apache-airflow/administration-and-deployment/cluster-policies.rst
Outdated
Show resolved
Hide resolved
b4d9089
to
3f6086b
Compare
Previously it was only possible to set "policy" functions via airflow_local_settings.py which is fine for "small clusters" but being able to control some of these policies from installed plugins/distributions is helpful in a few circumstances: it lets "platforms" (either of the SaaS variety, or internal platform teams) specify some common policies, but still let local Ariflow teams define other policies using airflow_local_settings This uses Pluggy[[1]] for plugin management (that we are already using for the listener interface), and out of the box that gives us the ability to have multiple implementations of the policy functions where it makes sense - on all the "mutation" hooks all implementations will be called. (i.e. for the policies that don't return a value) - for any policy function that returns a value (such as `get_dagbag_import_timeout`) then the plugins will be tried in LIFO order until a non-None value is returned. (No code needed for us -- Pluggy handles this out of the box with the `firstresult=True` property on the hookspec decorator). [1]: https://pluggy.readthedocs.io/en/latest/index.html
Co-authored-by: Kaxil Naik <kaxilnaik@gmail.com>
And move them back in to cluster-policies.rst -- it feels better where it was
c25696a
to
28987e8
Compare
Co-authored-by: Jed Cunningham <66968678+jedcunningham@users.noreply.github.com>
Previously it was only possible to set "policy" functions via
airflow_local_settings.py which is fine for "small clusters" but being
able to control some of these policies from installed
plugins/distributions is helpful in a few circumstances: it lets
"platforms" (either of the SaaS variety, or internal platform teams)
specify some common policies, but still let local Ariflow teams define
other policies using airflow_local_settings
This uses Pluggy[1] for plugin management (that we are already using
for the listener interface), and out of the box that
gives us the ability to have multiple implementations of the policy
functions where it makes sense
for the policies that don't return a value)
get_dagbag_import_timeout
) then the plugins will be tried in LIFOorder until a non-None value is returned. (No code needed for us --
Pluggy handles this out of the box with the
firstresult=True
property on the hookspec decorator).
Todo: