-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Add @task.kuberenetes_cmd
#46913
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add @task.kuberenetes_cmd
#46913
Conversation
4afbe38 to
f4567b7
Compare
|
Should I separate the test refactoring from this PR? |
6836426 to
da0a9c2
Compare
|
The taskflow tutorial docs change |
d0741ce to
64269e3
Compare
|
I think that this should be adjusted to use TaskSDK and not depend on direct DB access, so made it a draft for now, before I would understand the proper way of doing it. |
6ca0c03 to
99943c9
Compare
|
Updated based on the recent example of I am still unsure: how to trigger system tests for example DAG in CI. |
b305d6b to
683961a
Compare
683961a to
a766622
Compare
|
Rebased it again - just to check - but I think this one will not get too much attention at least before RC1 - but please keep on rebasing it. Being on top of the main and fresh in the inbox helps. |
Yeah, I understand, there are a lot of things to do for the maintainers and committers now, and this PR is vast. Thank you! |
eladkal
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Temporary block till we sort out the decorators in providers
a766622 to
07ae5c1
Compare
This should fix docstring and type-checking for decorator
f4000cf to
0bae401
Compare
Adding a new task flow decorator to run generated cmds in
KubernetesPodOperator.This decorator is based on the same idea as
@task.bash: enrich Kubernetes pod command generation ability with simplified Python usage on the executor side to generate the command.With the
args_onlyoption, the decorator would set the return value as arguments of the image entrypoint instead of rewriting the cmds.This can significantly simplify usage for pure orchestrator cases and dynamic task mapping or generating the command from another task flow task result.
Changes:
@task.kubernetes_cmdand register it inprovider.yamlforcncf.kubernetes;@task.kubernetesunit tests to use the same base and have common tests with@task.kubernetes_cmd;@overloadfor@task.kubernetesdescription in decorators pyi file to enable proper doc-string and signature checking in IDEscloses: #46414
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in newsfragments.