-
Notifications
You must be signed in to change notification settings - Fork 16.3k
Docs setup for the Task SDK #51153
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docs setup for the Task SDK #51153
Conversation
667ba76 to
209777c
Compare
caac5ca to
aab509c
Compare
89fc083 to
e8f309f
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It handles most of my comments, and the PR need not be perfect in the initial cut. I am OK with this, as it implements the initial skeleton for task SDK docs, so in the interest of setting up the basic layer to build things onto, looks good +1
Please ensure to handle the comments before merge.
Yes I have created follow up issues as well as mentioned in the thread of comments above. |
1078c49 to
00125a6
Compare
In order to have a nicer experience (i.e. rather than just being given an alphabetical list of classes/functions) I have chosen to not use the auto generate feature of the AutoAPI extension, but instead to precisely control the order and grouping of the classes. For this to be complete we will likely need some testing that compares the items on the generated `objects.inv` with the things we re-export from `airflow/sdk/__init__.py`. The `airflow/sdk/__init__.pyi` type stub is sadly needed, as without it (and with the imports only inside the `if TYPE_CHECKING` block) we weren't getting the re-exported classes showing up where we want them, specifically we want users to import things directly from airflow.sdk, not the submodule. It was generated with `stubgen -o task_sdk/src -m airflow.sdk` Test this with `sphinx-build -b html -Tv task_sdk/docs/ task_sdk/docs/_build`
… (e.g. airflow.sdk.DAG) fix static checks Fix the static checks Fix the tests Fix autoapimodule explicitly import all your public API into airflow/sdk/__init__.py (including literal, MappedOperator, SecretCache, etc.), and we wired up __lazy_imports and __all__ so AutoAPI only sees those names. Add SecretCache in the SDK module for both runtime and static typing use the top-level SDK path (e.g. airflow.sdk.DAG) Add Breeze build-task-sdk-docs command; tests for Task-SDK API vs docs; clean up Sphinx config
Add imports in example DAGs, fix spell checks in task-sdk, trigger doc build only on doc file changes; compare by file path, line number, context, spelling, and message; ensure context_line is a string, add spellcheck, nuke mapped operator from api.rst, remove skip_util_classes and setup from conf, fix PR comments, revert changes for SecretCache and MappedOperator, remove SecretCache and MappedOperator, fix tests and remove dir() from __init__, add breeze build-docs task-sdk, revert build-task-sdk-docs docs and related images.
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
|
Feel free to address other comments in a follow-up PR |
This PR is a continuation of apache#47357 To enhance the experience (i.e., rather than just being given an alphabetical list of classes/functions), I have opted not to use the AutoAPI extension's auto-generate feature, allowing me to precisely control the order and grouping of the classes. I have organized Task-SDK API into these sections: * **Defining DAGs**: `DAG`, `dag` * **Decorators**: `task`, `task_group`, `setup`, `teardown` * **Bases**: `BaseOperator`, `BaseSensorOperator`, `BaseNotifier`, `BaseOperatorLink`, `XComArg` * **Connections & Variables**: `Connection`, `Variable` * **Tasks & Operators**: `TaskGroup`, `get_current_context`, `get_parsing_context`, `Param`, `chain`, `chain_linear`, `cross_downstream` * **Assets**: `Asset`, `AssetAlias`, `AssetAll`, `AssetAny`, `AssetWatcher`, `Metadata` I also ensured that all documented references use the top-level SDK path (e.g., `airflow.sdk.DAG`) instead of exposing the underlying module paths (e.g., `airflow.sdk.definitions.dag.DAG`). I have incorporated a test that compares the items in the generated `objects.inv` with the elements I re-export from `airflow/sdk/__init__.py`. The `airflow/sdk/__init__.pyi` type stub is unfortunately necessary, as without it (and with the imports solely within the `if TYPE_CHECKING` block), the re-exported classes were not appearing where I wanted them. Specifically, I want users to import items directly from `airflow.sdk`, not the submodule. stubgen -o task-sdk/src -m airflow.sdk Test this with breeze build-docs task-sdk or uv run --group docs build-docs task-sdk **Screenshots** <img width="1720" alt="Screenshot 2025-06-02 at 6 15 40 PM" src="https://github.com/user-attachments/assets/7ce06d9f-1af3-4156-acf7-642d5f37f907" /> <img width="1726" alt="Screenshot 2025-06-02 at 7 04 10 PM" src="https://github.com/user-attachments/assets/adfa99b3-1c40-4c37-b523-4d6ee27381b2" /> <img width="1707" alt="Screenshot 2025-06-02 at 6 44 01 PM" src="https://github.com/user-attachments/assets/d5ccbabc-dfbd-465d-ae32-3697acdce827" /> <img width="1728" alt="Screenshot 2025-06-11 at 1 06 26 AM" src="https://github.com/user-attachments/assets/c8a5792b-59ad-4855-8937-4877a31a9a55" /> closes: apache#43010 closes: apache#51283 (cherry picked from commit ebfc0de)
This PR is a continuation of #47357 To enhance the experience (i.e., rather than just being given an alphabetical list of classes/functions), I have opted not to use the AutoAPI extension's auto-generate feature, allowing me to precisely control the order and grouping of the classes. I have organized Task-SDK API into these sections: * **Defining DAGs**: `DAG`, `dag` * **Decorators**: `task`, `task_group`, `setup`, `teardown` * **Bases**: `BaseOperator`, `BaseSensorOperator`, `BaseNotifier`, `BaseOperatorLink`, `XComArg` * **Connections & Variables**: `Connection`, `Variable` * **Tasks & Operators**: `TaskGroup`, `get_current_context`, `get_parsing_context`, `Param`, `chain`, `chain_linear`, `cross_downstream` * **Assets**: `Asset`, `AssetAlias`, `AssetAll`, `AssetAny`, `AssetWatcher`, `Metadata` I also ensured that all documented references use the top-level SDK path (e.g., `airflow.sdk.DAG`) instead of exposing the underlying module paths (e.g., `airflow.sdk.definitions.dag.DAG`). I have incorporated a test that compares the items in the generated `objects.inv` with the elements I re-export from `airflow/sdk/__init__.py`. The `airflow/sdk/__init__.pyi` type stub is unfortunately necessary, as without it (and with the imports solely within the `if TYPE_CHECKING` block), the re-exported classes were not appearing where I wanted them. Specifically, I want users to import items directly from `airflow.sdk`, not the submodule. stubgen -o task-sdk/src -m airflow.sdk Test this with breeze build-docs task-sdk or uv run --group docs build-docs task-sdk **Screenshots** <img width="1720" alt="Screenshot 2025-06-02 at 6 15 40 PM" src="https://github.com/user-attachments/assets/7ce06d9f-1af3-4156-acf7-642d5f37f907" /> <img width="1726" alt="Screenshot 2025-06-02 at 7 04 10 PM" src="https://github.com/user-attachments/assets/adfa99b3-1c40-4c37-b523-4d6ee27381b2" /> <img width="1707" alt="Screenshot 2025-06-02 at 6 44 01 PM" src="https://github.com/user-attachments/assets/d5ccbabc-dfbd-465d-ae32-3697acdce827" /> <img width="1728" alt="Screenshot 2025-06-11 at 1 06 26 AM" src="https://github.com/user-attachments/assets/c8a5792b-59ad-4855-8937-4877a31a9a55" /> closes: #43010 closes: #51283 (cherry picked from commit ebfc0de)
This PR is a continuation of #47357
To enhance the experience (i.e., rather than just being given an alphabetical list of classes/functions), I have opted not to use the AutoAPI extension's auto-generate feature, allowing me to precisely control the order and grouping of the classes.
I have organized Task-SDK API into these sections:
DAG,dagtask,task_group,setup,teardownBaseOperator,BaseSensorOperator,BaseNotifier,BaseOperatorLink,XComArgConnection,VariableTaskGroup,get_current_context,get_parsing_context,Param,chain,chain_linear,cross_downstreamAsset,AssetAlias,AssetAll,AssetAny,AssetWatcher,MetadataI also ensured that all documented references use the top-level SDK path (e.g.,
airflow.sdk.DAG) instead of exposing the underlying module paths (e.g.,airflow.sdk.definitions.dag.DAG).I have incorporated a test that compares the items in the generated
objects.invwith the elements I re-export fromairflow/sdk/__init__.py.The
airflow/sdk/__init__.pyitype stub is unfortunately necessary, as without it (and with the imports solely within theif TYPE_CHECKINGblock), the re-exported classes were not appearing where I wanted them. Specifically, I want users to import items directly fromairflow.sdk, not the submodule.Test this with
or
Screenshots




closes: #43010
closes: #51283
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.