Backport Docs setup for the Task SDK (#51153) #52682
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This PR is a continuation of apache#47357 To enhance the experience (i.e., rather than just being given an alphabetical list of classes/functions), I have opted not to use the AutoAPI extension's auto-generate feature, allowing me to precisely control the order and grouping of the classes. I have organized Task-SDK API into these sections: * **Defining DAGs**: `DAG`, `dag` * **Decorators**: `task`, `task_group`, `setup`, `teardown` * **Bases**: `BaseOperator`, `BaseSensorOperator`, `BaseNotifier`, `BaseOperatorLink`, `XComArg` * **Connections & Variables**: `Connection`, `Variable` * **Tasks & Operators**: `TaskGroup`, `get_current_context`, `get_parsing_context`, `Param`, `chain`, `chain_linear`, `cross_downstream` * **Assets**: `Asset`, `AssetAlias`, `AssetAll`, `AssetAny`, `AssetWatcher`, `Metadata` I also ensured that all documented references use the top-level SDK path (e.g., `airflow.sdk.DAG`) instead of exposing the underlying module paths (e.g., `airflow.sdk.definitions.dag.DAG`). I have incorporated a test that compares the items in the generated `objects.inv` with the elements I re-export from `airflow/sdk/__init__.py`. The `airflow/sdk/__init__.pyi` type stub is unfortunately necessary, as without it (and with the imports solely within the `if TYPE_CHECKING` block), the re-exported classes were not appearing where I wanted them. Specifically, I want users to import items directly from `airflow.sdk`, not the submodule. stubgen -o task-sdk/src -m airflow.sdk Test this with breeze build-docs task-sdk or uv run --group docs build-docs task-sdk **Screenshots** <img width="1720" alt="Screenshot 2025-06-02 at 6 15 40 PM" src="https://github.com/user-attachments/assets/7ce06d9f-1af3-4156-acf7-642d5f37f907" /> <img width="1726" alt="Screenshot 2025-06-02 at 7 04 10 PM" src="https://github.com/user-attachments/assets/adfa99b3-1c40-4c37-b523-4d6ee27381b2" /> <img width="1707" alt="Screenshot 2025-06-02 at 6 44 01 PM" src="https://github.com/user-attachments/assets/d5ccbabc-dfbd-465d-ae32-3697acdce827" /> <img width="1728" alt="Screenshot 2025-06-11 at 1 06 26 AM" src="https://github.com/user-attachments/assets/c8a5792b-59ad-4855-8937-4877a31a9a55" /> closes: apache#43010 closes: apache#51283 (cherry picked from commit ebfc0de)
sunank200
requested review from
amoghrajesh,
ashb,
dstandish,
ephraimbuddy,
jedcunningham,
kaxil and
pierrejeambrun
as code owners
boring-cyborg
bot
added
area:dev-tools
area:task-sdk
backport-to-v3-1-test
kaxil
approved these changes
Jul 2, 2025
This was referenced Jul 2, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is a continuation of #47357
To enhance the experience (i.e., rather than just being given an alphabetical list of classes/functions), I have opted not to use the AutoAPI extension's auto-generate feature, allowing me to precisely control the order and grouping of the classes.
I have organized Task-SDK API into these sections:
DAG,dagtask,task_group,setup,teardownBaseOperator,BaseSensorOperator,BaseNotifier,BaseOperatorLink,XComArgConnection,VariableTaskGroup,get_current_context,get_parsing_context,Param,chain,chain_linear,cross_downstreamAsset,AssetAlias,AssetAll,AssetAny,AssetWatcher,MetadataI also ensured that all documented references use the top-level SDK path (e.g.,
airflow.sdk.DAG) instead of exposing the underlying module paths (e.g.,airflow.sdk.definitions.dag.DAG).I have incorporated a test that compares the items in the generated
objects.invwith the elements I re-export fromairflow/sdk/__init__.py.The
airflow/sdk/__init__.pyitype stub is unfortunately necessary, as without it (and with the imports solely within theif TYPE_CHECKINGblock), the re-exported classes were not appearing where I wanted them. Specifically, I want users to import items directly fromairflow.sdk, not the submodule.Test this with
or
Screenshots
closes: #43010
closes: #51283
(cherry picked from commit ebfc0de)
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.