Skip to content

Conversation

@sunank200
Copy link
Collaborator

@sunank200 sunank200 commented May 28, 2025

This PR is a continuation of #47357

To enhance the experience (i.e., rather than just being given an alphabetical list of classes/functions), I have opted not to use the AutoAPI extension's auto-generate feature, allowing me to precisely control the order and grouping of the classes.

I have organized Task-SDK API into these sections:

  • Defining DAGs: DAG, dag
  • Decorators: task, task_group, setup, teardown
  • Bases: BaseOperator, BaseSensorOperator, BaseNotifier, BaseOperatorLink, XComArg
  • Connections & Variables: Connection, Variable
  • Tasks & Operators: TaskGroup, get_current_context, get_parsing_context, Param, chain, chain_linear, cross_downstream
  • Assets: Asset, AssetAlias, AssetAll, AssetAny, AssetWatcher, Metadata

I also ensured that all documented references use the top-level SDK path (e.g., airflow.sdk.DAG) instead of exposing the underlying module paths (e.g., airflow.sdk.definitions.dag.DAG).

I have incorporated a test that compares the items in the generated objects.inv with the elements I re-export from airflow/sdk/__init__.py.

The airflow/sdk/__init__.pyi type stub is unfortunately necessary, as without it (and with the imports solely within the if TYPE_CHECKING block), the re-exported classes were not appearing where I wanted them. Specifically, I want users to import items directly from airflow.sdk, not the submodule.

    stubgen -o task-sdk/src -m airflow.sdk

Test this with

 breeze build-docs task-sdk

or

uv run --group docs build-docs task-sdk

Screenshots
Screenshot 2025-06-02 at 6 15 40 PM
Screenshot 2025-06-02 at 7 04 10 PM
Screenshot 2025-06-02 at 6 44 01 PM
Screenshot 2025-06-11 at 1 06 26 AM

closes: #43010
closes: #51283


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@sunank200 sunank200 force-pushed the task-sdk-docs branch 12 times, most recently from caac5ca to aab509c Compare June 2, 2025 13:18
@sunank200 sunank200 changed the title Docs for the Task SDK Docs setup for the Task SDK Jun 2, 2025
@sunank200 sunank200 requested a review from kaxil June 11, 2025 05:28
@sunank200 sunank200 force-pushed the task-sdk-docs branch 3 times, most recently from 89fc083 to e8f309f Compare June 11, 2025 09:11
Copy link
Contributor

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It handles most of my comments, and the PR need not be perfect in the initial cut. I am OK with this, as it implements the initial skeleton for task SDK docs, so in the interest of setting up the basic layer to build things onto, looks good +1

Please ensure to handle the comments before merge.

@sunank200
Copy link
Collaborator Author

It handles most of my comments, and the PR need not be perfect in the initial cut. I am OK with this, as it implements the initial skeleton for task SDK docs, so in the interest of setting up the basic layer to build things onto, looks good +1

Please ensure to handle the comments before merge.

Yes I have created follow up issues as well as mentioned in the thread of comments above.

@sunank200 sunank200 force-pushed the task-sdk-docs branch 2 times, most recently from 1078c49 to 00125a6 Compare June 11, 2025 12:12
ashb and others added 6 commits June 11, 2025 19:08
In order to have a nicer experience (i.e. rather than just being given an
alphabetical list of classes/functions) I have chosen to not use the auto
generate feature of the AutoAPI extension, but instead to precisely control
the order and grouping of the classes.

For this to be complete we will likely need some testing that compares the
items on the generated `objects.inv` with the things we re-export from
`airflow/sdk/__init__.py`.

The `airflow/sdk/__init__.pyi` type stub is sadly needed, as without it (and
with the imports only inside the `if TYPE_CHECKING` block) we weren't getting
the re-exported classes showing up where we want them, specifically we want
users to import things directly from airflow.sdk, not the submodule.

It was generated with `stubgen -o task_sdk/src -m airflow.sdk`

Test this with `sphinx-build -b html -Tv task_sdk/docs/ task_sdk/docs/_build`
… (e.g. airflow.sdk.DAG)

fix static checks

Fix the static checks

Fix the tests

Fix autoapimodule

explicitly import all your public API into airflow/sdk/__init__.py (including literal, MappedOperator, SecretCache, etc.), and we wired up __lazy_imports and __all__ so AutoAPI only sees those names.

Add SecretCache in the SDK module for both runtime and static typing

use the top-level SDK path (e.g. airflow.sdk.DAG)

Add Breeze build-task-sdk-docs command; tests for Task-SDK API vs docs; clean up Sphinx config
Add imports in example DAGs, fix spell checks in task-sdk, trigger doc build only on doc file changes; compare by file path, line number, context, spelling, and message; ensure context_line is a string, add spellcheck, nuke mapped operator from api.rst, remove skip_util_classes and setup from conf, fix PR comments, revert changes for SecretCache and MappedOperator, remove SecretCache and MappedOperator, fix tests and remove dir() from __init__, add breeze build-docs task-sdk, revert build-task-sdk-docs docs and related images.
Co-authored-by: Ash Berlin-Taylor <ash_github@firemirror.com>
@sunank200 sunank200 requested a review from ashb June 11, 2025 13:27
@kaxil kaxil merged commit ebfc0de into apache:main Jun 11, 2025
94 of 95 checks passed
@kaxil kaxil deleted the task-sdk-docs branch June 11, 2025 14:24
@kaxil
Copy link
Member

kaxil commented Jun 11, 2025

Feel free to address other comments in a follow-up PR

@sunank200 sunank200 added the backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch label Jun 24, 2025
sunank200 added a commit to astronomer/airflow that referenced this pull request Jul 2, 2025
This PR is a continuation of apache#47357

To enhance the experience (i.e., rather than just being given an alphabetical list of classes/functions), I have opted not to use the AutoAPI extension's auto-generate feature, allowing me to precisely control the order and grouping of the classes.

I have organized Task-SDK API into these sections:
* **Defining DAGs**: `DAG`, `dag`
* **Decorators**: `task`, `task_group`, `setup`, `teardown`
* **Bases**: `BaseOperator`, `BaseSensorOperator`, `BaseNotifier`, `BaseOperatorLink`, `XComArg`
* **Connections & Variables**: `Connection`, `Variable`
* **Tasks & Operators**: `TaskGroup`, `get_current_context`, `get_parsing_context`, `Param`, `chain`, `chain_linear`, `cross_downstream`
* **Assets**: `Asset`, `AssetAlias`, `AssetAll`, `AssetAny`, `AssetWatcher`, `Metadata`

I also ensured that all documented references use the top-level SDK path (e.g., `airflow.sdk.DAG`) instead of exposing the underlying module paths (e.g., `airflow.sdk.definitions.dag.DAG`).

I have incorporated a test that compares the items in the generated `objects.inv` with the elements I re-export from `airflow/sdk/__init__.py`.

The `airflow/sdk/__init__.pyi` type stub is unfortunately necessary, as without it (and with the imports solely within the `if TYPE_CHECKING` block), the re-exported classes were not appearing where I wanted them. Specifically, I want users to import items directly from `airflow.sdk`, not the submodule.

        stubgen -o task-sdk/src -m airflow.sdk

Test this with

     breeze build-docs task-sdk

or

    uv run --group docs build-docs task-sdk

**Screenshots**
<img width="1720" alt="Screenshot 2025-06-02 at 6 15 40 PM" src="https://github.com/user-attachments/assets/7ce06d9f-1af3-4156-acf7-642d5f37f907" />
<img width="1726" alt="Screenshot 2025-06-02 at 7 04 10 PM" src="https://github.com/user-attachments/assets/adfa99b3-1c40-4c37-b523-4d6ee27381b2" />
<img width="1707" alt="Screenshot 2025-06-02 at 6 44 01 PM" src="https://github.com/user-attachments/assets/d5ccbabc-dfbd-465d-ae32-3697acdce827" />
<img width="1728" alt="Screenshot 2025-06-11 at 1 06 26 AM" src="https://github.com/user-attachments/assets/c8a5792b-59ad-4855-8937-4877a31a9a55" />

closes: apache#43010
closes: apache#51283
(cherry picked from commit ebfc0de)
kaxil pushed a commit that referenced this pull request Jul 2, 2025
This PR is a continuation of #47357

To enhance the experience (i.e., rather than just being given an alphabetical list of classes/functions), I have opted not to use the AutoAPI extension's auto-generate feature, allowing me to precisely control the order and grouping of the classes.

I have organized Task-SDK API into these sections:
* **Defining DAGs**: `DAG`, `dag`
* **Decorators**: `task`, `task_group`, `setup`, `teardown`
* **Bases**: `BaseOperator`, `BaseSensorOperator`, `BaseNotifier`, `BaseOperatorLink`, `XComArg`
* **Connections & Variables**: `Connection`, `Variable`
* **Tasks & Operators**: `TaskGroup`, `get_current_context`, `get_parsing_context`, `Param`, `chain`, `chain_linear`, `cross_downstream`
* **Assets**: `Asset`, `AssetAlias`, `AssetAll`, `AssetAny`, `AssetWatcher`, `Metadata`

I also ensured that all documented references use the top-level SDK path (e.g., `airflow.sdk.DAG`) instead of exposing the underlying module paths (e.g., `airflow.sdk.definitions.dag.DAG`).

I have incorporated a test that compares the items in the generated `objects.inv` with the elements I re-export from `airflow/sdk/__init__.py`.

The `airflow/sdk/__init__.pyi` type stub is unfortunately necessary, as without it (and with the imports solely within the `if TYPE_CHECKING` block), the re-exported classes were not appearing where I wanted them. Specifically, I want users to import items directly from `airflow.sdk`, not the submodule.

        stubgen -o task-sdk/src -m airflow.sdk

Test this with

     breeze build-docs task-sdk

or

    uv run --group docs build-docs task-sdk

**Screenshots**
<img width="1720" alt="Screenshot 2025-06-02 at 6 15 40 PM" src="https://github.com/user-attachments/assets/7ce06d9f-1af3-4156-acf7-642d5f37f907" />
<img width="1726" alt="Screenshot 2025-06-02 at 7 04 10 PM" src="https://github.com/user-attachments/assets/adfa99b3-1c40-4c37-b523-4d6ee27381b2" />
<img width="1707" alt="Screenshot 2025-06-02 at 6 44 01 PM" src="https://github.com/user-attachments/assets/d5ccbabc-dfbd-465d-ae32-3697acdce827" />
<img width="1728" alt="Screenshot 2025-06-11 at 1 06 26 AM" src="https://github.com/user-attachments/assets/c8a5792b-59ad-4855-8937-4877a31a9a55" />

closes: #43010
closes: #51283
(cherry picked from commit ebfc0de)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:dev-tools area:task-sdk backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch kind:documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Create Task SDK docs Docs: Replace airflow.sdk.definitions to airflow.sdk for public facing docs

5 participants