Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Dataset, Model asset subclasses #43142

Merged
merged 8 commits into from
Oct 23, 2024

Conversation

Lee-W
Copy link
Member

@Lee-W Lee-W commented Oct 18, 2024

Close: #42312


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@Lee-W Lee-W changed the title feat(asset): add dataset subclass Add Dataset, Model subclasses Oct 18, 2024
@Lee-W Lee-W force-pushed the add-dataset-and-model-subclass branch 2 times, most recently from 01b332d to 044ee6b Compare October 18, 2024 08:02
@Lee-W Lee-W marked this pull request as ready for review October 18, 2024 08:44
@Lee-W Lee-W changed the title Add Dataset, Model subclasses Add Dataset, Model asset subclasses Oct 18, 2024
@Lee-W Lee-W force-pushed the add-dataset-and-model-subclass branch from 218c7fe to b300945 Compare October 18, 2024 09:20
@uranusjr
Copy link
Member

Talked to Constance earlier; we should probably still make from airflow.datasets import Dataset work (and maybe also from airflow import Dataset). Without jumping through too many hoops, let’s just create a mostly empty airflow/datasets.py that emits a deprecation warning on import, and add a lazy import entry in airflow/__init__.py. Similar to how we handle name deprecations in providers.

Same for DatasetAlias; this one is not as important (I doubt many people are already using it aside from Cosmos), but we might as well add it since the mechanism is the same.

Those names should be removed in Airflow 3.2.

@Lee-W
Copy link
Member Author

Lee-W commented Oct 22, 2024

Talked to Constance earlier; we should probably still make from airflow.datasets import Dataset work (and maybe also from airflow import Dataset). Without jumping through too many hoops, let’s just create a mostly empty airflow/datasets.py that emits a deprecation warning on import, and add a lazy import entry in airflow/__init__.py. Similar to how we handle name deprecations in providers.

Same for DatasetAlias; this one is not as important (I doubt many people are already using it aside from Cosmos), but we might as well add it since the mechanism is the same.

Those names should be removed in Airflow 3.2.

Sounds good. Will do

@Lee-W Lee-W force-pushed the add-dataset-and-model-subclass branch 2 times, most recently from beb4d3a to ceefc1c Compare October 22, 2024 11:37
@Lee-W
Copy link
Member Author

Lee-W commented Oct 22, 2024

Talked to Constance earlier; we should probably still make from airflow.datasets import Dataset work (and maybe also from airflow import Dataset). Without jumping through too many hoops, let’s just create a mostly empty airflow/datasets.py that emits a deprecation warning on import, and add a lazy import entry in airflow/__init__.py. Similar to how we handle name deprecations in providers.

Same for DatasetAlias; this one is not as important (I doubt many people are already using it aside from Cosmos), but we might as well add it since the mechanism is the same.

Those names should be removed in Airflow 3.2.

Updated

@Lee-W Lee-W force-pushed the add-dataset-and-model-subclass branch from ceefc1c to 1e2f92e Compare October 22, 2024 14:44
@Lee-W Lee-W merged commit 6163b39 into apache:main Oct 23, 2024
52 checks passed
@Lee-W Lee-W deleted the add-dataset-and-model-subclass branch October 23, 2024 06:10
harjeevanmaan pushed a commit to harjeevanmaan/airflow that referenced this pull request Oct 23, 2024
* feat(assets): add dataset subclass
* feat(assets): add model subclass
* feat(assets): make group a default instead of overwriting user input
* feat(assets): allow "airflow.Dataset" and "airflow.datasets.Dataset", "airflow.datasets.DatasetAlias" import for backward compat
PaulKobow7536 pushed a commit to PaulKobow7536/airflow that referenced this pull request Oct 24, 2024
* feat(assets): add dataset subclass
* feat(assets): add model subclass
* feat(assets): make group a default instead of overwriting user input
* feat(assets): allow "airflow.Dataset" and "airflow.datasets.Dataset", "airflow.datasets.DatasetAlias" import for backward compat
ellisms pushed a commit to ellisms/airflow that referenced this pull request Nov 13, 2024
* feat(assets): add dataset subclass
* feat(assets): add model subclass
* feat(assets): make group a default instead of overwriting user input
* feat(assets): allow "airflow.Dataset" and "airflow.datasets.Dataset", "airflow.datasets.DatasetAlias" import for backward compat
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Dataset and Model subclasses
2 participants