Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inherit the run_ordering from DatasetTriggeredTimetable for DatasetOrTimeSchedule #37775

Merged

Conversation

sunank200
Copy link
Collaborator

@sunank200 sunank200 commented Feb 28, 2024

  • Inherit the run_ordering from DatasetTriggeredTimetable for DatasetOrTimeSchedule

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@sunank200 sunank200 force-pushed the add_reordering_crontrigger_timetable branch from 7de5aa9 to de3e6e5 Compare February 28, 2024 16:36
@sunank200 sunank200 marked this pull request as ready for review February 28, 2024 16:36
@sunank200 sunank200 self-assigned this Feb 28, 2024
@bbovenzi
Copy link
Contributor

Why do we want to change it on CronTrigger instead of on DatasetOrTimeSchedule?

@sunank200
Copy link
Collaborator Author

Why do we want to change it on CronTrigger instead of on DatasetOrTimeSchedule?

DatasetOrTimeSchedule for example uses the reordering timetable object as per https://github.com/apache/airflow/blob/main/airflow/timetables/datasets.py#L55 For example:

schedule=DatasetOrTimeSchedule(
        timetable=CronTriggerTimetable("* * * * *", timezone="UTC"), datasets=(dag1_dataset & dag2_dataset)
    ),

@bbovenzi
Copy link
Contributor

@uranusjr Does this make sense to you?

@sunank200
Copy link
Collaborator Author

@uranusjr Does this make sense to you?

@bbovenzi I have added the reordering for DatasetOrTimeSchedule now.

@sunank200 sunank200 force-pushed the add_reordering_crontrigger_timetable branch from de3e6e5 to 4c43885 Compare February 28, 2024 18:21
@sunank200 sunank200 changed the title Add default reordering as execution_date for CronTriggerTimetable Add default reordering as start_date for DatasetOrTimeSchedule Feb 28, 2024
@sunank200 sunank200 force-pushed the add_reordering_crontrigger_timetable branch from 4c43885 to 16f7f02 Compare February 29, 2024 12:19
@sunank200 sunank200 changed the title Add default reordering as start_date for DatasetOrTimeSchedule Inherit the run_ordering from DatasetTriggeredSchedule for DatasetOrTimeSchedule Feb 29, 2024
@sunank200 sunank200 changed the title Inherit the run_ordering from DatasetTriggeredSchedule for DatasetOrTimeSchedule Inherit the run_ordering from DatasetTriggeredTimetable for DatasetOrTimeSchedule Feb 29, 2024
@sunank200 sunank200 force-pushed the add_reordering_crontrigger_timetable branch from 16f7f02 to 54c1791 Compare February 29, 2024 12:21
@vatsrahul1001
Copy link
Collaborator

@sunank200 tested, LGTM

@phanikumv phanikumv merged commit b4c1b3f into apache:main Feb 29, 2024
59 checks passed
@phanikumv phanikumv deleted the add_reordering_crontrigger_timetable branch February 29, 2024 15:29
@dstandish
Copy link
Contributor

nice job finding the problem @sunank200

@utkarsharma2 utkarsharma2 added the type:new-feature Changelog: New Features label Mar 6, 2024
@utkarsharma2 utkarsharma2 added this to the Airflow 2.9.0 milestone Mar 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type:new-feature Changelog: New Features
Projects
None yet
Development

Successfully merging this pull request may close these issues.

8 participants