-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Migrate gcs to new system tests design #22778
Conversation
The PR is bigger than I want but it's mostly because there are several setup fixtures in old design (in pytest classes) that can now be covered by operators - thus the structure changed and multiple examples had to be done in one go. Everything relates to gcs though. |
@bhirsz |
I've commented in your PR. But I've also encountered this problem whenever I run the same DAG_ID twice in the same environment (resetting env or using unique name between runs helps). I will investigate why it happens. |
Change-Id: I2aeb518d236296bcdbd489850ed1813cefc0ce3f
Change-Id: I2558eab8a98706e693c0cc9bed8bc33354e6a23f
Change-Id: I4d7d88e13de00b25cbce66b0319870cae1be249b
Change-Id: Iec9be2369a93f2b9bab62f6e5eef298492852c90
7665283
to
5bb09f6
Compare
Resolved by changing State.NONE (which works only for Tasks) to State.Queued (default state for dag) |
Change-Id: I8af21344a836c232ff505c1442b960253eec31b7
…into gcs_system_tests Change-Id: Ie7a572289872d10c649d8697a5819ddcac9d9e99
Change-Id: I2aeb518d236296bcdbd489850ed1813cefc0ce3f
Change-Id: I2558eab8a98706e693c0cc9bed8bc33354e6a23f
Change-Id: Iec9be2369a93f2b9bab62f6e5eef298492852c90
Change-Id: I8af21344a836c232ff505c1442b960253eec31b7
6492a1e
to
6e8baa6
Compare
Rebased it - to test it with the new CI based on Breeze mostly. |
The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease. |
Stuff to fix :) |
…into gcs_system_tests Change-Id: I3b7f9c184a54a52a39ead74345990eccdc3571b1
Change-Id: Ia75d3d3df42cd708425eb8bbbf3e3978907b6a71
…into gcs_system_tests Change-Id: I63b511054ea694c8f8548fc68e21fd77ee2f88ad
Change-Id: I349a3f0555c04c22d5b095a790ec3be43c343ca9
@potiuk All green :) |
Migrate GCS system tests/example dags to be compliant with new design (AIP-47).
Covers
airflow/providers/google/cloud/example_dags/example_gcs.py
airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
airflow/providers/google/cloud/example_dags/example_local_to_gcs.py
from #22447