-
Notifications
You must be signed in to change notification settings - Fork 14.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[AIRFLOW-5088][AIP-24] Persisting serialized DAG in DB for webserver scalability #5743
Commits on Oct 22, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 67515c5 - Browse repository at this point
Copy the full SHA 67515c5View commit details -
Configuration menu - View commit details
-
Copy full SHA for 07a7481 - Browse repository at this point
Copy the full SHA 07a7481View commit details -
Configuration menu - View commit details
-
Copy full SHA for bf11bba - Browse repository at this point
Copy the full SHA bf11bbaView commit details -
make dagbag.get_dag/dag.get_bag called by webserver/scheduler control…
…led by dagcached_enabled and only_from_file
Configuration menu - View commit details
-
Copy full SHA for e58d100 - Browse repository at this point
Copy the full SHA e58d100View commit details -
Configuration menu - View commit details
-
Copy full SHA for 1db7933 - Browse repository at this point
Copy the full SHA 1db7933View commit details -
Configuration menu - View commit details
-
Copy full SHA for f104fc3 - Browse repository at this point
Copy the full SHA f104fc3View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4559939 - Browse repository at this point
Copy the full SHA 4559939View commit details -
Configuration menu - View commit details
-
Copy full SHA for bb65a22 - Browse repository at this point
Copy the full SHA bb65a22View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2efd089 - Browse repository at this point
Copy the full SHA 2efd089View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0eb6898 - Browse repository at this point
Copy the full SHA 0eb6898View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2124fe8 - Browse repository at this point
Copy the full SHA 2124fe8View commit details -
Configuration menu - View commit details
-
Copy full SHA for ab443c4 - Browse repository at this point
Copy the full SHA ab443c4View commit details -
Configuration menu - View commit details
-
Copy full SHA for b8cbd55 - Browse repository at this point
Copy the full SHA b8cbd55View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4d46b8d - Browse repository at this point
Copy the full SHA 4d46b8dView commit details -
Configuration menu - View commit details
-
Copy full SHA for 480dccc - Browse repository at this point
Copy the full SHA 480dcccView commit details -
Configuration menu - View commit details
-
Copy full SHA for ffc1441 - Browse repository at this point
Copy the full SHA ffc1441View commit details -
Configuration menu - View commit details
-
Copy full SHA for 55825ee - Browse repository at this point
Copy the full SHA 55825eeView commit details -
Configuration menu - View commit details
-
Copy full SHA for 79c5f42 - Browse repository at this point
Copy the full SHA 79c5f42View commit details -
Configuration menu - View commit details
-
Copy full SHA for cb77194 - Browse repository at this point
Copy the full SHA cb77194View commit details -
Configuration menu - View commit details
-
Copy full SHA for 6dd110f - Browse repository at this point
Copy the full SHA 6dd110fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 35f62b5 - Browse repository at this point
Copy the full SHA 35f62b5View commit details -
Add test to validate if task.subdag is None if operator is not SubDag…
…Operator Fix CI by filtering DAG list and removing SubDAGs Figured out that the issue was because the it tries to remove SubDAG from DB which does not exist as a separate row and hence fails.
Configuration menu - View commit details
-
Copy full SHA for ef65c7b - Browse repository at this point
Copy the full SHA ef65c7bView commit details -
Configuration menu - View commit details
-
Copy full SHA for cb688d9 - Browse repository at this point
Copy the full SHA cb688d9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 296c217 - Browse repository at this point
Copy the full SHA 296c217View commit details -
Configuration menu - View commit details
-
Copy full SHA for 09f6f5c - Browse repository at this point
Copy the full SHA 09f6f5cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 2e00f9c - Browse repository at this point
Copy the full SHA 2e00f9cView commit details -
Store JSON schema as static, package-data JSON file
- Since it is now in a single document we can use internal schema references to reuse common chunks - Don't store dag_id against for serialized tasks We never store serialized operators apart from their dags, so we don't need to include the dag_id in the structure again. This change means we need to re-associate the task and the dag after inflating the dag, but does also mean we don't need to pass `visited_dags` around all over the place - Always serialize template_fields as an list, never an array Although many operator classes set these as tuples instead of lists, that is a distinction that is not important to us here, and makes the schema more complex.
Configuration menu - View commit details
-
Copy full SHA for a1102a9 - Browse repository at this point
Copy the full SHA a1102a9View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3423182 - Browse repository at this point
Copy the full SHA 3423182View commit details -
Configuration menu - View commit details
-
Copy full SHA for bfb3f32 - Browse repository at this point
Copy the full SHA bfb3f32View commit details -
Configuration menu - View commit details
-
Copy full SHA for d2436ad - Browse repository at this point
Copy the full SHA d2436adView commit details -
Configuration menu - View commit details
-
Copy full SHA for 927f610 - Browse repository at this point
Copy the full SHA 927f610View commit details -
Configuration menu - View commit details
-
Copy full SHA for 825abe0 - Browse repository at this point
Copy the full SHA 825abe0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 3228d04 - Browse repository at this point
Copy the full SHA 3228d04View commit details -
Configuration menu - View commit details
-
Copy full SHA for c2c0ddb - Browse repository at this point
Copy the full SHA c2c0ddbView commit details -
Configuration menu - View commit details
-
Copy full SHA for 6e8da2c - Browse repository at this point
Copy the full SHA 6e8da2cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 8b84343 - Browse repository at this point
Copy the full SHA 8b84343View commit details -
Configuration menu - View commit details
-
Copy full SHA for 94c69c7 - Browse repository at this point
Copy the full SHA 94c69c7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 01e5f73 - Browse repository at this point
Copy the full SHA 01e5f73View commit details -
Configuration menu - View commit details
-
Copy full SHA for 88ce053 - Browse repository at this point
Copy the full SHA 88ce053View commit details -
Configuration menu - View commit details
-
Copy full SHA for b5ee858 - Browse repository at this point
Copy the full SHA b5ee858View commit details -
Configuration menu - View commit details
-
Copy full SHA for 4932254 - Browse repository at this point
Copy the full SHA 4932254View commit details -
Configuration menu - View commit details
-
Copy full SHA for ebe4ec7 - Browse repository at this point
Copy the full SHA ebe4ec7View commit details -
Configuration menu - View commit details
-
Copy full SHA for a2b27f0 - Browse repository at this point
Copy the full SHA a2b27f0View commit details -
Configuration menu - View commit details
-
Copy full SHA for 89a03a6 - Browse repository at this point
Copy the full SHA 89a03a6View commit details -
Configuration menu - View commit details
-
Copy full SHA for 0754f61 - Browse repository at this point
Copy the full SHA 0754f61View commit details -
Trial reducing size of SerializedDAGs
Don't store defaults, and remove many (but not all) of the type annotations. in the JSON). Combined this reduces the size of our large test DAGs to 40% of what they were (cherry picked from commit a590ed3253f1f41e46a30df966cf17d5dae989f3)
Configuration menu - View commit details
-
Copy full SHA for da0d59a - Browse repository at this point
Copy the full SHA da0d59aView commit details -
Add specific test for schedule_interval serialization
(cherry picked from commit 073464a17ab1b997b50994df7513b57e99f570c5)
Configuration menu - View commit details
-
Copy full SHA for ae03cf0 - Browse repository at this point
Copy the full SHA ae03cf0View commit details -
Support dateutil.relativedelta in SerializedDAGs
This was a valid type for schedule_interval already, so we should continue supporting it (cherry picked from commit ec9d705f1a90790bdcb099196269c77d3cc3d53c)
Configuration menu - View commit details
-
Copy full SHA for 69b242c - Browse repository at this point
Copy the full SHA 69b242cView commit details -
Configuration menu - View commit details
-
Copy full SHA for 4bf9eb5 - Browse repository at this point
Copy the full SHA 4bf9eb5View commit details -
Configuration menu - View commit details
-
Copy full SHA for a498a62 - Browse repository at this point
Copy the full SHA a498a62View commit details -
Configuration menu - View commit details
-
Copy full SHA for b127f37 - Browse repository at this point
Copy the full SHA b127f37View commit details -
Configuration menu - View commit details
-
Copy full SHA for 525cccd - Browse repository at this point
Copy the full SHA 525cccdView commit details -
Configuration menu - View commit details
-
Copy full SHA for 21fafb6 - Browse repository at this point
Copy the full SHA 21fafb6View commit details -
Configuration menu - View commit details
-
Copy full SHA for f4b3e7d - Browse repository at this point
Copy the full SHA f4b3e7dView commit details -
Configuration menu - View commit details
-
Copy full SHA for facbef7 - Browse repository at this point
Copy the full SHA facbef7View commit details -
Configuration menu - View commit details
-
Copy full SHA for 83f2007 - Browse repository at this point
Copy the full SHA 83f2007View commit details -
Just-in-time loading of DagBag in webserver
To save start-up time (and memory) this changes the DabBag to not be populated by the webserver on start up - and when a specific dag is asked for it will be loaded on-demand from the SerializedDAG table. Co-Authored-By: Ash Berlin-Taylor <ash_github@firemirror.com>
Configuration menu - View commit details
-
Copy full SHA for 69f7854 - Browse repository at this point
Copy the full SHA 69f7854View commit details -
Configuration menu - View commit details
-
Copy full SHA for 7c96ed8 - Browse repository at this point
Copy the full SHA 7c96ed8View commit details -
ExtraOperatorLinks are supported if Plugins are registered for them
Configuration menu - View commit details
-
Copy full SHA for 9cb6e28 - Browse repository at this point
Copy the full SHA 9cb6e28View commit details -
Configuration menu - View commit details
-
Copy full SHA for e840616 - Browse repository at this point
Copy the full SHA e840616View commit details
Commits on Oct 23, 2019
-
Configuration menu - View commit details
-
Copy full SHA for 5ccd878 - Browse repository at this point
Copy the full SHA 5ccd878View commit details -
Configuration menu - View commit details
-
Copy full SHA for 2b23a65 - Browse repository at this point
Copy the full SHA 2b23a65View commit details -
Configuration menu - View commit details
-
Copy full SHA for be21a0f - Browse repository at this point
Copy the full SHA be21a0fView commit details -
Configuration menu - View commit details
-
Copy full SHA for 162da54 - Browse repository at this point
Copy the full SHA 162da54View commit details