Add guardrail to handle DAG deserialization errors in scheduler#61162
Conversation
f044371 to
db0f75c
Compare
db0f75c to
976a060
Compare
976a060 to
f3b642c
Compare
f3b642c to
5466ffe
Compare
ashb
left a comment
There was a problem hiding this comment.
Better than crashing the scheduler hard, though I wonder if will this continue to try and get it very loop around the scheduler leading to log spam and other knock on issues? Since the dag that is failing will still get returned in the DagModel.dags_needing_dagruns list (Ditto for anything asset related too?)
Not sure how best to address this...
Maybe modifying dags_needing_dagruns query to exclude DAGs with certain warning types? |
Add try/except in _get_current_dag(). If deserialization fails, the scheduler logs the error and continues to the next DAG instead of crashing.
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.