Huge increase on DagFileProcessorProcess #11936
Description
Apache Airflow version: 1.10.12
Kubernetes version (if you are using kubernetes) (use kubectl version): 1.10
Environment:
- Cloud provider or hardware configuration: AWS
- OS (e.g. from /etc/os-release): Debian GNU/Linux 9 (stretch)
- Kernel (e.g.
uname -a): 4.19.123-coreos
What happened:
After rolling out to Airflow 1.10.12, we've started to notice a huge increase on loading up DAG files. Some of our DAGs started timing out, and we see the following error on the logs:
Processor for /airflow/airflow-dags/dags/my_dag.py with PID 27954 started at 2020-10-29T02:21:04.630540+00:00 has timed out, killing it
Processor for /airflow/airflow-dags/dags/my_dag.py exited with return code -9
What you expected to happen:
We didn't have this issue on the 1.10.9. Our DAGs load time went from 1 second to over 40 seconds, and therefore going over the dag_file_processor_timeout setting.
How to reproduce it:
Any DAG with more than 200 tasks seem to be affected. On our case we have one DAG full of Sensors to other DAGs, but the issue doesn't seem to be related to a particular type of task, but rather the total size of the DAG.
Anything else we need to know:
The problem occurs every time the scheduled time is reached, so that the tasks are created, and the process times out.