You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Setting min_file_process_interval to a high value delays the execution of DAGs by up to the time specified for min_file_process_interval.
What you expected to happen:
We use Airflow as a deployed Docker container. We modify our DAGs locally and deploy a new version of the image every now and then. Thus the DAG definitions the deployed Airflow service uses will never update while the service runs, and to save processing resources, we expected setting min_file_process_interval will make Airflow very seldom look for updated DAG definitions.
The description was elaborated in 2.0.1 docs Number of seconds after which a DAG file is parsed. The DAG file is parsed every min_file_process_interval number of seconds. Updates to DAGs are reflected after this interval. Keeping this number low will increase CPU usage.
Apache Airflow version: 1.10.11
Environment: Service as part of a Docker Swarm
uname -a
): Linux 5ed6acafbcee 5.3.0-1034-aws Adding support for catalog option to PrestoHook + refactor #36-Ubuntu SMP Tue Aug 18 08:58:43 UTC 2020 x86_64 GNU/LinuxWhat happened:
Setting
min_file_process_interval
to a high value delays the execution of DAGs by up to the time specified formin_file_process_interval
.What you expected to happen:
We use Airflow as a deployed Docker container. We modify our DAGs locally and deploy a new version of the image every now and then. Thus the DAG definitions the deployed Airflow service uses will never update while the service runs, and to save processing resources, we expected setting
min_file_process_interval
will make Airflow very seldom look for updated DAG definitions.From the documentation
This sounds to me as the delay between checks for updated DAG definitions, so I don't understand why this setting delays DAG executions.
The text was updated successfully, but these errors were encountered: