Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missleading Documentation for min_file_process_interval #11122

Closed
CodingJonas opened this issue Sep 24, 2020 · 1 comment
Closed

Missleading Documentation for min_file_process_interval #11122

CodingJonas opened this issue Sep 24, 2020 · 1 comment
Labels
kind:bug This is a clearly a bug kind:documentation

Comments

@CodingJonas
Copy link
Contributor

Apache Airflow version: 1.10.11

Environment: Service as part of a Docker Swarm

What happened:

Setting min_file_process_interval to a high value delays the execution of DAGs by up to the time specified for min_file_process_interval.

What you expected to happen:
We use Airflow as a deployed Docker container. We modify our DAGs locally and deploy a new version of the image every now and then. Thus the DAG definitions the deployed Airflow service uses will never update while the service runs, and to save processing resources, we expected setting min_file_process_interval will make Airflow very seldom look for updated DAG definitions.

From the documentation

after how much time (seconds) a new DAGs should be picked up from the filesystem

This sounds to me as the delay between checks for updated DAG definitions, so I don't understand why this setting delays DAG executions.

@eladkal
Copy link
Contributor

eladkal commented Feb 18, 2021

The description was elaborated in 2.0.1 docs
Number of seconds after which a DAG file is parsed. The DAG file is parsed every min_file_process_interval number of seconds. Updates to DAGs are reflected after this interval. Keeping this number low will increase CPU usage.

see #13664

If you believe this still isn't clear enough please comment and explain.
Closing for now.

@eladkal eladkal closed this as completed Feb 18, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind:bug This is a clearly a bug kind:documentation
Projects
None yet
Development

No branches or pull requests

3 participants