File descriptor leak: task log files not closed, causing 'Too many open files' in Airflow #51624
Description
Apache Airflow version
3.0.2
If "Other Airflow 2 version" selected, which one?
No response
What happened?
When using the LocalExecutor with the default logging configuration, task log files are not properly closed after task completion. Over time, Airflow crashes with the error: too many open file
What you think should happen instead?
File descriptors for task log files should be properly and promptly closed once the task finishes, to prevent resource exhaustion. This is especially important in long-running environments with many tasks.
How to reproduce
Run this dag:
import pendulum
from airflow import DAG
from airflow.providers.http.operators.http import HttpOperator
def send_or_not(openweather_data):
return 'cloudiness' in openweather_data['measure']
with DAG(
dag_id='test-fd-leak',
start_date=pendulum.datetime(2025, 5, 5, 0,0, tz="UTC"),
schedule='*/5 * * * *',
catchup=False,
render_template_as_native_obj=True,
) as dag:
get_data = HttpOperator(
task_id='get_github_data',
http_conn_id='api_github_endpoint',
method='GET',
endpoint='/repos/apache/airflow/releases',
headers={"Accept": "application/json"},
)
get_data
api_github_endpoint is a http connection with host: api.github.com scheme: https
Operating System
Debian GNU/Linux 12 (bookworm)
Versions of Apache Airflow Providers
apache-airflow-providers-amazon==9.8.0
apache-airflow-providers-common-compat==1.7.0
apache-airflow-providers-common-io==1.6.0
apache-airflow-providers-common-sql==1.27.1
apache-airflow-providers-elasticsearch==6.3.0
apache-airflow-providers-fab==2.2.0
apache-airflow-providers-git==0.0.2
apache-airflow-providers-http==5.3.0
apache-airflow-providers-postgres==6.2.0
apache-airflow-providers-samba==4.10.0
apache-airflow-providers-smtp==2.1.0
apache-airflow-providers-standard==1.2.0
Deployment
Virtualenv installation
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct