High memory retention and gradual increase over time

[LourTV]

Apache Airflow version

3.0.6

If "Other Airflow 2 version" selected, which one?

No response

What happened?

Hi,

After upgrading from Airflow 2.3.3 to Airflow 3.0.6, I’ve noticed significantly higher memory usage, even when the system is mostly idle. Over time, memory consumption keeps increasing until swap is heavily used and eventually the system hits OOM. This looks similar to a memory leak.

Observed behavior

• After starting the service, within ~15 hours memory dropped from 16GB free RAM → 5GB free RAM.
• Swap usage grew from 10% → 80%.
• During this time, the same DAGs were running (some every 5 minutes, others every 1 minute).
• top does not show significant memory usage because most of it is reported as inactive memory.

Example from vmstat -s:

20334264 K total memory
13993448 K used memory
3498148 K active memory
10782420 K inactive memory

Tests performed

Disabled all DAGs → memory usage did not drop after 2 hours.
Restarted Airflow components individually and observed memory released:

• DAG processor: ~100MB
• Triggerer: ~300MB
• API server: ~300MB
• Scheduler: ~10GB

Notes

DAGs are unchanged except for updates required by Airflow 3.0.6.
On Airflow 2.3.3, this same setup, with only 10GB RAM, ran stably for months without memory issues.

What you think should happen instead?

Memory used by Airflow components (especially the scheduler) should be released when no longer in use, instead of growing indefinitely over time.

How to reproduce

I simply start the airflow and let my dags run. I have 6 ETL dags running that get data from other systems and insert them into elasticsearchdb.
5 dags run every 5 minutes.
1 dag runs every minute (This one uses taskgroups (multithreading)).

Operating System

RHEL 8.10

Versions of Apache Airflow Providers

apache-airflow-providers-airbyte==5.2.3
apache-airflow-providers-alibaba==3.2.2
apache-airflow-providers-amazon==9.12.0
apache-airflow-providers-apache-beam==6.1.3
apache-airflow-providers-apache-cassandra==3.8.2
apache-airflow-providers-apache-drill==3.1.2
apache-airflow-providers-apache-druid==4.3.0
apache-airflow-providers-apache-flink==1.7.2
apache-airflow-providers-apache-hdfs==4.10.2
apache-airflow-providers-apache-hive==9.1.2
apache-airflow-providers-apache-iceberg==1.3.2
apache-airflow-providers-apache-impala==1.7.2
apache-airflow-providers-apache-kafka==1.10.2
apache-airflow-providers-apache-kylin==3.9.2
apache-airflow-providers-apache-livy==4.4.2
apache-airflow-providers-apache-pig==4.7.2
apache-airflow-providers-apache-pinot==4.8.2
apache-airflow-providers-apache-spark==5.3.2
apache-airflow-providers-apprise==2.1.2
apache-airflow-providers-arangodb==2.8.2
apache-airflow-providers-asana==2.10.2
apache-airflow-providers-atlassian-jira==3.1.2
apache-airflow-providers-celery==3.12.2
apache-airflow-providers-cloudant==4.2.2
apache-airflow-providers-cncf-kubernetes==10.7.0
apache-airflow-providers-cohere==1.5.3
apache-airflow-providers-common-compat==1.7.3
apache-airflow-providers-common-io==1.6.2
apache-airflow-providers-common-messaging==1.0.5
apache-airflow-providers-common-sql==1.27.5
apache-airflow-providers-databricks==7.7.1
apache-airflow-providers-datadog==3.9.2
apache-airflow-providers-dbt-cloud==4.4.2
apache-airflow-providers-dingding==3.8.2
apache-airflow-providers-discord==3.10.2
apache-airflow-providers-docker==4.4.2
apache-airflow-providers-edge3==1.1.3
apache-airflow-providers-elasticsearch==6.3.2
apache-airflow-providers-exasol==4.8.2
apache-airflow-providers-fab==2.4.1
apache-airflow-providers-facebook==3.8.2
apache-airflow-providers-ftp==3.13.2
apache-airflow-providers-git==0.0.6
apache-airflow-providers-github==2.9.2
apache-airflow-providers-google==17.1.0
apache-airflow-providers-grpc==3.8.2
apache-airflow-providers-hashicorp==4.3.2
apache-airflow-providers-http==5.3.3
apache-airflow-providers-imap==3.9.2
apache-airflow-providers-influxdb==2.9.3
apache-airflow-providers-jdbc==5.2.2
apache-airflow-providers-jenkins==4.1.2
apache-airflow-providers-microsoft-azure==12.6.1
apache-airflow-providers-microsoft-mssql==4.3.2
apache-airflow-providers-microsoft-psrp==3.1.2
apache-airflow-providers-microsoft-winrm==3.11.0
apache-airflow-providers-mongo==5.2.2
apache-airflow-providers-mysql==6.3.3
apache-airflow-providers-neo4j==3.10.0
apache-airflow-providers-odbc==4.10.2
apache-airflow-providers-openai==1.6.2
apache-airflow-providers-openfaas==3.8.2
apache-airflow-providers-openlineage==2.6.1
apache-airflow-providers-opensearch==1.7.2
apache-airflow-providers-opsgenie==5.9.2
apache-airflow-providers-oracle==4.1.3
apache-airflow-providers-pagerduty==5.0.2
apache-airflow-providers-papermill==3.11.2
apache-airflow-providers-pgvector==1.5.2
apache-airflow-providers-pinecone==2.3.3
apache-airflow-providers-postgres==6.2.3
apache-airflow-providers-presto==5.9.2
apache-airflow-providers-qdrant==1.4.3
apache-airflow-providers-redis==4.2.0
apache-airflow-providers-salesforce==5.11.2
apache-airflow-providers-samba==4.10.3
apache-airflow-providers-segment==3.8.2
apache-airflow-providers-sendgrid==4.1.3
apache-airflow-providers-sftp==5.3.4
apache-airflow-providers-singularity==3.8.2
apache-airflow-providers-slack==9.1.4
apache-airflow-providers-smtp==2.2.0
apache-airflow-providers-snowflake==6.4.0
apache-airflow-providers-sqlite==4.1.2
apache-airflow-providers-ssh==4.1.3
apache-airflow-providers-standard==1.6.0
apache-airflow-providers-tableau==5.2.0
apache-airflow-providers-telegram==4.8.2
apache-airflow-providers-teradata==3.2.1
apache-airflow-providers-trino==6.3.3
apache-airflow-providers-vertica==4.1.2
apache-airflow-providers-weaviate==3.2.2
apache-airflow-providers-yandex==4.2.0
apache-airflow-providers-ydb==2.2.2
apache-airflow-providers-zendesk==4.10.2

Deployment

Virtualenv installation

Deployment details

OS: RHEL 8.10

• Deployment: systemd (non-containerized)
• Python: 3.12 (virtualenv)
• VM resources: 8 cores, 20GB RAM, 4GB swap
• Airflow configs: Default

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[alkismavridis]

We are also affected. We are running Airflow 3.0.6 inside docker (via docker compose file). We are using LocalExecutor.
The compose file starts couple of containers, all of which have pretty constant memory usage over time except from the scheduler which eats all available memory (we constrained it to 4GB), and then restarts.

Restarting is actually the good scenario, because then it at least works. Sometimes it stays in a "zombie" state where it simply does nothing and no tasks are being triggered for days, until we realise that somethings is off. When this happens, we see in the WebUI tones of tasks being stuck on "Queued" state.

Here is our docker-compose section:

  airflow-scheduler:
    <<: *airflow-common
    container_name: airflow-scheduler
    command: scheduler
    healthcheck:
      test: ["CMD-SHELL", "airflow jobs check --job-type SchedulerJob --local || bash -c 'kill -s 15 -1 && (sleep 10; kill -s 9 -1)'"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 150s
    restart: always
    depends_on:
      <<: *airflow-common-depends-on
      airflow-init:
        condition: service_completed_successfully
    mem_limit: 4000m

Our healthcheck commant is what does the actual restarting of the container. Still, when the scheduler stays in the "zombie" state I described, it seems like airflow jobs check --job-type SchedulerJob --local exits with status code 0, eventhough the scheduler is not really working.

Any help is most welcome. This makes our whole infrastructure unstable :(

[LengYue12389]

I also encountered this problem, it's also version 3.0.6

[vertIcod3r]

I am having the same issue. Simple development docker running 3.0.6 airflow scheduler and apiserver do not seem to be releasing memory . When scheduler reaches the limit it stops working as in #56045

[LengYue12389]

This issue still exists in version 3.1.0

[lucidumio]

I am having the same issue too. We also run Airflow in dockers. The same DAGs ran well in Airflow 2 for almost 2 years, only using less than 15 GB of memory. After migrating to 3.1.0, the scheduler alone takes up over 128 GB of memory in 2 days. This is certainly unacceptable in the production environment. Hopefully this problem can be resolved soon.

[jedcunningham]

Can someone check if there is growth in the dag_version or serialized_dag tables in your environment when memory use climbs? Row count is sufficient.

[alkismavridis]

@jedcunningham will check it tomorrow and let you know.