providers-fab: Airflow 3.1.3 sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back

[berglh]

Apache Airflow Provider(s)

fab

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.16.0
apache-airflow-providers-apache-spark==5.4.0
apache-airflow-providers-celery==3.13.0
apache-airflow-providers-cncf-kubernetes==10.11.0
apache-airflow-providers-common-compat==1.10.0
apache-airflow-providers-common-io==1.6.4
apache-airflow-providers-common-messaging==2.0.0
apache-airflow-providers-common-sql==1.28.2
apache-airflow-providers-docker==4.4.4
apache-airflow-providers-elasticsearch==6.3.4
apache-airflow-providers-fab==3.0.3
apache-airflow-providers-ftp==3.13.2
apache-airflow-providers-git==0.0.9
apache-airflow-providers-google==18.1.0
apache-airflow-providers-grpc==3.8.2
apache-airflow-providers-hashicorp==4.3.3
apache-airflow-providers-http==5.4.0
apache-airflow-providers-microsoft-azure==12.8.0
apache-airflow-providers-mysql==6.3.4
apache-airflow-providers-odbc==4.10.2
apache-airflow-providers-openlineage==2.7.3
apache-airflow-providers-postgres==6.4.0
apache-airflow-providers-redis==4.3.2
apache-airflow-providers-sendgrid==4.1.4
apache-airflow-providers-sftp==5.4.1
apache-airflow-providers-slack==9.4.0
apache-airflow-providers-smtp==2.3.1
apache-airflow-providers-snowflake==6.6.0
apache-airflow-providers-ssh==4.1.5
apache-airflow-providers-standard==1.10.0

Apache Airflow version

3.1.3

Operating System

Amazon Linux 2023

Deployment

Official Apache Airflow Helm Chart

Deployment details

EKS v1.33
Airflow Chart 1.18.0
Aurora Postgres 16.9
Replicas 2 of every component
Have three roles and users (admin, group1, group 2)

What happened

I'm receiving the same errors mentioned in the following issues, despite running version 3.0.3 of the FAB provider. The login works for several days before failing with the error:

2025-12-11T23:56:28.992968Z [error    ] Exception on /login/ [GET]     [airflow.providers.fab.www.app] loc=app.py:1744
Traceback (most recent call last)
...
 raise exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. (Background on this error at: https://sqlalche.me/e/14/8s2b)

• sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back #57470
• Airflow Web Interface Crashes with 500 Error (FAB Auth, MySQL Timeout) #57859

Stacktrace

airflow-api-error.log

What you think should happen instead

Ideally, the sqlAlchemy error shouldn't occur.

How to reproduce

Run FAB with in-built user authentication for several days and wait for the error to appear.

Anything else

I'm wondering if the issue may be resulted to running API replicas and hitting some irrecoverable lock state of some kind.

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[keshan17]

we're also having the same issue even updating the latest versions.

[ovintuan]

I also have the same issue, can see that this error can be reproduce by action with new login into Airflow (may by any times) then they will have this issue after along times (hours). You still keep the session with old cookies.

---> I must restart my API-Server for my team for new login. I meet this issues from fab 3.0.1 to 3.0.3.

[rmalai-amdaris]

Confirming I'm seeing the same issue at my end. Steps to reproduce: stay logged in and re-access the airflow after one day. It will usually error out with this error.

[ryan-dyer-sp]

Same problem here.

[alex-brunshtein]

Same problem with:

• apache-airflow 3.1.5
• apache-airflow-providers-fab 3.1.0

Problem started after:

• apache-airflow 3.0.6
• apache-airflow-providers-fab 2.4.1

[vincbeck]

Can someone please try these fixes:

[database]
sql_alchemy_pool_pre_ping = True

[fab]
session_lifetime_minutes = 1440

I would like to see if these config changes fix the issue. If yes, I'll create a PR to change the default value of these configs

[berglh]

@vincbeck I've made the changes as per your suggestion, I will have my team keep an eye on it and let you know.

[database]
sql_alchemy_pool_pre_ping = True

[fab]
session_lifetime_minutes = 1440

[vincbeck]

@vincbeck I've made the changes as per your suggestion, I will have my team keep an eye on it and let you know.

[database]
sql_alchemy_pool_pre_ping = True

[fab]
session_lifetime_minutes = 1440

Thank you!

[alex-brunshtein]

Can someone please try these fixes:

I try it

[rmalai-amdaris]

Can someone please try these fixes:

[database]
sql_alchemy_pool_pre_ping = True

[fab]
session_lifetime_minutes = 1440

I would like to see if these config changes fix the issue. If yes, I'll create a PR to change the default value of these configs

I'm already on the pre-ping settings. The new thing which is shown here is the session lifetime. Doesn't fix the issue.

  # SQLAlchemy engine options for both Airflow database and Celery result backend
  # Optimized for RDS Proxy with TCP keepalive and connection recycling
  sqlalchemy_engine_options = jsonencode({
    pool_size            = 5
    pool_recycle         = var.proxy_idle_client_timeout - 100 # 1700s (100s before proxy timeout)
    pool_pre_ping        = true                                # Test connections on checkout
    pool_reset_on_return = "rollback"                          # Auto-rollback failed transactions on connection return
  })
}