Airflow 3.1.7 - Internal Server Error

[anavrotski]

Apache Airflow version

3.1.7

If "Other Airflow 3 version" selected, which one?

No response

What happened?

After several hours since deploy - Internal Server Error, GUI can not be opened.

wait-for-airflow-migrations 
wait-for-airflow-migrations /home/airflow/.local/lib/python3.12/site-packages/airflow/providers/common/compat/security/permissions.py:30 RemovedInAirflow4Warning: The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html
wait-for-airflow-migrations 2026-02-05T16:44:38.119574Z [info     [] Context impl MySQLImpl.        [alembic.runtime.migration] loc=migration.py:210
wait-for-airflow-migrations 2026-02-05T16:44:38.119946Z [info     [] Will assume non-transactional DDL. [alembic.runtime.migration] loc=migration.py:213
api-server     |     return meth(
api-server     |            ^^^^^
api-server     |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
api-server     |     return connection._execute_clauseelement(
api-server     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server     |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
api-server     |     ret = self._execute_context(
api-server     |           ^^^^^^^^^^^^^^^^^^^^^^
api-server     |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
api-server     |     conn = self._revalidate_connection()
api-server     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server     |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
api-server     |     self._invalid_transaction()
api-server     |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
api-server     |     raise exc.PendingRollbackError(
api-server     | sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back.  Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
api-server     +------------------------------------
api-server 
api-server During handling of the above exception, another exception occurred:
api-server 
api-server Traceback (most recent call last):
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
api-server     result = await app(  # type: ignore[func-returns-value]
api-server              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
api-server     await super().__call__(scope, receive, send)
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
api-server     await self.middleware_stack(scope, receive, send)
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
api-server     raise exc
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
api-server     await self.app(scope, receive, _send)
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in __call__
api-server     await responder(scope, receive, send)
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in __call__
api-server     await super().__call__(scope, receive, send)
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in __call__
api-server     await self.app(scope, receive, self.send_with_compression)
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
api-server     await self.app(scope, receive, send)
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in __call__
api-server     with recv_stream, send_stream, collapse_excgroups():
api-server                                    ^^^^^^^^^^^^^^^^^^^^
api-server   File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
api-server     self.gen.throw(value)
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
api-server     raise exc
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in __call__
api-server     response = await self.dispatch_func(request, call_next)
api-server                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch
api-server     new_user, current_user = await self._refresh_user(current_token)
api-server                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 80, in _refresh_user
api-server     user = await resolve_user_from_token(current_token)
api-server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 100, in resolve_user_from_token
api-server     return await get_auth_manager().get_user_from_token(token_str)
api-server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/managers/base_auth_manager.py", line 111, in get_user_from_token
api-server     return self.deserialize_user(payload)
api-server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 375, in __call__
api-server     return wrapper(self._obj, *args, **kwargs)
api-server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 358, in wrapper
api-server     v = method(self, *args, **kwargs)
api-server         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 267, in deserialize_user
api-server     return self.session.scalars(select(User).where(User.id == int(token["sub"]))).one()
api-server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 1891, in scalars
api-server     return self._proxied.scalars(
api-server            ^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
api-server     return self._execute_internal(
api-server            ^^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
api-server     result: Result[Any] = compile_state_cls.orm_execute_statement(
api-server                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
api-server     result = conn.execute(
api-server              ^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
api-server     return meth(
api-server            ^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
api-server     return connection._execute_clauseelement(
api-server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
api-server     ret = self._execute_context(
api-server           ^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
api-server     conn = self._revalidate_connection()
api-server            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
api-server     self._invalid_transaction()
api-server   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
api-server     raise exc.PendingRollbackError(
api-server sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back.  Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
stream closed: EOF for airflow/airflow-api-server-f6b468f8-45gwp (wait-for-airflow-migrations)
api-server INFO:     172.25.127.162:45116 - "GET /api/v2/version HTTP/1.1" 200 OK
api-server INFO:     172.25.127.162:45132 - "GET /api/v2/version HTTP/1.1" 200 OK

What you think should happen instead?

No response

How to reproduce

Just deploy and wait ~ 16 hours.

Operating System

linux

Versions of Apache Airflow Providers

fab provider - 3.2.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

Official helm chart 1.1.8

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[abhijeets25012-tech]

i want to contribute

[alex-brunshtein]

How to reproduce: don't logon and don't touch Security menu for 10 min. Try any Security or logon for error.
Works with fixed FAB provider 3.1.2. In FAB 3.2.0 error appear again.

[tirkarthi]

Related #59349,#61480

[abhijeets25012-tech]

Hi @anavrotski , I’m trying to reproduce/fix the PendingRollbackError in Airflow 3.1.7 + FAB 3.2.0.
It seems the problem originates in fab_auth_manager.py, but I cannot open the file in my environment. Could you advise on:

Which exact method/file to look at for the token/user session handling.

Any guidance on reproducing the issue locally or applying a workaround.

[anavrotski]

Hi @anavrotski , I’m trying to reproduce/fix the PendingRollbackError in Airflow 3.1.7 + FAB 3.2.0. It seems the problem originates in fab_auth_manager.py, but I cannot open the file in my environment. Could you advise on:

Which exact method/file to look at for the token/user session handling.

Any guidance on reproducing the issue locally or applying a workaround.

Hi, I think it is better to ask code owners, not users, just reporting the issue

[tvc12]

Same the issue with Airflow 3.1.7 + FAB 3.2.0.

[pranav-pw]

Earlier, we were seeing a SQLAlchemy PendingRollbackError, but the issue is now resolved.

We are using Google OAuth, and the setup works correctly with Airflow 3.1.3 deployed via the Helm chart, when apache-airflow-providers-fab==3.1.2 is used.

Based on this, I believe newer Airflow versions should also work fine. Just make sure to pin the FAB provider version to 3.1.2.