Status of testing of Apache Airflow 3.1.4rc2 and Task SDK 1.1.4rc2

[ephraimbuddy]

Body

We are kindly requesting that contributors to Apache Airflow RC 3.1.4rc2 help test the RC.

Please let us know by commenting if the issue is addressed in the latest RC.

• Fix refresh token middleware error handling (#56892): @pierrejeambrun
• Handle invalid token in JWTRefreshMiddleware (#56904): @vincbeck
• [v3-1-test] Fix inconsistent Dag hashes when template fields contain unordered dicts (#59091) (#59175): @ephraimbuddy
• [v3-1-test] Update release candidate commands (#59186) (#59187): @ephraimbuddy
• UI Fix for backfill date picker (#57880): @matthieuauger @Tomi-1997
  Linked issues:
  • [UI] Datepicker goes crazy if there is validation error (#57544)
• Add number of queries guard for ui grid (#57977): @pierrejeambrun
• Add Consuming Tasks in asset page header (#58060): @guan404ming
• [v3-1-test] Patch pools should have an optional description (#58066) (#58169): @shubhamraj-git
• [v3-1-test] Fix CI upgrade script to not delete branch from origin remote (#58163) (#58179): @potiuk
• [v3-1-test] fix(migration): disable sqlite fkeys (#58130) (#58192): @Lee-W
• [v3-1-test] Make sure regenerating provider dependencies happens only once (#58198) (#58214): @potiuk
• [v3-1-test] Fix check_files.py script after source tar was renamed (#58216) (#58220): @jscheffl
• [v3-1-test] Fix a few typos in release process for Airflow (#58219) (#58240): @potiuk
• [v3-1-test] Update pyproject.toml files with pytest>=9.0.0 TOML synta… (#58241): @potiuk @Lee-W
• [v3-1-test] Add check for schedule parameter for system tests (#58254) (#58255): @VladaZakharova
• [v3-1-test] Add support for "reconnecting" Supervisor Comms and logs in task processes (#57212) (#58263): @gsaivenkateshzs @ashb @Neon-Face
  Linked issues:
  • @task.virtualenv doesn't show logs (#54706)
  • Couldn't able to access Airflow variables inside @task.virtualenv operator (#51422)
• [v3-1-test] Support for "reconnecting" Supervisor Comms from task process when dag.test() is used (#58147) (#58266): @ashb
  Linked issues:
  • Add support for "reconnecting" Supervisor Comms and logs in task processes (#57212)
• [v3-1-test] Fix atomicity issue in SerializedDagModel.write_dag preventing orphaned DagVersions (#58259) (#58281): @ephraimbuddy
• [v3-1-test] Mask kwargs on illegal args (#58252) (#58283): @jscheffl
• fix(hitl): fix HITLTrigger params dump (#58297): @Lee-W
• [v3-1-test] Fix: TriggerDagRunOperator stuck in deferred state with reset_dag_run (#57756) (#57968) (#58333): @mykola-shyshov
• [v3-1-test] Breeze: Automatically set CHOKIDAR_USEPOLLING for WSL use… (#58345): @potiuk @Lohith625
• [v3-1-test] Fix config list output for multi-line values (#58115) (#58378): @potiuk @nailo2c @zachliu
  Linked issues:
  • Configuration parameter dag_bundle_config_list is generated without indentation (#57355)
• [v3-1-test] Respect default_args in DAG when its set to a "falsy" value (#57853) (#58396): @amoghrajesh
• [v3-1-test] Fix TypeError in parseStreamingLogContent for non-string data (#58314) (#58399): @feluelle
• [v3-1-test] Update Airflow version references to 3.X in README_RELEASE_PROVIDER.md (#58392) (#58402): @jroachgolf84
• [v3-1-test] Add changelog for python client 3.1.3 release (#58394) (#58427): @ephraimbuddy
• [v3-1-test] Minor fixes to the release command (#58426) (#58431): @ephraimbuddy
• make sure the taskInstances's endDate is not null (#58435): @gvergnolle @Pei-Cheng-Yu
  Linked issues:
  • UI: Last Task run duration doesn't show progress for running tasks (#58335)
• [v3-1-test] Limit Python prek hooks to use Python 3.10 (#58439) (#58458): @potiuk
• [v3-1-test] Bump glob in React core UI (#58452) (#58461): @jscheffl
• [v3-1-test] Bump glob from 10.4.5 to 10.5.0 in /airflow-core/src/airflow/api_fastapi/auth/managers/simple/ui (#58443) (#58463): @dependabot[bot]
• [v3-1-test] Minor updates to the python client release guide (#58442) (#58465): @ephraimbuddy
• [v3-1-test] Fix release process for Python Client (#58447) (#58471): @potiuk
• [v3-1-test] Manages cross-distribution dependencies better (#58430) (#58472): @potiuk
• [v3-1-test] Fix task retry logic to respect retries for all exit codes (#58478): @amoghrajesh
• [v3-1-test] Fix Asset URI normalization for user info without password (#51877) (#57650) (#58485): @jgoedeke
• [v3-1-test] Fix operator extra links not appearing on failed tasks (#58227) (#58508): @sunank200
• [v3-1-test] Popup is getting automatically closed when there is a DAG… (#58538): @bbovenzi
• [v3-1-test] Fix group TI tab memory (#58288) (#58557): @guan404ming @RoyLee1224
• [v3-1-test] Fix tag filter with special characters (#58280) (#58558): @guan404ming @steveahnahn
• [v3-1-test] Properly highlight TaskGroup nodes when selected (#58118) (#58559): @guan404ming @steveahnahn
• [v3-1-test] Fix duration chart duration format (#58561) (#58564): @pierrejeambrun
• Add missing Dag run table translation (#58572): @RoyLee1224
• Fix for object rendering in HITL interface (#58611): @stuart23
• [v3-1-test] Use filelock to fix race condition in version suffix test for xdist runs (#58608) (#58616): @amoghrajesh
• [v3-1-test] Fix race condition in test_execute_wildcard for LocalFile… (#58617): @potiuk @sunank200
• [v3-1-test] Improve 404 handling by adding fallback route for invalid URLs (#58546) (#58629): @choo121600
• [v3-1-test] Add number of queries guard for ui structure (#58051) (#58632): @pierrejeambrun
• [v3-1-test] Update refresh token flow (#58649): @vincbeck @xartii
  Linked issues:
  • Update refresh token flow (#55506)
  • Keycloak provider not refreshing auth token properly (#56614)
• [v3-1-test] Fix connection retrieval from secrets backend without conn_type (#58528) (#58664): @amoghrajesh
• [v3-1-test] 56058: Removing refresh state consumption for Dag header (#58389) (#58692): @viveknanda
• [v3-1-test] Make click group naming consistent - all contain now _gro… (#58699): @potiuk
• [v3-1-test] BUGFIX: Fixed timeout_after in run_trigger method of TriggerRunner (#58282) (#58703): @dabla @Lee-W
• [v3-1-test] Fix automatic backport workflow race condition (#58705) (#58706): @amoghrajesh
• [v3-1-test] fix(asset-alias): Preserve Asset.extra when using AssetAlias (#58038) (#58712): @Lee-W
• [v3-1-test] Mask secrets properly when using deprecated import path (#58662) (#58726): @amoghrajesh
• [v3-1-test] Add one as a fallback grid value for number of dagruns (#58680) (#58735): @pierrejeambrun
• [v3-1-test] Redact secrets in rendered templates properly to not expose them on UI (#58767) (#58772): @amoghrajesh
• [v3-1-test] fix(models/dag): handle pre AIP-39 DagRuns (#58229) (#58773): @Lee-W
• [v3-1-test] build: upgrade hatch to 1.16.1 (#58800) (#58804): @Lee-W
• [v3-1-test] Fix incorrect backfill duration calculation in Grid view (#58813) (#58816): @ephraimbuddy
• Manual backport 160db5b from PR #58835 (#58847): @jscheffl
• [v3-1-test] Fix(UI): Improve mobile responsiveness of Dashboard sections (#58811) (#58853): @anshuksi282-ksolves
• [v3-1-test] update zizmor prek hook to version 1.18.0 (#58860) (#58862): @bugraoz93
• [v3-1-test] Fix trigger ui parameter field is dict when param.value is null (#58682) (#58899): @Tomi-1997
• [v3-1-test] Return dag tags in alphabetical order (#58883) (#58904): @stuart23
• [v3-1-test] Select deadlines by id so that deadlines associated with other dagruns of same run_id are not pruned. (#58574) (#58910): @tirkarthi
• [v3-1-test] fix string to datetime pydantic and test (#58791) (#58916): @steveahnahn
• [v3-1-test] Improve startup information printed when Breeze starts (#58926) (#58927): @potiuk
• [v3-1-test] Fix LocalExecutor memory spike by applying gc.freeze (#58365) (#58934): @wjddn279
• Fix backfill max_active_runs race condition with concurrent schedulers (#58807) (#58935): @ephraimbuddy
• [v3-1-test] Do not remove .pyc and .pyo files after building Python (#58944) (#58947): @potiuk @tfagan25
  Linked issues:
  • Airflow Memory Leak in Dag-Processor, Scheduler, Worker, Triggerer (#58509)
• [v3-1-test] Update badges of Airflow: add prek and use nicer table formatting (#58930) (#58960): @potiuk
• [v3-1-test] Fix exception when logging stdout with a custom %-fmt string. (#58959) (#58963): @ashb
• [v3-1-test] Fix assets used only as inlets being incorrectly orphaned (#58303) (#58368) (#58986): @mykola-shyshov
• [v3-1-test] Find only relevant up/downstream tis when clearing (#58987): @uranusjr

Thanks to all who contributed to the release (probably not a complete list!):
@amoghrajesh @zachliu @jroachgolf84 @Tomi-1997 @mykola-shyshov @tfagan25 @RoyLee1224 @gsaivenkateshzs @ashb @steveahnahn @wjddn279 @feluelle @stuart23 @jgoedeke @choo121600 @bugraoz93 @anshuksi282-ksolves @dabla @viveknanda @tirkarthi @potiuk @Lee-W @uranusjr @matthieuauger @jscheffl @xartii @nailo2c @Pei-Cheng-Yu @Neon-Face @vincbeck @sunank200 @VladaZakharova @shubhamraj-git @guan404ming @bbovenzi @ephraimbuddy @pierrejeambrun @gvergnolle @Lohith625

Committer

• I acknowledge that I am a maintainer/committer of the Apache Airflow project.

[matthieuauger]

Hello @ephraimbuddy, I'd be happy to test it. Is there a docker image for this RC1? I looked at the docker registry but didn't find it. Thanks

[ephraimbuddy]

Hello @ephraimbuddy, I'd be happy to test it. Is there a docker image for this RC1? I looked at the docker registry but didn't find it. Thanks

The docker image is still building and would be pushed registry in few minutes

[wjddn279]

I deployed both the 3.1.3 version image and the 3.1.4rc1 image simultaneously using docker compose to test the following PR #58365, and after running 88,100 tasks over 14 hours, I compared the memory usage per process of the scheduler.

As a result, I confirmed that the existing memory usage of 3.9GiB was reduced to 1.6GiB.

airflow:3.1.3	airflow:3.1.4rc1

CONTAINER ID   NAME                               CPU %     MEM USAGE / LIMIT     MEM %     NET I/O           BLOCK I/O         PIDS
0a8484d2f548   af_test2-airflow-scheduler-1       0.81%     1.743GiB / 19.49GiB   8.94%     18.8GB / 10.7GB   656MB / 2.35GB    40
5f1ab497195d   af_test-airflow-scheduler-1        2.02%     4.047GiB / 19.49GiB   20.76%    22.4GB / 12.1GB   638MB / 9.37GB    40

[ephraimbuddy]

Thanks @wjddn279 !

[nailo2c]

Hi, I've tested #58115 from my local device, and it works well. Thank you!

[Lee-W]

#58192, #58773 works as expected

[matthieuauger]

I'd say #57880 partially fixes the problem. There is no problem anymore with day and month fields, but there is still an issue with the year field as mentioned here: #57880 (review)

If you fill completely one of the datetime fields, then focus back on the year it doesn't work properly.

Should I create another issue?

[potiuk]

Checked all my changes, All look good ! The memory improvements look fantastic :)

[jscheffl]

As per devlist technically all looks good, checked bugs but failed miserably in opening browser UI after fresh start with a browser and an expired/invalid JWT token, got a HTTP 500. Started via breeze with command

breeze start-airflow --python 3.12 --load-example-dags --backend postgres --executor EdgeExecutor --answer y --use-airflow-version 3.1.3 --use-distributions-from-dist

And opening browser (as of breeze requiring a new session) expected a login page but received an "INTERNAL SERVER ERROR". Some fix in JWT token handling seems to generate an exception instead of forwarding to login page. In view this is not acceptable as we must expect invalid sessions from users and need to redirect to login page. Exception seen in breeze tmux panel for API server:

2025-12-07T15:56:23.562782Z [error    ] JWT token is not valid: Signature verification failed [airflow.api_fastapi.auth.managers.base_auth_manager] loc=base_auth_manager.py:107
INFO:     172.18.0.1:34366 - "GET / HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
  + Exception Group Traceback (most recent call last):
  |   File "/usr/python/lib/python3.12/site-packages/starlette/_utils.py", line 79, in collapse_excgroups
  |     yield
  |   File "/usr/python/lib/python3.12/site-packages/starlette/middleware/base.py", line 183, in __call__
  |     async with anyio.create_task_group() as task_group:
  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/usr/python/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 783, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/usr/python/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
    |     result = await app(  # type: ignore[func-returns-value]
    |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/python/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
    |     await super().__call__(scope, receive, send)
    |   File "/usr/python/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    |     await self.middleware_stack(scope, receive, send)
    |   File "/usr/python/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
    |     raise exc
    |   File "/usr/python/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
    |     await self.app(scope, receive, _send)
    |   File "/usr/python/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in __call__
    |     await responder(scope, receive, send)
    |   File "/usr/python/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in __call__
    |     await super().__call__(scope, receive, send)
    |   File "/usr/python/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in __call__
    |     await self.app(scope, receive, self.send_with_compression)
    |   File "/usr/python/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    |     await self.app(scope, receive, send)
    |   File "/usr/python/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in __call__
    |     with recv_stream, send_stream, collapse_excgroups():
    |                                    ^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
    |     self.gen.throw(value)
    |   File "/usr/python/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
    |     raise exc
    |   File "/usr/python/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in __call__
    |     response = await self.dispatch_func(request, call_next)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/python/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 45, in dispatch
    |     new_user = await self._refresh_user(current_token)
    |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/python/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 67, in _refresh_user
    |     user = await resolve_user_from_token(current_token)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/usr/python/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 103, in resolve_user_from_token
    |     raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Invalid JWT token")
    | fastapi.exceptions.HTTPException: 403: Invalid JWT token
    +------------------------------------

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/python/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/python/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/python/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/python/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/usr/python/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/usr/python/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in __call__
    await responder(scope, receive, send)
  File "/usr/python/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/python/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in __call__
    await self.app(scope, receive, self.send_with_compression)
  File "/usr/python/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
    await self.app(scope, receive, send)
  File "/usr/python/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in __call__
    with recv_stream, send_stream, collapse_excgroups():
                                   ^^^^^^^^^^^^^^^^^^^^
  File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
    self.gen.throw(value)
  File "/usr/python/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
    raise exc
  File "/usr/python/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in __call__
    response = await self.dispatch_func(request, call_next)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/python/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 45, in dispatch
    new_user = await self._refresh_user(current_token)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/python/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 67, in _refresh_user
    user = await resolve_user_from_token(current_token)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/python/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 103, in resolve_user_from_token
    raise HTTPException(status_code=status.HTTP_403_FORBIDDEN, detail="Invalid JWT token")
fastapi.exceptions.HTTPException: 403: Invalid JWT token

Clearing all local browser session tokens helped though but this we can not release.

How to reproduce:

• Start Airflow via breeze as described above
• Log in (either with a fresh borwser or clear the cookies/sotrages and log in
• Stop Airflow and run "breeze down"
• Start Airflow again and attempt to open a page. Will raise 500 INTERNAL SERVER ERROR

Tested both with Firefox and Chromium on Ubuntu Linux

[bugraoz93]

My changes looks good! Development and CI related

[choo121600]

My changes looks good ;)

[gvergnolle]

@jscheffl I've seen issues like these before, and usually opening Airflow in private mode seems to remove the 500