DAG-processor RuntimeError: can't start new thread

[zachliu]

Apache Airflow version

3.0.0

If "Other Airflow 2 version" selected, which one?

No response

What happened?

The dag-processor raises RuntimeError: can't start new thread after it has been running for over 6~7 hours. I suspect there might be a memory-or-something-else-leak.

processor-1  | [2025-04-23T12:31:47.475+0000] {dag_processor_job_runner.py:63} ERROR - Exception when executing DagProcessorJob
processor-1  |   + Exception Group Traceback (most recent call last):
processor-1  |   |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
processor-1  |   |     yield
processor-1  |   |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 177, in __call__
processor-1  |   |     async with anyio.create_task_group() as task_group:
processor-1  |   |                ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
processor-1  |   |     raise BaseExceptionGroup(
processor-1  |   | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
processor-1  |   +-+---------------- 1 ----------------
processor-1  |     | Traceback (most recent call last):
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
processor-1  |     |     await self.app(scope, receive, _send)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 176, in __call__
processor-1  |     |     with recv_stream, send_stream, collapse_excgroups():
processor-1  |     |                                    ^^^^^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
processor-1  |     |     self.gen.throw(value)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
processor-1  |     |     raise exc
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 178, in __call__
processor-1  |     |     response = await self.dispatch_func(request, call_next)
processor-1  |     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/middleware.py", line 124, in dispatch
processor-1  |     |     response = await call_next(request)
processor-1  |     |                ^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 156, in call_next
processor-1  |     |     raise app_exc
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
processor-1  |     |     await self.app(scope, receive_or_disconnect, send_no_error)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
processor-1  |     |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1  |     |     raise exc
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1  |     |     await app(scope, receive, sender)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 93, in __call__
processor-1  |     |     await self.process_request(scope=scope, receive=receive, send=send, routes=routes)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 131, in process_request
processor-1  |     |     await route.handle(scope, receive, send)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
processor-1  |     |     await self.app(scope, receive, send)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
processor-1  |     |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1  |     |     raise exc
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1  |     |     await app(scope, receive, sender)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
processor-1  |     |     response = await f(request)
processor-1  |     |                ^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 291, in app
processor-1  |     |     solved_result = await solve_dependencies(
processor-1  |     |                     ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 640, in solve_dependencies
processor-1  |     |     solved = await run_in_threadpool(call, **solved_result.values)
processor-1  |     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
processor-1  |     |     return await anyio.to_thread.run_sync(func)
processor-1  |     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
processor-1  |     |     return await get_async_backend().run_sync_in_worker_thread(
processor-1  |     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2438, in run_sync_in_worker_thread
processor-1  |     |     worker.start()
processor-1  |     |   File "/usr/local/lib/python3.12/threading.py", line 994, in start
processor-1  |     |     _start_new_thread(self._bootstrap, ())
processor-1  |     | RuntimeError: can't start new thread
processor-1  |     +------------------------------------
processor-1  |
processor-1  | During handling of the above exception, another exception occurred:
processor-1  |
processor-1  | Traceback (most recent call last):
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
processor-1  |     await self.app(scope, receive, _send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 176, in __call__
processor-1  |     with recv_stream, send_stream, collapse_excgroups():
processor-1  |                                    ^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
processor-1  |     self.gen.throw(value)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
processor-1  |     raise exc
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 178, in __call__
processor-1  |     response = await self.dispatch_func(request, call_next)
processor-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/middleware.py", line 124, in dispatch
processor-1  |     response = await call_next(request)
processor-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 156, in call_next
processor-1  |     raise app_exc
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
processor-1  |     await self.app(scope, receive_or_disconnect, send_no_error)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
processor-1  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1  |     raise exc
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1  |     await app(scope, receive, sender)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 93, in __call__
processor-1  |     await self.process_request(scope=scope, receive=receive, send=send, routes=routes)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 131, in process_request
processor-1  |     await route.handle(scope, receive, send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
processor-1  |     await self.app(scope, receive, send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
processor-1  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1  |     raise exc
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1  |     await app(scope, receive, sender)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
processor-1  |     response = await f(request)
processor-1  |                ^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 291, in app
processor-1  |     solved_result = await solve_dependencies(
processor-1  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 640, in solve_dependencies
processor-1  |     solved = await run_in_threadpool(call, **solved_result.values)
processor-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
processor-1  |     return await anyio.to_thread.run_sync(func)
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
processor-1  |     return await get_async_backend().run_sync_in_worker_thread(
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2438, in run_sync_in_worker_thread
processor-1  |     worker.start()
processor-1  |   File "/usr/local/lib/python3.12/threading.py", line 994, in start
processor-1  |     _start_new_thread(self._bootstrap, ())
processor-1  | RuntimeError: can't start new thread
processor-1  |
processor-1  | During handling of the above exception, another exception occurred:
processor-1  |
processor-1  | Traceback (most recent call last):
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/jobs/dag_processor_job_runner.py", line 61, in _execute
processor-1  |     self.processor.run()
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 262, in run
processor-1  |     return self._run_parsing_loop()
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 349, in _run_parsing_loop
processor-1  |     self._service_processor_sockets(timeout=poll_time)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 391, in _service_processor_sockets
processor-1  |     need_more = socket_handler(key.fileobj)
processor-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1169, in cb
processor-1  |     gen.send(line)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 554, in handle_requests
processor-1  |     self._handle_request(msg, log)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/processor.py", line 289, in _handle_request
processor-1  |     var = self.client.variables.get(msg.key)
processor-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 293, in get
processor-1  |     resp = self.client.get(f"variables/{key}")
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1054, in get
processor-1  |     return self.request(
processor-1  |            ^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 338, in wrapped_f
processor-1  |     return copy(f, *args, **kw)
processor-1  |            ^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__
processor-1  |     do = self.iter(retry_state=retry_state)
processor-1  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter
processor-1  |     result = action(retry_state)
processor-1  |              ^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 400, in <lambda>
processor-1  |     self._add_action_func(lambda rs: rs.outcome.result())
processor-1  |                                      ^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
processor-1  |     return self.__get_result()
processor-1  |            ^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
processor-1  |     raise self._exception
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__
processor-1  |     result = fn(*args, **kwargs)
processor-1  |              ^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 629, in request
processor-1  |     return super().request(*args, **kwargs)
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 827, in request
processor-1  |     return self.send(request, auth=auth, follow_redirects=follow_redirects)
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
processor-1  |     response = self._send_handling_auth(
processor-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
processor-1  |     response = self._send_handling_redirects(
processor-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
processor-1  |     response = self._send_single_request(request)
processor-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1015, in _send_single_request
processor-1  |     response = transport.handle_request(request)
processor-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_transports/wsgi.py", line 138, in handle_request
processor-1  |     raise seen_exc_info[1]
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/applications.py", line 259, in __call__
processor-1  |     await self.__call__(scope, receive, send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
processor-1  |     await super().__call__(scope, receive, send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
processor-1  |     await self.middleware_stack(scope, receive, send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 179, in __call__
processor-1  |     response = await run_in_threadpool(self.handler, request, exc)
processor-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
processor-1  |     return await anyio.to_thread.run_sync(func)
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
processor-1  |     return await get_async_backend().run_sync_in_worker_thread(
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2438, in run_sync_in_worker_thread
processor-1  |     worker.start()
processor-1  |   File "/usr/local/lib/python3.12/threading.py", line 994, in start
processor-1  |     _start_new_thread(self._bootstrap, ())
processor-1  | RuntimeError: can't start new thread
processor-1  | 2025-04-23 12:31:47 [info     ] Process exited                 [supervisor] exit_code=<Negsignal.SIGTERM: -15> pid=745051 signal_sent=SIGTERM
processor-1  | [2025-04-23T12:31:47.736+0000] {process_utils.py:285} INFO - Waiting up to 5 seconds for processes to exit...
processor-1  |   + Exception Group Traceback (most recent call last):
processor-1  |   |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
processor-1  |   |     yield
processor-1  |   |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 177, in __call__
processor-1  |   |     async with anyio.create_task_group() as task_group:
processor-1  |   |                ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
processor-1  |   |     raise BaseExceptionGroup(
processor-1  |   | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
processor-1  |   +-+---------------- 1 ----------------
processor-1  |     | Traceback (most recent call last):
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
processor-1  |     |     await self.app(scope, receive, _send)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 176, in __call__
processor-1  |     |     with recv_stream, send_stream, collapse_excgroups():
processor-1  |     |                                    ^^^^^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
processor-1  |     |     self.gen.throw(value)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
processor-1  |     |     raise exc
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 178, in __call__
processor-1  |     |     response = await self.dispatch_func(request, call_next)
processor-1  |     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/middleware.py", line 124, in dispatch
processor-1  |     |     response = await call_next(request)
processor-1  |     |                ^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 156, in call_next
processor-1  |     |     raise app_exc
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
processor-1  |     |     await self.app(scope, receive_or_disconnect, send_no_error)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
processor-1  |     |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1  |     |     raise exc
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1  |     |     await app(scope, receive, sender)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 93, in __call__
processor-1  |     |     await self.process_request(scope=scope, receive=receive, send=send, routes=routes)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 131, in process_request
processor-1  |     |     await route.handle(scope, receive, send)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
processor-1  |     |     await self.app(scope, receive, send)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
processor-1  |     |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1  |     |     raise exc
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1  |     |     await app(scope, receive, sender)
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
processor-1  |     |     response = await f(request)
processor-1  |     |                ^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 291, in app
processor-1  |     |     solved_result = await solve_dependencies(
processor-1  |     |                     ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 640, in solve_dependencies
processor-1  |     |     solved = await run_in_threadpool(call, **solved_result.values)
processor-1  |     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
processor-1  |     |     return await anyio.to_thread.run_sync(func)
processor-1  |     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
processor-1  |     |     return await get_async_backend().run_sync_in_worker_thread(
processor-1  |     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |     |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2438, in run_sync_in_worker_thread
processor-1  |     |     worker.start()
processor-1  |     |   File "/usr/local/lib/python3.12/threading.py", line 994, in start
processor-1  |     |     _start_new_thread(self._bootstrap, ())
processor-1  |     | RuntimeError: can't start new thread
processor-1  |     +------------------------------------
processor-1  |
processor-1  | During handling of the above exception, another exception occurred:
processor-1  |
processor-1  | Traceback (most recent call last):
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in __call__
processor-1  |     await self.app(scope, receive, _send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 176, in __call__
processor-1  |     with recv_stream, send_stream, collapse_excgroups():
processor-1  |                                    ^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/lib/python3.12/contextlib.py", line 158, in __exit__
processor-1  |     self.gen.throw(value)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
processor-1  |     raise exc
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 178, in __call__
processor-1  |     response = await self.dispatch_func(request, call_next)
processor-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/middleware.py", line 124, in dispatch
processor-1  |     response = await call_next(request)
processor-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 156, in call_next
processor-1  |     raise app_exc
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 141, in coro
processor-1  |     await self.app(scope, receive_or_disconnect, send_no_error)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
processor-1  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1  |     raise exc
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1  |     await app(scope, receive, sender)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 93, in __call__
processor-1  |     await self.process_request(scope=scope, receive=receive, send=send, routes=routes)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/routing.py", line 131, in process_request
processor-1  |     await route.handle(scope, receive, send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
processor-1  |     await self.app(scope, receive, send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
processor-1  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
processor-1  |     raise exc
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
processor-1  |     await app(scope, receive, sender)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 73, in app
processor-1  |     response = await f(request)
processor-1  |                ^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/routing.py", line 291, in app
processor-1  |     solved_result = await solve_dependencies(
processor-1  |                     ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 640, in solve_dependencies
processor-1  |     solved = await run_in_threadpool(call, **solved_result.values)
processor-1  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
processor-1  |     return await anyio.to_thread.run_sync(func)
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
processor-1  |     return await get_async_backend().run_sync_in_worker_thread(
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2438, in run_sync_in_worker_thread
processor-1  |     worker.start()
processor-1  |   File "/usr/local/lib/python3.12/threading.py", line 994, in start
processor-1  |     _start_new_thread(self._bootstrap, ())
processor-1  | RuntimeError: can't start new thread
processor-1  |
processor-1  | During handling of the above exception, another exception occurred:
processor-1  |
processor-1  | Traceback (most recent call last):
processor-1  |   File "/usr/local/airflow/.local/bin/airflow", line 8, in <module>
processor-1  |     sys.exit(main())
processor-1  |              ^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/__main__.py", line 55, in main
processor-1  |     args.func(args)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/cli/cli_config.py", line 48, in command
processor-1  |     return func(*args, **kwargs)
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/utils/cli.py", line 111, in wrapper
processor-1  |     return f(*args, **kwargs)
processor-1  |            ^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/utils/providers_configuration_loader.py", line 55, in wrapped_function
processor-1  |     return func(*args, **kwargs)
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/dag_processor_command.py", line 54, in dag_processor
processor-1  |     run_command_with_daemon_option(
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/daemon_utils.py", line 86, in run_command_with_daemon_option
processor-1  |     callback()
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/cli/commands/dag_processor_command.py", line 57, in <lambda>
processor-1  |     callback=lambda: run_job(job=job_runner.job, execute_callable=job_runner._execute),
processor-1  |                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/utils/session.py", line 101, in wrapper
processor-1  |     return func(*args, session=session, **kwargs)
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/jobs/job.py", line 347, in run_job
processor-1  |     return execute_job(job, execute_callable=execute_callable)
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/jobs/job.py", line 376, in execute_job
processor-1  |     ret = execute_callable()
processor-1  |           ^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/jobs/dag_processor_job_runner.py", line 61, in _execute
processor-1  |     self.processor.run()
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 262, in run
processor-1  |     return self._run_parsing_loop()
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 349, in _run_parsing_loop
processor-1  |     self._service_processor_sockets(timeout=poll_time)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/manager.py", line 391, in _service_processor_sockets
processor-1  |     need_more = socket_handler(key.fileobj)
processor-1  |                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 1169, in cb
processor-1  |     gen.send(line)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/execution_time/supervisor.py", line 554, in handle_requests
processor-1  |     self._handle_request(msg, log)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/dag_processing/processor.py", line 289, in _handle_request
processor-1  |     var = self.client.variables.get(msg.key)
processor-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 293, in get
processor-1  |     resp = self.client.get(f"variables/{key}")
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1054, in get
processor-1  |     return self.request(
processor-1  |            ^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 338, in wrapped_f
processor-1  |     return copy(f, *args, **kw)
processor-1  |            ^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 477, in __call__
processor-1  |     do = self.iter(retry_state=retry_state)
processor-1  |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 378, in iter
processor-1  |     result = action(retry_state)
processor-1  |              ^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 400, in <lambda>
processor-1  |     self._add_action_func(lambda rs: rs.outcome.result())
processor-1  |                                      ^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
processor-1  |     return self.__get_result()
processor-1  |            ^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
processor-1  |     raise self._exception
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/tenacity/__init__.py", line 480, in __call__
processor-1  |     result = fn(*args, **kwargs)
processor-1  |              ^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/airflow/sdk/api/client.py", line 629, in request
processor-1  |     return super().request(*args, **kwargs)
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 827, in request
processor-1  |     return self.send(request, auth=auth, follow_redirects=follow_redirects)
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 914, in send
processor-1  |     response = self._send_handling_auth(
processor-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 942, in _send_handling_auth
processor-1  |     response = self._send_handling_redirects(
processor-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 979, in _send_handling_redirects
processor-1  |     response = self._send_single_request(request)
processor-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_client.py", line 1015, in _send_single_request
processor-1  |     response = transport.handle_request(request)
processor-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/httpx/_transports/wsgi.py", line 138, in handle_request
processor-1  |     raise seen_exc_info[1]
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/cadwyn/applications.py", line 259, in __call__
processor-1  |     await self.__call__(scope, receive, send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in __call__
processor-1  |     await super().__call__(scope, receive, send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 112, in __call__
processor-1  |     await self.middleware_stack(scope, receive, send)
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 179, in __call__
processor-1  |     response = await run_in_threadpool(self.handler, request, exc)
processor-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/starlette/concurrency.py", line 37, in run_in_threadpool
processor-1  |     return await anyio.to_thread.run_sync(func)
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
processor-1  |     return await get_async_backend().run_sync_in_worker_thread(
processor-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
processor-1  |   File "/usr/local/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2438, in run_sync_in_worker_thread
processor-1  |     worker.start()
processor-1  |   File "/usr/local/lib/python3.12/threading.py", line 994, in start
processor-1  |     _start_new_thread(self._bootstrap, ())
processor-1  | RuntimeError: can't start new thread

What you think should happen instead?

The dag-processor should be stable enough to run indefinitely

How to reproduce

Keep running dag-processor for long period of time.

Operating System

LM 21.3 with docker python:3.12-slim-bookworm

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else?

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[phanikumv]

cc @uranusjr

[uranusjr]

This happens when a process has too many active threads the OS allows no more to be created. This error is just a consequence, we need to instead look into where we spawn threads and where we end them instead.

Adding to the fact we’re not directly spawning threads, they are created deep instead the HTTP server. This will be a fun one to debug…

[uranusjr]

Is more log available right before this happens?

[uranusjr]

https://forums.docker.com/t/138142

This suggests it could be a Docker issue and you may need to add --privileged so the container have access to more thread capacity. Can you try if this works?

[zachliu]

this is the cpu/memory metrics of the dag-processor at the time when the process reboots (not the k8s pod but the process exec airflow dag-processor itself)

damn thing is i couldn't find any error messages at these moments. i searched all these keywords:

• oom
• errno
• health
• heart
• failed
• lost
• zombie
• exit
• etc.

almost like something scheduled the reboots but my [dag_processor] section of the airflow.cfg is entirely default:

airflow/airflow-core/src/airflow/config_templates/config.yml

Lines 2426 to 2597 in 40cb51d

	dag_processor:
	description: |
	Configuration for the Airflow DAG processor. This includes, for example:
	- DAG bundles, which allows Airflow to load DAGs from different sources
	- Parsing configuration, like:
	- how often to refresh DAGs from those sources
	- how many files to parse concurrently
	options:
	dag_bundle_storage_path:
	description: |
	String path to folder where Airflow bundles can store files locally. Not templated.
	If no path is provided, Airflow will use ``Path(tempfile.gettempdir()) / "airflow"``.
	This path must be absolute.
	version_added: 3.0.0
	type: string
	example: "/tmp/some-place"
	default: ~
	dag_bundle_config_list:
	description: |
	List of backend configs. Must supply name, classpath, and kwargs for each backend.
	By default, ``refresh_interval`` is set to ``[dag_processor] refresh_interval``, but that can
	also be overridden in kwargs if desired.
	The default is the dags folder dag bundle.
	Note: As shown below, you can split your json config over multiple lines by indenting.
	See configparser documentation for an example:
	https://docs.python.org/3/library/configparser.html#supported-ini-file-structure.
	version_added: 3.0.0
	type: string
	example: >
	[
	{
	"name": "my-git-repo",
	"classpath": "airflow.providers.git.bundles.git.GitDagBundle",
	"kwargs": {
	"subdir": "dags",
	"tracking_ref": "main",
	"refresh_interval": 0
	}
	}
	]
	default: >
	[
	{{
	"name": "dags-folder",
	"classpath": "airflow.dag_processing.bundles.local.LocalDagBundle",
	"kwargs": {{}}
	}}
	]
	refresh_interval:
	description: |
	How often (in seconds) to refresh, or look for new files, in a DAG bundle.
	version_added: ~
	type: integer
	example: ~
	default: "300"
	parsing_processes:
	description: |
	The DAG processor can run multiple processes in parallel to parse dags.
	This defines how many processes will run.
	version_added: ~
	type: integer
	example: ~
	default: "2"
	file_parsing_sort_mode:
	description: |
	One of ``modified_time``, ``random_seeded_by_host`` and ``alphabetical``.
	The DAG processor will list and sort the dag files to decide the parsing order.
	* ``modified_time``: Sort by modified time of the files. This is useful on large scale to parse the
	recently modified DAGs first.
	* ``random_seeded_by_host``: Sort randomly across multiple DAG processors but with same order on the
	same host, allowing each processor to parse the files in a different order.
	* ``alphabetical``: Sort by filename
	version_added: ~
	type: string
	example: ~
	default: "modified_time"
	max_callbacks_per_loop:
	description: |
	The maximum number of callbacks that are fetched during a single loop.
	version_added: ~
	type: integer
	example: ~
	default: "20"
	min_file_process_interval:
	description: |
	Number of seconds after which a DAG file is parsed. The DAG file is parsed every
	``[dag_processor] min_file_process_interval`` number of seconds. Updates to DAGs are reflected after
	this interval. Keeping this number low will increase CPU usage.
	version_added: ~
	type: integer
	example: ~
	default: "30"
	stale_dag_threshold:
	description: |
	How long (in seconds) to wait after we have re-parsed a DAG file before deactivating stale
	DAGs (DAGs which are no longer present in the expected files). The reason why we need
	this threshold is to account for the time between when the file is parsed and when the
	DAG is loaded. The absolute maximum that this could take is
	``[dag_processor] dag_file_processor_timeout``, but when you have a long timeout configured,
	it results in a significant delay in the deactivation of stale dags.
	version_added: ~
	type: integer
	example: ~
	default: "50"
	dag_file_processor_timeout:
	description: |
	How long before timing out a DagFileProcessor, which processes a dag file
	version_added: ~
	type: string
	example: ~
	default: "50"
	print_stats_interval:
	description: |
	How often should DAG processor stats be printed to the logs. Setting to 0 will disable printing stats
	version_added: ~
	type: integer
	example: ~
	default: "30"
	disable_bundle_versioning:
	description: |
	Always run tasks with the latest code. If set to True, the bundle version will not
	be stored on the dag run and therefore, the latest code will always be used.
	version_added: ~
	type: boolean
	example: ~
	default: "False"
	bundle_refresh_check_interval:
	description: |
	How often the DAG processor should check if any DAG bundles are ready for a refresh, either by hitting
	the bundles refresh_interval or because another DAG processor has seen a newer version of the bundle.
	A low value means we check more frequently, and have a smaller window of time where DAG processors are
	out of sync with each other, parsing different versions of the same bundle.
	version_added: ~
	type: integer
	example: ~
	default: "5"
	stale_bundle_cleanup_interval:
	description: |
	On shared workers, bundle copies accumulate in local storage as tasks run
	and version of the bundle changes.
	This setting represents the delta in seconds between checks for these stale bundles.
	Bundles which are older than `stale_bundle_cleanup_age_threshold` may be removed. But
	we always keep `stale_bundle_cleanup_min_versions` versions locally.
	Set to 0 or negative to disable.
	version_added: ~
	type: integer
	example: ~
	default: "1800"
	stale_bundle_cleanup_age_threshold:
	description: |
	Bundle versions used more recently than this threshold will not be removed.
	Recency of use is determined by when the task began running on the worker,
	that age is compared with this setting, given as time delta in seconds.
	version_added: ~
	type: integer
	example: ~
	default: "21600"
	stale_bundle_cleanup_min_versions:
	description: |
	Minimum number of local bundle versions to retain on disk.
	Local bundle versions older than `stale_bundle_cleanup_age_threshold` will
	only be deleted we have more than `stale_bundle_cleanup_min_versions` versions
	accumulated on the worker.
	version_added: ~
	type: integer
	example: ~
	default: "10"