Latest KubernetesPodOperator (Ver: 10.10.0) get 403 error (Forbidden to list K8s Events)

[yovio-rca]

Apache Airflow Provider(s)

cncf-kubernetes

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.17.0
apache-airflow-providers-apprise==2.2.0
apache-airflow-providers-celery==3.13.1
apache-airflow-providers-cncf-kubernetes==10.10.0
apache-airflow-providers-common-compat==1.9.0
apache-airflow-providers-common-io==1.6.0
apache-airflow-providers-common-sql==1.27.1
apache-airflow-providers-docker==4.4.5
apache-airflow-providers-fab==1.5.3
apache-airflow-providers-ftp==3.13.0
apache-airflow-providers-github==2.9.4
apache-airflow-providers-grpc==3.8.3
apache-airflow-providers-http==5.3.0
apache-airflow-providers-imap==3.9.0
apache-airflow-providers-microsoft-winrm==3.12.0
apache-airflow-providers-mongo==5.2.3
apache-airflow-providers-mysql==6.3.5
apache-airflow-providers-opsgenie==5.9.3
apache-airflow-providers-postgres==6.4.1
apache-airflow-providers-redis==4.3.3
apache-airflow-providers-samba==4.11.1
apache-airflow-providers-sftp==5.4.2
apache-airflow-providers-smtp==2.1.0
apache-airflow-providers-sqlite==4.1.0
apache-airflow-providers-ssh==4.1.6
apache-airflow-providers-standard==1.9.2

Apache Airflow version

2.11.0

Operating System

Debian GNU/Linux 12 (bookworm)

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

What happened

Due to New feature in version 10.10.0: KubernetesPodTriggerer reads pod logs instead of KubernetesPodOperator (#57531)

Now airflow triggerer will read logs from airflow, but I suspect part of log read also listing Kubernetes events.

Meanwhile airflow-airflow-pod-log-reader-role created by helm chart does have list events permission.

Earlier version works well because log read was handled by KubernetesPodOperator which run in worker that uses airflow-airflow-pod-launcher-role and has list events permission.

What you think should happen instead

We should fix helm chart template for airflow-airflow-pod-log-reader-role

How to reproduce

Create a dag with a task using KubernetesPodOperator, make it defferable=true.

We will get error like:

kubernetes_asyncio.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: <CIMultiDictProxy('Audit-Id': '63f99090-5c09-45e5-9716-76cb8cb9e006', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '92570511-6de3-4f0c-b903-ef06f04261e2', 'X-Kubernetes-Pf-Prioritylevel-Uid': '271a661f-e33f-4d2d-8831-5979edf49286', 'Date': 'Wed, 26 Nov 2025 16:40:58 GMT', 'Content-Length': '299')>
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"events is forbidden: User "system:serviceaccount:airflow:airflow-triggerer" cannot list resource "events" in API group "" in the namespace "ra-batch"","reason":"Forbidden","details":{"kind":"events"},"code":403}

Anything else

Full stack of the error:
[2025-11-26, 10:42:08 CST] {taskinstance.py:3336} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 776, in _execute_task
result = _execute_callable(context=context, **execute_callable_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 742, in _execute_callable
return ExecutionCallableRunner(
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py", line 252, in run
return self.func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 1816, in resume_execution
return execute_callable(context)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 933, in trigger_reentry
raise AirflowException(message)
airflow.exceptions.AirflowException: Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/triggers/pod.py", line 158, in run
state = await self._wait_for_pod_start()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/triggers/pod.py", line 251, in _wait_for_pod_start
await asyncio.gather(pod_start_task, events_task)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 1013, in watch_pod_events
await watch_pod_events(pod_manager=self, pod=pod, check_interval=check_interval)
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 122, in watch_pod_events
events = await pod_manager.read_pod_events(pod)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 189, in async_wrapped
return await copy(fn, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 111, in call
do = await self.iter(retry_state=retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 153, in iter
result = await action(retry_state)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/_utils.py", line 99, in inner
return call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/init.py", line 420, in exc_check
raise retry_exc.reraise()
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/init.py", line 187, in reraise
raise self.last_attempt.result()
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 449, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
File "/home/airflow/.local/lib/python3.12/site-packages/tenacity/asyncio/init.py", line 114, in call
result = await fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 1006, in read_pod_events
return await self._hook.get_pod_events(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/hooks/kubernetes.py", line 950, in get_pod_events
events: CoreV1EventList = await v1_api.list_namespaced_event(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/client/api_client.py", line 192, in __call_api
raise e
File "/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/client/api_client.py", line 185, in __call_api
response_data = await self.request(
^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/client/rest.py", line 212, in GET
return (await self.request("GET", url,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/kubernetes_asyncio/client/rest.py", line 206, in request
raise ApiException(http_resp=r)
kubernetes_asyncio.client.exceptions.ApiException: (403)
Reason: Forbidden
HTTP response headers: <CIMultiDictProxy('Audit-Id': '63f99090-5c09-45e5-9716-76cb8cb9e006', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'X-Kubernetes-Pf-Flowschema-Uid': '92570511-6de3-4f0c-b903-ef06f04261e2', 'X-Kubernetes-Pf-Prioritylevel-Uid': '271a661f-e33f-4d2d-8831-5979edf49286', 'Date': 'Wed, 26 Nov 2025 16:40:58 GMT', 'Content-Length': '299')>
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"events is forbidden: User "system:serviceaccount:airflow:airflow-triggerer" cannot list resource "events" in API group "" in the namespace "ra-batch"","reason":"Forbidden","details":{"kind":"events"},"code":403}

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[shivainu1413]

Hi, I’m new to contributing to Airflow.
If this issue is still open, I’d like to work on it.

[yovio-rca]

Actually there is already a PR to fix this issue that already been merged but not released: #56872

The author of the PR that causes this issue (#57531) has mentioned that the above PR need to be release first (Helm chart change). However what actually happened was The solution merge but not release yet, and the PR that depends on it get released.