Optimize K8s API usage for watching events

[wolfdn]

Description

This PR optimizes how the KubernetesPodOperator interacts with the Kubernetes API when retrieving events.

Previously, the operator did not pass the resourceVersion parameter when listing events for a pod. This forced Kubernetes to perform a quorum read for every request—an expensive operation. Combined with frequent polling for new events, this created significant load on the Kubernetes API and etcd, especially when many pods were started in parallel.

Best practice is for clients to store the resourceVersion from each response and provide it in subsequent requests. This allows Kubernetes to serve the event list far more efficiently. As stated in the Kubernetes documentation:

Unless you have strong consistency requirements, using resourceVersionMatch=NotOlderThan and a known resourceVersion is preferable since it can achieve better performance and scalability of your cluster than leaving resourceVersion and resourceVersionMatch unset, which requires quorum read to be served.

Reference: https://kubernetes.io/docs/reference/using-api/api-concepts/#semantics-for-get-and-list

With this change, the operator performs one initial event listing without a resourceVersion, and all subsequent requests include the last known resourceVersion.

Additionally, this PR introduces usage of the Kubernetes watch API in deferred (asynchronous) mode. Instead of polling every few seconds, the operator can now watch for new events. This provides two major benefits:

• New events become visible almost immediately.
• The number of requests sent to the Kubernetes API is reduced because the watch connection remains active for a longer period.

We implemented this change after observing a high number of HTTP 429 (rate-limited) responses from our cluster’s API server. One contributing factor was the large volume of GET requests for event listings, which placed heavy load on etcd. After deploying a patched version of the operator with these improvements, the number of 429 responses dropped from several thousand per minute to nearly zero.

Changes

• Remember resourceVersion when retrieving events from K8s API
• Use Kubernetes watch API to watch events when running in deferred (asynchronous) mode
  • There is also a fallback to poll events in deferred mode in case that the Airflow triggerer does not have the permission to watch events (to stay compatible with older versions of Helm chart)
• Improve mechanism to avoid printing duplicate events (now remembers seen event UIDs instead of counting events)
• Add watch verb for events in pod launcher role in Helm chart (required so that Airflow triggerer has permission to watch events)

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

[jscheffl]

Thanks for this - looks impressive improvement!

One mini nit and before merging would leave the PR open a few days for other 4 eyes to review. LGTM in my view.

[jscheffl]

@jedcunningham Would be cool to have your opinion and having this in chart 1.19 release as well.

[jscheffl]

As I heard from @AutomationDev85 about some problems with the asny event polling we might need tomorrow to triage to double-check this is not adding more problems than benefits. Please do not merge before clarified tomorrow (which might be 10.00 CET Tuesday, 9th)

[AutomationDev85]

@jscheffl @wolfdn I added a commit which improves pod start handling by awaiting start completion and cancelling the parallel event watcher. This resolves a sporadic hang in the event stream; after testing, the new approach proved more stable.

[jscheffl]

Looks good and even a bit better like this. Would still wait (as no pressure to merge) for 1-2 days hoping for feedback from others prior merge. Some more eyes might be good.

[potiuk]

This looks lie a fantastic improvement.