Enhanced Pod Failure Logging in KubernetesExecutor #54115

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merged

jason810496 merged 6 commits into apache:main from HsiuChuanHsu:feature/kubernetes-executor-pod-failed-logging

Aug 22, 2025

Contributor

HsiuChuanHsu commented Aug 4, 2025 •

edited

Loading

Description

Add detailed logging of pod and container failure information when tasks fail, including pod phase, reasons, messages, and container states for better debugging.

Problem Statement

When tasks fail in KubernetesExecutor, operators often lack sufficient information to quickly diagnose the root cause of failures. The current implementation provides minimal failure context, making troubleshooting time-consuming and inefficient.

Solution

Introduced comprehensive logging for pod and container failure states, capturing detailed error information to streamline debugging.

Changes Made

Core Implementation

Enhanced _change_state method in KubernetesExecutor
Added pod status extraction using kube_client.read_namespaced_pod()
Implemented container state analysis for terminated and waiting states
Added graceful exception handling for Kubernetes API failures

Closes: #37548

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

HsiuChuanHsu requested review from ashb, gopidesupavan, hussein-awala, jason810496, jedcunningham and potiuk as code owners

August 4, 2025 23:00

boring-cyborg bot added area:kubernetes-tests area:providers provider:cncf-kubernetes labels

HsiuChuanHsu force-pushed the feature/kubernetes-executor-pod-failed-logging branch from 6db2f66 to 4991b20 Compare

August 4, 2025 23:00

HsiuChuanHsu mentioned this pull request

Print kubernetes failure Status and Reason on pod failures #37548

Closed

2 tasks

hterik reviewed

View reviewed changes

Contributor

hterik left a comment

Nice! 👍

...iders/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py Outdated Show resolved Hide resolved

...iders/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py Outdated Show resolved Hide resolved

...iders/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py Outdated Show resolved Hide resolved

...iders/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py Outdated Show resolved Hide resolved

...iders/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py Outdated Show resolved Hide resolved

...iders/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py Outdated Show resolved Hide resolved

...iders/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py Outdated Show resolved Hide resolved

HsiuChuanHsu requested review from XD-DENG, amoghrajesh, bbovenzi, bolkedebruin, bugraoz93, dstandish, eladkal, ephraimbuddy, jscheffl, kaxil, mobuchowski, o-nikolas, pierrejeambrun, rawwar, ryanahamilton, shubhamraj-git and vincbeck as code owners

August 5, 2025 22:56

HsiuChuanHsu force-pushed the feature/kubernetes-executor-pod-failed-logging branch from 9d554d6 to 2ffc67e Compare

August 5, 2025 23:02

hterik reviewed

View reviewed changes

Contributor

hterik left a comment

Some minor improvements suggested. Otherwise looks ok to me.

Someone else with more knowledge and authority should take a look as well.

...cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_types.py Show resolved Hide resolved

...cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_types.py Outdated Show resolved Hide resolved

...cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py Outdated Show resolved Hide resolved

HsiuChuanHsu force-pushed the feature/kubernetes-executor-pod-failed-logging branch from 2d2e741 to 1725912 Compare

August 10, 2025 23:33

Contributor Author

HsiuChuanHsu commented Aug 11, 2025

@hterik Thanks for the review!
Appreciate all the great suggestions - these are things I hadn't thought of. I've updated the code accordingly.

hterik reviewed

View reviewed changes

Contributor

hterik left a comment

LGTM! 👍

...cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py Show resolved Hide resolved

...cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py Outdated Show resolved Hide resolved

...cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_types.py Show resolved Hide resolved

HsiuChuanHsu force-pushed the feature/kubernetes-executor-pod-failed-logging branch from 1725912 to 6a153c0 Compare

August 11, 2025 13:16

hterik reviewed

View reviewed changes

Contributor

hterik left a comment

Minor nitpicks from me only, otherwise looks good.
Someone else need to approve.

...cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py Outdated Show resolved Hide resolved

...cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_types.py Outdated Show resolved Hide resolved

...cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py Show resolved Hide resolved

Contributor Author

HsiuChuanHsu commented Aug 12, 2025

Thanks again for the review ! 🤩

jason810496 reviewed

View reviewed changes

Member

jason810496 left a comment

Nice! Thanks for the PR!
Only a question regarding where to call collect_pod_failure_details(pod).

providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor.py Outdated Show resolved Hide resolved

...cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py Outdated Show resolved Hide resolved

...cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py Outdated Show resolved Hide resolved

...cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py Show resolved Hide resolved

...cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor_utils.py Outdated Show resolved Hide resolved

HsiuChuanHsu added 6 commits

August 21, 2025 06:43


          feat: Enhanced pod failure logging in KubernetesExecutor

e255fb4

Add detailed logging of pod and container failure information when tasks fail,
including pod phase, reasons, messages, and container states for better debugging.

- Extract pod status (phase, reason, message) on task failure
- Extract container state info (terminated/waiting reasons & messages)
- Add exception handling for Kubernetes API failures
- Only execute additional logging for FAILED task states


          feat(k8s-executor): improve pod failure diagnostics and reduce API calls

9e9731b

Move failure analysis to watcher thread, add detailed container status logging,
and include task keys for better log searchability.


          refactor: Extract pod failure analysis to dedicated TypedDict functions

f046a21

- Add FailureDetails TypedDict for type-safe failure information
- Extract collect_pod_failure_details() function for cleaner separation of concerns
- Create _analyze_init_containers() and _analyze_main_containers() helper functions


          feat(kubernetes): enhance pod failure diagnostics with TypedDict and …

ed8da67

…modular design

1. Introduce FailureDetails TypedDict
2. Implement collect_pod_failure_details Function
3. Enhanced Failure Logging


          fix(kubernetes): improve type safety with Literal types for container…

71adafc

… analysis


          feat: enhance pod failure logging in KubernetesExecutor with detailed…

9fad3af

… failure analysis

- Pass logger as parameter to collect_pod_failure_details for consistent logging context

- Move failure details collection logic to only execute for Failed status pods for better performance
- Update test imports to handle timezone module location changes between Airflow versions

HsiuChuanHsu force-pushed the feature/kubernetes-executor-pod-failed-logging branch from 6da99f2 to 9fad3af Compare

August 20, 2025 23:23

jason810496 approved these changes

View reviewed changes

jason810496 merged commit 89ea580 into apache:main

84 checks passed

Contributor Author

HsiuChuanHsu commented Aug 22, 2025

Thanks for the views! 🤩

HsiuChuanHsu mentioned this pull request

Refactor Kubernetes Executor Types to NamedTuples #54858

Merged

mangal-vairalkar pushed a commit to mangal-vairalkar/airflow that referenced this pull request


          Enhanced Pod Failure Logging in KubernetesExecutor (apache#54115)

9c529aa

* feat: Enhanced pod failure logging in KubernetesExecutor

Add detailed logging of pod and container failure information when tasks fail,
including pod phase, reasons, messages, and container states for better debugging.

- Extract pod status (phase, reason, message) on task failure
- Extract container state info (terminated/waiting reasons & messages)
- Add exception handling for Kubernetes API failures
- Only execute additional logging for FAILED task states

* feat(k8s-executor): improve pod failure diagnostics and reduce API calls

Move failure analysis to watcher thread, add detailed container status logging,
and include task keys for better log searchability.

* refactor: Extract pod failure analysis to dedicated TypedDict functions

- Add FailureDetails TypedDict for type-safe failure information
- Extract collect_pod_failure_details() function for cleaner separation of concerns
- Create _analyze_init_containers() and _analyze_main_containers() helper functions

* feat(kubernetes): enhance pod failure diagnostics with TypedDict and modular design
1. Introduce FailureDetails TypedDict
2. Implement collect_pod_failure_details Function
3. Enhanced Failure Logging

* fix(kubernetes): improve type safety with Literal types for container analysis

* feat: enhance pod failure logging in KubernetesExecutor with detailed failure analysis
- Pass logger as parameter to collect_pod_failure_details for consistent logging context

- Move failure details collection logic to only execute for Failed status pods for better performance
- Update test imports to handle timezone module location changes between Airflow versions

eladkal mentioned this pull request

Status of testing Providers that were prepared on September 05, 2025 #55285

Closed

HsiuChuanHsu deleted the feature/kubernetes-executor-pod-failed-logging branch

September 9, 2025 22:19

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

jason810496 jason810496 approved these changes

potiuk Awaiting requested review from potiuk potiuk is a code owner

ashb Awaiting requested review from ashb ashb is a code owner

gopidesupavan Awaiting requested review from gopidesupavan gopidesupavan is a code owner

jedcunningham Awaiting requested review from jedcunningham jedcunningham is a code owner

hussein-awala Awaiting requested review from hussein-awala hussein-awala is a code owner

kaxil Awaiting requested review from kaxil

amoghrajesh Awaiting requested review from amoghrajesh

dstandish Awaiting requested review from dstandish

eladkal Awaiting requested review from eladkal

vincbeck Awaiting requested review from vincbeck

o-nikolas Awaiting requested review from o-nikolas

mobuchowski Awaiting requested review from mobuchowski

jscheffl Awaiting requested review from jscheffl

bbovenzi Awaiting requested review from bbovenzi

pierrejeambrun Awaiting requested review from pierrejeambrun

ryanahamilton Awaiting requested review from ryanahamilton

shubhamraj-git Awaiting requested review from shubhamraj-git

bugraoz93 Awaiting requested review from bugraoz93

ephraimbuddy Awaiting requested review from ephraimbuddy

rawwar Awaiting requested review from rawwar

bolkedebruin Awaiting requested review from bolkedebruin

XD-DENG Awaiting requested review from XD-DENG

+1 more reviewer

hterik hterik left review comments

Labels

area:kubernetes-tests area:providers provider:cncf-kubernetes