Skip to content

Retry on 409 conflict #54777

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
eladkal merged 4 commits into from
Aug 23, 2025
Merged

Retry on 409 conflict #54777

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ff728a4
Retry on 409 conflict
rom-impala Aug 21, 2025
fabe791
Change comment
rom-impala Aug 21, 2025
cae7918
linting
romsharon98 Aug 23, 2025
3b37860
change or and hirarchy
romsharon98 Aug 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 4 additions & 5 deletions ...rs/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -373,12 +373,11 @@ def sync(self) -> None:
except ApiException as e:
body = json.loads(e.body)
retries = self.task_publish_retries[key]
# In case of exceeded quota errors, requeue the task as per the task_publish_max_retries
# In case of exceeded quota or conflict errors, requeue the task as per the task_publish_max_retries
if (
str(e.status) == "403"
and "exceeded quota" in body["message"]
and (self.task_publish_max_retries == -1 or retries < self.task_publish_max_retries)
):
(str(e.status) == "403" and "exceeded quota" in body["message"])
or (str(e.status) == "409" and "object has been modified" in body["message"])
) and (self.task_publish_max_retries == -1 or retries < self.task_publish_max_retries):
self.log.warning(
"[Try %s of %s] Kube ApiException for Task: (%s). Reason: %r. Message: %s",
self.task_publish_retries[key] + 1,
Expand Down
10 changes: 10 additions & 0 deletions providers/cncf/kubernetes/tests/unit/cncf/kubernetes/executors/test_kubernetes_executor.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -373,6 +373,16 @@ def setup_method(self) -> None:
State.FAILED,
id="12345 fake-unhandled-reason (task_publish_max_retries=1) (retry failed)",
),
pytest.param(
HTTPResponse(
body='{"message": "the object has been modified; please apply your changes to the latest version and try again"}',
status=409,
),
1,
True,
State.SUCCESS,
id="409 conflict",
),
],
)
@mock.patch("airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher")
Expand Down