Make DataprocDeleteClusterOperator idempotent #60083

chirodip98 · 2026-01-03T15:50:38Z

Fixes : #59812 (comment)
In Case of Cluster does not exists , DataprocDeleteClusterOperator will simply log meaningfull message and will exit.

chirodip98

@shivannakarthik , @VladaZakharova and @MaksYermak Can you review now?

chirodip98 · 2026-01-03T16:18:04Z

dataproc_delete_cluster_test.html

System Test Logs - Updated

jscheffl

Looks good from reading but really would be good having a review from one of the Google team members as well

shahar1 · 2026-01-03T17:54:49Z

cc: @VladaZakharova

providers/google/tests/unit/google/cloud/operators/test_dataproc.py

providers/google/src/airflow/providers/google/cloud/operators/dataproc.py

chirodip98 · 2026-01-05T06:33:46Z

Added deferable test cases
Corrected the timezone (airflow.util.timezone is non-existent , falling back to airflow.sdk.timezone)
Updated System Test (reports updated)

Requesting review : @MaksYermak @VladaZakharova

chirodip98 · 2026-01-05T09:47:14Z

Can anyone help / suggest why the test are failing?
https://github.com/apache/airflow/blob/main/airflow-core/src/airflow/utils/__init__.py

providers/google/src/airflow/providers/google/cloud/operators/dataproc.py

providers/google/tests/system/google/cloud/dataproc/example_dataproc_delete_cluster.py

providers/google/src/airflow/providers/google/cloud/operators/dataproc.py

…pache#60259) * Initial plan * Add portForward section to _build_skaffold_config for API server Co-authored-by: jason810496 <68415893+jason810496@users.noreply.github.com> * Add kubectl section instead of portForward section * Fix: hooks should place under helm instead of kubectl --------- Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>

* refactor: Implemented BaseAsyncOperator in task-sdk * refactor: Now PythonOperator extends BaseAsyncOperator * refactor: Also implement BaseAsyncOperator in common-compat provider to support older Airflow versions --------- Co-authored-by: Jason(Zhe-You) Liu <68415893+jason810496@users.noreply.github.com>

…pache#59883) Extract the listeners infrastructure to `shared/listeners/` library to eliminate cross dependencies between airflow-core and task-sdk. - ListenerManager and hookimpl marker now in shared library - Hook specs split by callers: - shared: lifecycle, taskinstance (called from both sdk and core) - core: dagrun, asset, importerrors (called only from core) - sdk registers only specs it actually uses (lifecycle, taskinstance) - core registers all specs for full listener support

* Refactor airflow-core/tests cli commands to use SQLA2 * Refactor airflow-core/tests cli commands to use SQLA2

…)" (apache#60266) This reverts commit 9cab6fb.

…pache#60264) When installing Airflow 2 in Breeze, we need to add pydantic as extra, because pydantic in Airflow 2 was not a required dependency and installation of airflow even with constraints willl not downgrade pydantic to the version that was supported in Airflow 2. When we detect that airflow 2 is installed (either by specified version number or by retrieving the version from the dist package) we simply extend the extras with pydantic and that causes airflow installation to downgrade pydantic to the version that is specified in constraints of selected airflow version.

…tests. (apache#60027) Co-authored-by: Sameer Mesiah <smesiah971@gmail.com>

…trib into fix/59812

chirodip98 · 2026-01-09T04:16:55Z

kindly approve workflows

shahar1 · 2026-01-10T08:28:37Z

kindly approve workflows

It seems that workflows were approved.
Please fix the PR according to Vlada's comments, thanks!

chirodip98 · 2026-01-12T18:22:32Z

Can we have review @VladaZakharova @shivannakarthik @shahar1 .
I have refactored the code.

shahar1 · 2026-01-12T20:11:43Z

My approval is a bit hesitated - IMO we should have an explicit flag for that (something like ignore_is_missing in Microsoft azure operator), as I could imagine some edge scenarios where actions are taken upon "not found" entities.
However, considering the benefits for solving the issue in discussion and the fact that in other GCP operators we already catch NotFound error, I'm ok with merging it as-is for now.
I'll put this matter into a discussion in the dev. list, and we could add this flag later if the community finds it necessary.

boring-cyborg · 2026-01-12T20:14:10Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

chirodip98 · 2026-01-13T02:05:15Z

Let's create a issue to update the documentation for the same accordingly.

shahar1 · 2026-01-13T21:05:21Z

Let's create a issue to update the documentation for the same accordingly.

Feel free, but please keep in mind that specifically for documentation it's quicker just to create the PR directly instead of creating an issue for that :)

boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Jan 3, 2026

chirodip98 mentioned this pull request Jan 3, 2026

DataprocDeleteClusterOperator to raise AirflowSkipException if the cluster is already Deleted / Not Found #59854

Closed

chirodip98 commented Jan 3, 2026

View reviewed changes

chirodip98 mentioned this pull request Jan 3, 2026

DataprocDeleteClusterOperator fails if cluster was already deleted by DataprocCreateClusterOperator(delete_on_error=True) #59812

Closed

2 tasks

jscheffl approved these changes Jan 3, 2026

View reviewed changes

shahar1 changed the title ~~FIX: DataprocDeleteClusterOperator to raise AirflowSkipException if the cluster is already Deleted / Not Found~~ DataprocDeleteClusterOperator to raise AirflowSkipException if the cluster is already Deleted / Not Found Jan 3, 2026

shahar1 approved these changes Jan 3, 2026

View reviewed changes

shivannakarthik suggested changes Jan 5, 2026

View reviewed changes

providers/google/tests/unit/google/cloud/operators/test_dataproc.py Show resolved Hide resolved

providers/google/src/airflow/providers/google/cloud/operators/dataproc.py Outdated Show resolved Hide resolved

chirodip98 force-pushed the fix/59812 branch from db2b7fe to c9d0df1 Compare January 5, 2026 06:38

VladaZakharova suggested changes Jan 5, 2026

View reviewed changes

chirodip98 changed the title ~~DataprocDeleteClusterOperator to raise AirflowSkipException if the cluster is already Deleted / Not Found~~ DataprocDeleteClusterOperator to log message and not cause failure, if the cluster is already Deleted / Not Found. Jan 5, 2026

chirodip98 force-pushed the fix/59812 branch from cd83ecf to 6d37d79 Compare January 5, 2026 12:06

chirodip98 requested review from XD-DENG, amoghrajesh, ashb, bugraoz93, gopidesupavan, jason810496, kaxil, mobuchowski, o-nikolas and potiuk as code owners January 5, 2026 12:06

potiuk requested review from VladaZakharova and shivannakarthik January 7, 2026 13:46

fix: ruff concerns

d2bd2b2

jason810496 and others added 9 commits January 9, 2026 09:43

Refactor airflow-core/tests cli commands to use SQLA2 (apache#60263)

045a283

* Refactor airflow-core/tests cli commands to use SQLA2 * Refactor airflow-core/tests cli commands to use SQLA2

Revert "Add support for async callables in PythonOperator (apache#59087…

7b7c416

…)" (apache#60266) This reverts commit 9cab6fb.

Fix OAuth token refresh for long-running SnowflakeHook tasks and add …

d92b5b1

…tests. (apache#60027) Co-authored-by: Sameer Mesiah <smesiah971@gmail.com>

fix: reverting timezone import changes to suggested one

5323f28

Merge branch 'fix/59812' of https://github.com/chirodip98/airflow-con…

e44105c

…trib into fix/59812

chirodip98 requested review from VladaZakharova, shivannakarthik and uranusjr January 9, 2026 04:15

Merge branch 'main' into fix/59812

7b5f88a

shahar1 self-requested a review January 10, 2026 08:24

shahar1 approved these changes Jan 12, 2026

View reviewed changes

shahar1 changed the title ~~DataprocDeleteClusterOperator to log message and not cause failure, if the cluster is already Deleted / Not Found.~~ Make DataprocDeleteClusterOperator idempotent Jan 12, 2026

shahar1 merged commit bd133c0 into apache:main Jan 12, 2026
89 checks passed

shahar1 mentioned this pull request Jan 12, 2026

Add 'ignore_if_missing' to DataprocDeleteClusterOperator #60433

Merged

1 task

chirodip98 deleted the fix/59812 branch January 13, 2026 02:05

iharsh02 pushed a commit to iharsh02/airflow that referenced this pull request Jan 13, 2026

Make DataprocDeleteClusterOperator idempotent (apache#60083)

4e8fe71

This was referenced Jan 14, 2026

Status of testing Providers that were prepared on January 13th, 2026 #60496

Closed

Status of testing Providers that were prepared on January 17, 2026 #60741

Closed

jason810496 pushed a commit to jason810496/airflow that referenced this pull request Jan 22, 2026

Make DataprocDeleteClusterOperator idempotent (apache#60083)

959adcf

Make DataprocDeleteClusterOperator idempotent #60083

Make DataprocDeleteClusterOperator idempotent #60083

Conversation

chirodip98 commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chirodip98 left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chirodip98 commented Jan 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jscheffl left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

shahar1 commented Jan 3, 2026

Uh oh!

Uh oh!

Uh oh!

chirodip98 commented Jan 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chirodip98 commented Jan 5, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

chirodip98 commented Jan 9, 2026

Uh oh!

shahar1 commented Jan 10, 2026

Uh oh!

chirodip98 commented Jan 12, 2026

Uh oh!

shahar1 commented Jan 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

boring-cyborg bot commented Jan 12, 2026

Uh oh!

chirodip98 commented Jan 13, 2026

Uh oh!

shahar1 commented Jan 13, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

20 participants

chirodip98 commented Jan 3, 2026 •

edited

Loading

chirodip98 left a comment •

edited

Loading

chirodip98 commented Jan 3, 2026 •

edited

Loading

jscheffl left a comment •

edited

Loading

chirodip98 commented Jan 5, 2026 •

edited

Loading

shahar1 commented Jan 12, 2026 •

edited

Loading