Skip to content

Comments

Fix deferrable mode in CloudRunExecuteJobOperator#61546

Merged
shahar1 merged 1 commit intoapache:mainfrom
shahar1:fix-deferrable-cloud-run-execute-job-op
Feb 10, 2026
Merged

Fix deferrable mode in CloudRunExecuteJobOperator#61546
shahar1 merged 1 commit intoapache:mainfrom
shahar1:fix-deferrable-cloud-run-execute-job-op

Conversation

@shahar1
Copy link
Contributor

@shahar1 shahar1 commented Feb 6, 2026


related: #60394

Bug description

PR #60394 introduced the transport parameter for CloudRunExecuteJobOperator, but unfortunately it broke its deferrable mode.
There were two underlying issues:

  1. The default transport type of the async hook connection is grpc_asyncio (not grpc), which is more optimized to work with async logic (in fact, the only way - see below).
  2. When providing any transport other than grpc_asyncio - the returned Operation object is not awaitable.

The combination of those issues caused the TypeError: Operation can't be used in 'await' expression when we ran the opeartor in deferrable mode.

Proposed solutions

  1. When running the operator in defferable mode with grpc, we now use grpc_asyncio in the underlying async client.
  2. When running the operator in deferrable mode with rest, we now call the sync client on a backround thread when polling the operation. This is a workaround, as the returned Operation must be awaitable.

Extra details

  • System tests were updated (see screenshot below), with additional tests for the rest transport (sync + async).
  • Docs were updated, including recommandation to prefer usage of grpc when possible and including the caveat of rest transport in deferrable mode.

Discussion points

  • Are we ok with supporting the deferrable rest in its current state? While we could revert to an implementation where I actually prohibited it, I thought that would be useful as a "last resort" for those who can't run with grpc (which is probably the only reason people actually need rest transport at all).

@VladaZakharova @MaksYermak - As we had already released the transport feature, I wanted to avoid making sudden breaking changes by reverting the original PR, so I preferred just to fix the issue instead. I intend to include it in the upcoming release on Tuesday, so I'll appreciate if you could review it by Monday.

image
Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Sonnet 4.5 following the guidelines


  • Read the Pull Request Guidelines for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
  • For fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
  • When adding dependency, check compliance with the ASF 3rd Party License Policy.
  • For significant user-facing changes create newsfragment: {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

@boring-cyborg boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Feb 6, 2026
@shahar1 shahar1 force-pushed the fix-deferrable-cloud-run-execute-job-op branch 4 times, most recently from 10b9ab7 to acd39c4 Compare February 6, 2026 19:35
@shahar1 shahar1 marked this pull request as ready for review February 6, 2026 20:00
@shahar1 shahar1 requested a review from vincbeck February 7, 2026 08:09
Copy link
Contributor

@MaksYermak MaksYermak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general this PR looks good to me, only small note about tag in System Test for documentation.
@VladaZakharova @olegkachur-e what do you think?

Copy link
Contributor

@dabla dabla left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Only small nits

@shahar1 shahar1 force-pushed the fix-deferrable-cloud-run-execute-job-op branch from 7f0186b to aa3b59d Compare February 10, 2026 06:47
@shahar1 shahar1 force-pushed the fix-deferrable-cloud-run-execute-job-op branch from aa3b59d to d6d33d5 Compare February 10, 2026 07:04
@shahar1 shahar1 merged commit 9a5b225 into apache:main Feb 10, 2026
90 checks passed
@shahar1 shahar1 deleted the fix-deferrable-cloud-run-execute-job-op branch February 10, 2026 08:09
Ratasa143 pushed a commit to Ratasa143/airflow that referenced this pull request Feb 15, 2026
choo121600 pushed a commit to choo121600/airflow that referenced this pull request Feb 22, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants