-
Notifications
You must be signed in to change notification settings - Fork 14.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
#16691 Providing more information in docs for DataprocCreateCluster operator migration #19446
#16691 Providing more information in docs for DataprocCreateCluster operator migration #19446
Conversation
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease. |
@pateash I am facing similar issue with DataprocCreateClusterOperator as well(Airflow 2+) . Also I see metadata field in DataprocCreateClusterOperator is of type Sequence[Tuple[str, str]] but the ClusterGenerator() has type dict for metadata field. |
@guptaneha0908 ClusterGenerator() has been kept for easy migration, ideally you should have our CLUSTER_CONFIG as dictionary. You can avoid using ClusterGenerator by creating CLUSTER_CONFIG from it once and then storing that dictionary and passing to DataProcClusterCreateOperator going forward. |
closes: #16911
Add meaningful description above
I am proposing to add a bit more information in the documentation regarding usage of DataprocClusterCreateOperator() as per refactoring in #6371, users find it a bit difficult to generate CLUSTER_CONFIG.