Extra links for Google Cloud operators

[mik-laj]

Hello,

Airflow has support for extra links, but it is wide-supported common. Only 4 operators support it.

I think we should focus on the most popular services first.

• Cloud Storage Cloud Storage assets & StorageLink update #23865
• BigQuery Bigquery assets #23165
• Dataflow Dataflow Assets #21639
• Dataproc Add extra links for google dataproc #10343

In the next step, we can add operators for the remaining services.

• AutoML Auto ML assets #25466
• BigQuery Data Transfer Service Add links for BigQuery Data Transfer #22280
• Bigtable Create links for Biqtable operators #23164
• Cloud Build
• Compute Engine Add new Compute Engine Operators and fix system tests #25608
• Cloud Data Loss Prevention (DLP)
• DataFusion
• Datacatalog
• Datastore Create links for Cloud Datastore operators #23110
• Deployment Manager
• Cloud Functions
• Cloud Firestore
• Cloud Key Management Service (KMS)
• Cloud Life Sciences
• Kubernetes Engine Add links for Google Kubernetes Engine operators #24786
• Machine Learning Engine Cloud ML Engine operators assets (AIP-47) #26836
• Cloud Memorystore
• Natural Language
• Cloud Pub/Sub
• Cloud Secret Manager
• Cloud Spanner
• Cloud Speech-to-Text
• Cloud SQL
• Cloud Stackdriver
• Storage Transfer Service
• Cloud Tasks
• Cloud Text-to-Speech
• Cloud Translation
• Cloud Video Intelligence
• Cloud Vision

If you decide to finish this ticket you don't have to do all the work yourself. One PR can only deal with a single file/provider and it's ok.

Are you wondering how to start contributing to this project? Start by reading our contributor guide

If you haven't used the GCP yet, after creating the account you will get $300. The implementation of this task will allow a better understanding of GCP services. If anyone is interested in this task, I am willing to provide all the necessary tips and information.

If anyone is interested in this task, I am happy to provide all the necessary tips and information.

Best regards,
Kamil

[mik-laj]

@dossett Would you like to help with this ticket?

[dossett]

@mik-laj Once we upgrade past 1.10.3 I will definitely try to contribute to this. Hopefully soon!

[edithturn]

@mik-laj I have some experience with Google Cloud, I would really happy to work on this issue. Could you please provide me with the tips and information to start it? I will appreciate it.

[Dark-Knight11]

Hi, I would like to add operators for Compute Engine service. Can you please guide me on it?

[potiuk]

Just look for similar PRs (above). Those are the best to see what needs to be done. For general contribution guildeline please follow https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst :)

[lwyszomi]

We already working to update all existing google operator, I think we sent several PR related to this issue. @wojsamjan please include this issue in the PR's with the links to the google operators.

[eladkal]

@lwyszomi Do you need this issue for tracking? it seems like you are doing it independently and no one else in the community chime in this effort so I wonder if we should keep this issue open?
If you would like community assistance I suggest maybe add some description/instructions so new contributors can help?

[lwyszomi]

@eladkal I forgot that this issue exists, I will check with my list and I will update this one. Maybe is not needed any more

[lwyszomi]

@eladkal I don't have permissions to update this issue, but I added link to all PR that was sent to the community.

[eladkal]

OK since this list doesn't really help you I'm closing this issue.
If you will need community help please comment and we will revisit what is the best way to handle it (single vs multiple issues)