Update SDKs for google provider package #30067

lwyszomi · 2023-03-13T10:14:58Z

As everyone know google provider package have a lot of old dependencies. I would like to start migration to the latest versions of the SDK. For now we are blocked by some other dependencies because they are using protobuf<4.

apache-beam
mysql-connector-python
yandexcloud

Also in the google SDKs we had a lot of breaking changes so after updating we need to adjust broken operators. I did investigation how big is this problem and I'm attaching the list of services where some of the operators are broken:

Fixes: #27292

^ Add meaningful description above

Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

felicienveldema · 2023-03-21T09:26:36Z

Got referred to here from #27292 .
I'm experiencing an issue where I am not able to upgrade the deprecated Google Ads API 18.

I eventually get stuck with apache-airflow-providers-google depending on google-cloud-secret-manager < 2.x . Which depend on protobuf 3 which causes my predicament. Higher versions depend on protobuf > 4.5.x .

Is there progress on this ticket? Just wondering but keep up the good work

lwyszomi · 2023-03-21T10:04:58Z

@felicienveldema We are still working on the changes in the google-provider package, but still we have some problems with the dependencies in 3 packages because they are still depends on protobuf<4.0 and probably at May we will have updates. So I think the will have new google-provider version supporting latest version of SDKs at the end of May or later.

r-richmond · 2023-03-27T04:53:44Z

@potiuk

I got curious about the 3 listed packages above causing issues. One of them mysql-connector-python's latest version has protobuf pinned at 3.20.3. How do you anticipate this conflict will be resolved? (After reviewing the commit activity & Oracle's ownership I'm assuming they will be slow to update if they do at all, apologies if this is a bad assumption).

potiuk · 2023-03-27T06:27:10Z

I got curious about the 3 listed packages above causing issues. One of them mysql-connector-python's latest version has protobuf pinned at 3.20.3. How do you anticipate this conflict will be resolved? (After reviewing the commit activity & Oracle's ownership I'm assuming they will be slow to update if they do at all, apologies if this is a bad assumption).

I have not looked at it yet. Do you have some ideas ?

potiuk · 2023-03-27T06:29:13Z

Generally the options are:

replace the library with something else
exclude such provider (and stop releasing it) that holds us back
vendor-in the library and bump the dependency
make the dependency optional and skip tests for it
work with the maintainers and actively help them to upgrade

So we have a number of options we can follow.

potiuk · 2023-03-27T06:44:24Z

I just opened the issue to: yandex-cloud: yandex-cloud/python-sdk#71 and I will prepare support for disabling providers and excluding them if they are holding us back (cc: @eladkal).

I will also raise this to our devlist.

potiuk · 2023-03-27T07:03:09Z

Devlist discussion started: https://lists.apache.org/thread/j98bgw9jo7xr4fvjh27d6bfoyxr1omcm (especially CC: @eladkal especially) I am curious what you think.

potiuk · 2023-03-27T07:07:31Z

FYI: We have no problem with apache-beam: apache/beam#24599 - 2 weeks ago they marged protobuf bump, so we just neeed to wait for the next release

potiuk · 2023-03-27T08:04:36Z

I also asked Oracle/MySQL in their forums (the only way we can do it) https://forums.mysql.com/read.php?50,708413 and see what they say. But I am also all for disabling mysql provider if they don't respond.

potiuk · 2023-03-27T08:52:27Z

We have first reaction: yandex-cloud/python-sdk#71 (comment)

r-richmond · 2023-03-27T17:39:49Z

But I am also all for disabling mysql provider if they don't respond.

I'm also 100% for this (but I've got some bias in that I always use Postgres over mysql). I hesitated to suggest that since I was unsure if that would impact offering mysql as one of Airflow Meta DB backends.

potiuk · 2023-03-27T18:20:26Z

I'm also 100% for this (but I've got some bias in that I always use Postgres over mysql). I hesitated to suggest that since I was unsure if that would impact offering mysql as one of Airflow Meta DB backends.

I have to check but I think this has actually nothing to do with mysql metadata backend. For that we are using sqlalchemy and it has a few drivers it can choose from. And I think our driver for CI/tests is mysqlclient not mysql-connector-python.

BTW. This is another possibilty to rewrite the hooks to use mysqlclient. I might take a look at that actually.

cgadam · 2023-03-29T12:35:56Z

Hi, is it too risky for Airflow to just update from google-ads v18.0.0 to v18.2.0? See: #30353

Today v11 of Google API is sunsetting: https://developers.google.com/google-ads/api/docs/sunset-dates which means that current latest version of Airflow won't be officially compatible (due to its constraint file: https://raw.githubusercontent.com/apache/airflow/constraints-2.5.2/constraints-3.7.txt) with any google-ads package that can actually interact with the Google Ads API. (API calls will start failing)

Latest compatibility to a new API version was added in: https://github.com/googleads/google-ads-python/pull/672/files#diff-91c5b46dc84a94604a4e4d0caed9bf85590a2eddbb12d2e8dc80badf324a9dfbR9 (v17.0.0) and it added support v11 of the API.

v18.2.0 actually added support for v12 of the API. See here.

cgadam · 2023-03-30T11:54:45Z

Hi, is it too risky for Airflow to just update from google-ads v18.0.0 to v18.2.0? See: #30353

Today v11 of Google API is sunsetting: https://developers.google.com/google-ads/api/docs/sunset-dates which means that current latest version of Airflow won't be officially compatible (due to its constraint file: https://raw.githubusercontent.com/apache/airflow/constraints-2.5.2/constraints-3.7.txt) with any google-ads package that can actually interact with the Google Ads API. (API calls will start failing)

Latest compatibility to a new API version was added in: https://github.com/googleads/google-ads-python/pull/672/files#diff-91c5b46dc84a94604a4e4d0caed9bf85590a2eddbb12d2e8dc80badf324a9dfbR9 (v17.0.0) and it added support v11 of the API.

v18.2.0 actually added support for v12 of the API. See here.

We're in the dark night now. Sunset has passed 😅 We're now getting error: "Version v11 is deprecated. Requests to this version will be blocked."

moiseenkov · 2023-03-31T10:30:10Z

Hi everyone,
Speaking about disabling mysql-connector-python, I found that the current MySqlHook implementation allows users to choose which library to use in an Airflow connection: mysql-connector-python or mysqlclient (default). What is the reason for it?

I'm wondering, because after removing the mysql-connector-python this feature will be no longer needed and can be removed as well. However, new libraries might appear in the future, and we will probably need it back then, so in this case it would be nice to save it for future use even if there will be only one option available. WDYT @potiuk, should we save it?

potiuk · 2023-03-31T14:14:51Z

I'm wondering, because after removing the mysql-connector-python this feature will be no longer needed and can be removed as well. However, new libraries might appear in the future, and we will probably need it back then, so in this case it would be nice to save it for future use even if there will be only one option available. WDYT @potiuk, should we save it?

I have not thought about it yet. I am waiting for the response of Oracle (If it comes) for a week - according to our new policy that's being "lazy consent now" and then I will take a closer look at that after. There is also an option to tunr mysql-connector-python into ACTUALLY optional feature (which I think is the best option) - so make it an extra (we already have a few of those). In this case we should leave it.

potiuk · 2023-03-31T14:17:20Z

@cgadam: It is likely we might have a proposal how to solve it soon - would you be willing to test it if I give you access to a beta/pre-release of google provider that you could test with it with an implemented worakround (with an intention of making it into next release?)

Changes: - update train model that is used for prediction - update version and runner for ApacheBeam in utils for MLEngine - update connection inside async hook

Changes: - fix tests/system/providers/google/cloud/dataprep/example_dataprep.py

- Secret Manager was missing updating to v2, now expects a request dict - Compute ssh had a bug when no cmd_timeout was passed - Cloud Build tests were improved/refactored in community, so deleting old ones - googleapiclient.errors.HttpError was incorrectly used in our tests, it it didn´t matter before but a change in the class makes HttpError() raise an error in initialization the way we were using it before - fix static checks ``` $ pytest tests/providers/google/cloud/ ... ===== 2763 passed, 71 skipped, 21 warnings in 193.46s (0:03:13) ===== ```

potiuk · 2023-05-17T20:58:01Z

👀 👀 👀 👀

kristopherkane · 2023-05-17T21:14:59Z

I wanted to say thanks for all this work and I've been tracking it from a distance. I'm looking forward to the updated Dataproc libs for further enhancements to the Dataproc serverless operator.

potiuk · 2023-05-17T21:48:35Z

I wanted to say thanks for all this work and I've been tracking it from a distance. I'm looking forward to the updated Dataproc libs for further enhancements to the Dataproc serverless operator.

Thanks in the name of all the people who worked on that (I was also just helping) - it's rare to get an unsolicited positive feedback and a thank you note. So rare :).

potiuk · 2023-05-17T22:34:05Z

Those are intermittent errors only (I need to make them more stable). Merging

potiuk · 2023-05-17T22:36:11Z

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉
🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉
🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

potiuk · 2023-05-20T11:50:52Z

CC: @ephraimbuddy -> I just realized we will need it - I marked this one also for 2.6.2. While the "code" changes aren't used in the release from 2.6.2, the "dependency" part (provider.yaml and generated/provider_dependencies.json) will be needed to properly build CI once we release the new google provider with all its deps

eladkal · 2023-05-20T13:03:47Z

CC: @ephraimbuddy -> I just realized we will need it - I marked this one also for 2.6.2. While the "code" changes aren't used in the release from 2.6.2, the "dependency" part (provider.yaml and generated/provider_dependencies.json) will be needed to properly build CI once we release the new google provider with all its deps

I'm hoping that we will release 2.6.2 as followup right after provider wave is released.

* Update SDK versions for Google provider * Adjust google ads operators to v12 Changes: - fix tests/system/providers/google/cloud/bigquery/example_bigquery_queries.py - fix tests/system/providers/google/cloud/bigquery/example_bigquery_queries_async.py * Fix GCS system tests * Fix CloudBuild unit test * Update BigTable operators to accomodate for new dependencies. * Fix Cloud Tasks System tests Tasks dag was quite flaky without the retry option in the run_task step, but it's consistently green with the option set. We also add a GCP_APP_ENGINE_LOCATION env variable since this depends on the used GCP Project App Engine's location * Add setup docstring to Tasks system tests. * Update Vision operators to accommodate new dependencies. Changes: - fix methods for CloudVisionHook - fix Vision Operators - fix tests/providers/google/cloud/hooks/test_vision.py - fix tests/providers/google/cloud/operators/test_vision.py - fix tests/system/providers/google/cloud/vision/example_vision_annotate_image.py - fix tests/system/providers/google/cloud/vision/example_vision_autogenerated.py - fix tests/system/providers/google/cloud/vision/example_vision_explicit.py * Update SpeechToText operators to accommodate new dependencies. Changes: - fix synthesize_speech method for CloudTextToSpeechHook - fix CloudSpeechToTextRecognizeSpeechOperator - fix tests/providers/google/cloud/operators/test_speech_to_text.py - fix tests/providers/google/cloud/hooks/test_text_to_speech.py - fix tests/providers/google/cloud/hooks/test_speech_to_text.py * Update Translate Speech operators to accommodate new dependencies. Changes: - fix synthesize_speech method for CloudTextToSpeechHook - fix CloudTranslateSpeechOperator - tests/providers/google/cloud/operators/test_translate_speech.py * Update VideoIntelligence operators to accommodate new dependencies. Changes: - fix annotate_video method for CloudVideoIntelligenceHook - fix VideoIntelligence Operators - fix tests/providers/google/cloud/hooks/test_video_intelligence.py - fix tests/providers/google/cloud/operators/test_video_intelligence.py * Update Compute Engine operators to accomodate for new dependencies. Changes: - added wait_for_operation_complete() method to check the execution flow - added new attribute cmd_timeout for ComputeEngineSSHHook * Fix Stackdriver system test This test has not worked because of slack channel and credentials not being setup. We now test the same operators by creating notification channels and policy alerts against pubsub topics, which don't need to exist before the test is ran, making the test self-contained. * Update Natural Language operators to accommodate new dependencies. Changes: - fix airflow/providers/google/cloud/operators/natural_language.py - fix airflow/providers/google/cloud/hooks/natural_language.py - fix tests/providers/google/cloud/hooks/test_natural_language.py - fix tests/providers/google/cloud/operators/test_natural_language.py - fix tests/system/providers/google/cloud/natural_language/example_natural_language.py * Update Composer system tests. Fix environment id to contain underscores. * Update AutoML operators to accommodate new dependencies. Changes: - add timeout parameter to all long-running operations for operators - fix tests/system/providers/google/cloud/automl/example_automl_dataset.py - fix tests/system/providers/google/cloud/automl/example_automl_model.py - fix tests/system/providers/google/cloud/automl/example_automl_nl_text_extraction.py - fix tests/system/providers/google/cloud/automl/example_automl_vision_classification.py * Fix Cloud SQL delete operator For some delete instance operations, the operation stops being available ~9 seconds after completion, so we need a shorter sleep time to make sure we don'tmiss the DONE status. * Update VertexAI operators to accommodate new dependencies. * Add SQL to Sheets Test instructions * Update Dataproc Metastore operators to accommodate new dependencies. * Update Dataproc operators to accommodate new dependencies. * Update Dataflow sys tests to new sdk * Update Dataproc on gke operators to accommodate new dependencies. * Update MLEngine operators to accomodate new dependencies. Changes: - update train model that is used for prediction - update version and runner for ApacheBeam in utils for MLEngine - update connection inside async hook * Update Dataprep operators to accommodate new dependencies. Changes: - fix tests/system/providers/google/cloud/dataprep/example_dataprep.py * Add Dataflow Go system test * Update providers.yaml for google * fixup! Update providers.yaml for google * Google SDK Fixes after rebase - Secret Manager was missing updating to v2, now expects a request dict - Compute ssh had a bug when no cmd_timeout was passed - Cloud Build tests were improved/refactored in community, so deleting old ones - googleapiclient.errors.HttpError was incorrectly used in our tests, it it didn´t matter before but a change in the class makes HttpError() raise an error in initialization the way we were using it before - fix static checks * Fix Google providers type errors --------- Co-authored-by: Lukasz Wyszomirski <wyszomirski@google.com> Co-authored-by: Maksim Moiseenkov <maksim_moiseenkov@epam.com> Co-authored-by: Eugene Kostieiev <kosteev@google.com> Co-authored-by: Augusto Hidalgo <augustoh@google.com> Co-authored-by: Beata Kossakowska <bkossakowska@google.com> Co-authored-by: Ulada Zakharava <uladaz@google.com> Co-authored-by: Jarek Potiuk <jarek@potiuk.com> (cherry picked from commit 28d1bf8)

boring-cyborg bot added area:providers provider:google Google (including GCP) related issues labels Mar 13, 2023

This was referenced Mar 15, 2023

Support for Python 3.11 for Google Provider (upgrading all dependencies) #27292

Closed

Remove protobuf limitation from eager upgrade #30182

Merged

Underlying C extension code doesn't work with upgrades required for Google Ads to keep functioning #30202

Closed

potiuk mentioned this pull request Mar 21, 2023

Separate out bigquery as standlone package from apache-airflow-providers-google #30224

Closed

2 tasks

potiuk mentioned this pull request Mar 27, 2023

Protobuf4 support yandex-cloud/python-sdk#71

Closed

potiuk mentioned this pull request Mar 29, 2023

Google Ads API Sunset for Version 11 #30353

Closed

2 tasks

potiuk mentioned this pull request Mar 29, 2023

Add description of the provider suspension process #30359

Merged

potiuk mentioned this pull request Mar 31, 2023

Bump dependencies of google provider after ads removal #30403

Closed

Beata Kossakowska and others added 11 commits May 17, 2023 20:57

Update Dataproc Metastore operators to accommodate new dependencies.

734b0b0

Update Dataproc operators to accommodate new dependencies.

d34fbf4

Update Dataflow sys tests to new sdk

5562618

Update Dataproc on gke operators to accommodate new dependencies.

1b7aa2e

Update MLEngine operators to accomodate new dependencies.

7420103

Changes: - update train model that is used for prediction - update version and runner for ApacheBeam in utils for MLEngine - update connection inside async hook

Update Dataprep operators to accommodate new dependencies.

d783329

Changes: - fix tests/system/providers/google/cloud/dataprep/example_dataprep.py

Add Dataflow Go system test

b160d3b

Update providers.yaml for google

0bc3cc6

fixup! Update providers.yaml for google

c60683c

Fix Google providers type errors

31d7cd6

ahidalgob force-pushed the sdk-update-google branch from e6bb498 to 31d7cd6 Compare May 17, 2023 21:01

potiuk approved these changes May 17, 2023

View reviewed changes

potiuk merged commit 28d1bf8 into apache:main May 17, 2023

eladkal mentioned this pull request May 19, 2023

Status of testing Providers that were prepared on May 19, 2023 #31322

Closed

80 tasks

potiuk added this to the Airflow 2.6.2 milestone May 20, 2023

potiuk mentioned this pull request Jun 9, 2023

Sync v2-6-stable with v2-6-test to release 2.6.2 #31796

Merged

eladkal added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Jun 9, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update SDKs for google provider package #30067

Update SDKs for google provider package #30067

lwyszomi commented Mar 13, 2023 •

edited by potiuk

Loading

felicienveldema commented Mar 21, 2023 •

edited

Loading

lwyszomi commented Mar 21, 2023

r-richmond commented Mar 27, 2023

potiuk commented Mar 27, 2023

potiuk commented Mar 27, 2023 •

edited

Loading

potiuk commented Mar 27, 2023

potiuk commented Mar 27, 2023

potiuk commented Mar 27, 2023

potiuk commented Mar 27, 2023

potiuk commented Mar 27, 2023

r-richmond commented Mar 27, 2023

potiuk commented Mar 27, 2023

cgadam commented Mar 29, 2023

cgadam commented Mar 30, 2023 •

edited

Loading

moiseenkov commented Mar 31, 2023 •

edited

Loading

potiuk commented Mar 31, 2023 •

edited

Loading

potiuk commented Mar 31, 2023

potiuk commented May 17, 2023

kristopherkane commented May 17, 2023

potiuk commented May 17, 2023

potiuk commented May 17, 2023

potiuk commented May 17, 2023

potiuk commented May 20, 2023 •

edited

Loading

eladkal commented May 20, 2023

Update SDKs for google provider package #30067

Update SDKs for google provider package #30067

Conversation

lwyszomi commented Mar 13, 2023 • edited by potiuk Loading

felicienveldema commented Mar 21, 2023 • edited Loading

lwyszomi commented Mar 21, 2023

r-richmond commented Mar 27, 2023

potiuk commented Mar 27, 2023

potiuk commented Mar 27, 2023 • edited Loading

potiuk commented Mar 27, 2023

potiuk commented Mar 27, 2023

potiuk commented Mar 27, 2023

potiuk commented Mar 27, 2023

potiuk commented Mar 27, 2023

r-richmond commented Mar 27, 2023

potiuk commented Mar 27, 2023

cgadam commented Mar 29, 2023

cgadam commented Mar 30, 2023 • edited Loading

moiseenkov commented Mar 31, 2023 • edited Loading

potiuk commented Mar 31, 2023 • edited Loading

potiuk commented Mar 31, 2023

potiuk commented May 17, 2023

kristopherkane commented May 17, 2023

potiuk commented May 17, 2023

potiuk commented May 17, 2023

potiuk commented May 17, 2023

potiuk commented May 20, 2023 • edited Loading

eladkal commented May 20, 2023

lwyszomi commented Mar 13, 2023 •

edited by potiuk

Loading

felicienveldema commented Mar 21, 2023 •

edited

Loading

potiuk commented Mar 27, 2023 •

edited

Loading

cgadam commented Mar 30, 2023 •

edited

Loading

moiseenkov commented Mar 31, 2023 •

edited

Loading

potiuk commented Mar 31, 2023 •

edited

Loading

potiuk commented May 20, 2023 •

edited

Loading