🎉 Source Google Directory: support oauth #7409

vitaliizazmic · 2021-10-27T13:16:21Z

What

Supporting oauth

How

Source Google Directory already supports authentication via Service account. So oauth webflow support was added. And specification was updated.

Closes #6265

vitaliizazmic · 2021-10-27T13:18:29Z

/test connector=source-google-directory

🕑 source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1390291596
❌ source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1390291596
🐛

vitaliizazmic · 2021-10-27T13:43:57Z

/test connector=source-google-directory

🕑 source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1390394479
❌ source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1390394479
🐛 https://gradle.com/s/e6iitkb22d532

avida · 2021-10-27T14:08:25Z

airbyte-integrations/connectors/source-google-directory/source_google_directory/spec.json

@@ -4,18 +4,97 @@
    "$schema": "http://json-schema.org/draft-07/schema#",
    "title": "Google Directory Spec",
    "type": "object",
-    "required": ["credentials_json", "email"],
+    "required": ["credentials"],


How would it work w/ old config format with email / credentials_json fields in the root? You need to remove required field and set additionalProperties to true if you want make it backward compatible.

You can also use OneOf in the required field

avida · 2021-10-27T14:09:28Z

airbyte-integrations/connectors/source-google-directory/source_google_directory/spec.json

+                "description": "The token for obtaining new access token",
+                "airbyte_secret": true
+              },
+              "scopes": {


you dont need scopes, just client_id, client_secret and refresh_token

avida · 2021-10-27T14:09:46Z

airbyte-integrations/connectors/source-google-directory/source_google_directory/spec.json

+    "oauth2Specification": {
+      "rootObject": ["credentials", 0],
+      "oauthFlowInitParameters": [["client_id"], ["client_secret"]],
+      "oauthFlowOutputParameters": [["access_token"], ["refresh_token"]]


Please remove access_token

vitaliizazmic · 2021-10-28T15:01:11Z

/test connector=source-google-directory

🕑 source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1395184400
❌ source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1395184400
🐛 https://gradle.com/s/di46dm4yi72ve

sherifnada

I didn't scrutinize the implementation, mostly looked at docs/UX. Do we need unit tests for auth?

sherifnada · 2021-10-28T19:46:50Z

docs/integrations/sources/google-directory.md

@@ -37,7 +37,9 @@ This connector attempts to back off gracefully when it hits Directory API's rate

 ## Getting started

-### Requirements
+Google APIs use the OAuth 2.0 protocol for authentication and authorization. The Source supports [Web server application](https://developers.google.com/identity/protocols/oauth2#webserver) and [Service accounts](https://developers.google.com/identity/protocols/oauth2#serviceaccount) scenarios.


Suggested change

Google APIs use the OAuth 2.0 protocol for authentication and authorization. The Source supports [Web server application](https://developers.google.com/identity/protocols/oauth2#webserver) and [Service accounts](https://developers.google.com/identity/protocols/oauth2#serviceaccount) scenarios.

Google APIs use the OAuth 2.0 protocol for authentication and authorization. This connector supports [Web server application](https://developers.google.com/identity/protocols/oauth2#webserver) and [Service accounts](https://developers.google.com/identity/protocols/oauth2#serviceaccount) scenarios.

Can you also add a section explaining the difference in setup between OSS and Cloud? Similar to this one https://docs.airbyte.io/integrations/sources/google-analytics-v4#getting-started-airbyte-cloud

sherifnada · 2021-10-28T19:47:58Z

airbyte-integrations/connectors/source-google-directory/source_google_directory/spec.json

+        "type": "object",
+        "oneOf": [
+          {
+            "title": "Web server application",


making this suggestion because the non-technical personas explained in the UX handbook e.g: data analyst won't know what a "Web Server Application" means

Suggested change

"title": "Web server application",

"title": "Sign in via Google (Oauth)",

sherifnada · 2021-10-28T19:48:19Z

airbyte-integrations/connectors/source-google-directory/source_google_directory/spec.json

+            }
+          },
+          {
+            "title": "Service accounts",


Suggested change

"title": "Service accounts",

"title": "Service account Key",

keu · 2021-10-29T14:11:57Z

tools/bin/ci_credentials.sh

@@ -82,6 +82,7 @@ write_standard_creds source-google-analytics-v4 "$GOOGLE_ANALYTICS_V4_TEST_CREDS
 write_standard_creds source-google-analytics-v4 "$GOOGLE_ANALYTICS_V4_TEST_CREDS_SRV_ACC" "service_config.json"
 write_standard_creds source-google-analytics-v4 "$GOOGLE_ANALYTICS_V4_TEST_CREDS_OLD" "old_config.json"
 write_standard_creds source-google-directory "$GOOGLE_DIRECTORY_TEST_CREDS"
+write_standard_creds source-google-directory "$GOOGLE_DIRECTORY_TEST_CREDS_OAUTH" "config_oauth.json"


does SAT use this config?
is there any tests for it?

@keu SAT isn't enabled in this source. I've created PR #7464. I will add required test there.

# Conflicts: # airbyte-integrations/connectors/source-google-directory/source_google_directory/api.py

vitaliizazmic · 2021-11-03T15:00:00Z

/test connector=source-google-directory

🕑 source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1417313485
✅ source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1417313485
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        75      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              200     94    53%
	 source_acceptance_test/tests/test_full_refresh.py       33     22    33%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  891    435    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                  Stmts   Miss  Cover
	 ---------------------------------------------------------
	 source_google_directory/__init__.py       2      2     0%
	 source_google_directory/api.py           90     90     0%
	 Name                                  Stmts   Miss  Cover/actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-google-directory/.venv/lib/python3.8/site-packages/coverage/control.py:761: CoverageWarning: No data was collected. (no-data-collected)
	 source_google_directory/client.py        26     26     0%
	   self._warn("No data was collected.", slug="no-data-collected")
	 source_google_directory/source.py         4      4     0%
	 source_google_directory/utils.py          3      3     0%
	 ---------------------------------------------------------
	 TOTAL                                   125    125     0%

vitaliizazmic · 2021-11-05T12:12:38Z

/publish connector=connectors/source-google-directory

🕑 connectors/source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1425621807
❌ connectors/source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1425621807

vitaliizazmic · 2021-11-05T12:25:12Z

/publish connector=connectors/source-google-directory

🕑 connectors/source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1425662807
❌ connectors/source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1425662807

…date publish)

vitaliizazmic · 2021-11-05T16:27:40Z

/publish connector=connectors/source-google-directory

🕑 connectors/source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1426531755
✅ connectors/source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1426531755

vitaliizazmic added 2 commits October 27, 2021 15:30

Source Google Directory #6265 - add oauth support

7e93198

Source Google Directory #6265 - update credentials

237a4b3

vitaliizazmic self-assigned this Oct 27, 2021

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Oct 27, 2021

jrhizor temporarily deployed to more-secrets October 27, 2021 13:20 Inactive

jrhizor temporarily deployed to more-secrets October 27, 2021 13:46 Inactive

avida reviewed Oct 27, 2021

View reviewed changes

Merge branch 'master' into vitalii/6265_google_direcotry_support_oauth

475636a

vitaliizazmic temporarily deployed to more-secrets October 28, 2021 14:57 Inactive

jrhizor temporarily deployed to more-secrets October 28, 2021 15:03 Inactive

Source Google Directory #6265 - fixing according to PR

98ba803

vitaliizazmic requested review from avida, eliziario, gaart and keu October 28, 2021 15:40

vitaliizazmic temporarily deployed to more-secrets October 28, 2021 15:42 Inactive

avida approved these changes Oct 28, 2021

View reviewed changes

vitaliizazmic temporarily deployed to more-secrets October 28, 2021 17:39 Inactive

vitaliizazmic requested a review from sherifnada October 28, 2021 19:44

sherifnada approved these changes Oct 28, 2021

View reviewed changes

Source Google directory #6265 - update docs

dec5094

vitaliizazmic temporarily deployed to more-secrets October 29, 2021 14:03 Inactive

keu changed the title ~~🎉 Source Google Direcotry: support oauth~~ 🎉 Source Google Directory: support oauth Oct 29, 2021

keu reviewed Oct 29, 2021

View reviewed changes

Source Google directory #5190 - update doc

e7a6580

vitaliizazmic temporarily deployed to more-secrets November 1, 2021 16:13 Inactive

vitaliizazmic added 2 commits November 2, 2021 22:21

Merge branch 'master' into vitalii/6265_google_direcotry_support_oauth

d418813

# Conflicts: # airbyte-integrations/connectors/source-google-directory/source_google_directory/api.py

Source Google Directory #6265 - resolve merge conflict

3954b15

vitaliizazmic temporarily deployed to more-secrets November 2, 2021 20:26 Inactive

Source Google Directory #6265 - SAT for oauth

2ed8369

vitaliizazmic temporarily deployed to more-secrets November 3, 2021 15:01 Inactive

jrhizor temporarily deployed to more-secrets November 3, 2021 15:02 Inactive

vitaliizazmic added 2 commits November 5, 2021 13:45

Merge branch 'master' into vitalii/6265_google_direcotry_support_oauth

021a154

Source Google Directory #6265 - bump version and update changelog

96ef16d

vitaliizazmic temporarily deployed to more-secrets November 5, 2021 12:13 Inactive

jrhizor temporarily deployed to more-secrets November 5, 2021 12:14 Inactive

jrhizor temporarily deployed to more-secrets November 5, 2021 12:27 Inactive

vitaliizazmic temporarily deployed to more-secrets November 5, 2021 12:49 Inactive

Source Google Directory #6265 - bump version and update changelog (up…

96130d6

…date publish)

vitaliizazmic temporarily deployed to more-secrets November 5, 2021 16:29 Inactive

jrhizor temporarily deployed to more-secrets November 5, 2021 16:29 Inactive

vitaliizazmic temporarily deployed to more-secrets November 5, 2021 16:41 Inactive

vitaliizazmic merged commit 1f295f2 into master Nov 5, 2021

vitaliizazmic deleted the vitalii/6265_google_direcotry_support_oauth branch November 5, 2021 19:40

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source Google Directory: support oauth #7409

🎉 Source Google Directory: support oauth #7409

vitaliizazmic commented Oct 27, 2021

vitaliizazmic commented Oct 27, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Oct 27, 2021 •

edited by github-actions bot

Loading

avida Oct 27, 2021

eliziario Oct 28, 2021

avida Oct 27, 2021

avida Oct 27, 2021

vitaliizazmic commented Oct 28, 2021 •

edited by github-actions bot

Loading

sherifnada left a comment

sherifnada Oct 28, 2021

sherifnada Oct 28, 2021

sherifnada Oct 28, 2021

keu Oct 29, 2021

vitaliizazmic Nov 1, 2021

vitaliizazmic commented Nov 3, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Nov 5, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Nov 5, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Nov 5, 2021 •

edited by github-actions bot

Loading

	Google APIs use the OAuth 2.0 protocol for authentication and authorization. The Source supports [Web server application](https://developers.google.com/identity/protocols/oauth2#webserver) and [Service accounts](https://developers.google.com/identity/protocols/oauth2#serviceaccount) scenarios.
	Google APIs use the OAuth 2.0 protocol for authentication and authorization. This connector supports [Web server application](https://developers.google.com/identity/protocols/oauth2#webserver) and [Service accounts](https://developers.google.com/identity/protocols/oauth2#serviceaccount) scenarios.

	"title": "Web server application",
	"title": "Sign in via Google (Oauth)",

🎉 Source Google Directory: support oauth #7409

🎉 Source Google Directory: support oauth #7409

Conversation

vitaliizazmic commented Oct 27, 2021

What

How

vitaliizazmic commented Oct 27, 2021 • edited by github-actions bot Loading

vitaliizazmic commented Oct 27, 2021 • edited by github-actions bot Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vitaliizazmic commented Oct 28, 2021 • edited by github-actions bot Loading

sherifnada left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vitaliizazmic commented Nov 3, 2021 • edited by github-actions bot Loading

vitaliizazmic commented Nov 5, 2021 • edited by github-actions bot Loading

vitaliizazmic commented Nov 5, 2021 • edited by github-actions bot Loading

vitaliizazmic commented Nov 5, 2021 • edited by github-actions bot Loading

vitaliizazmic commented Oct 27, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Oct 27, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Oct 28, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Nov 3, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Nov 5, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Nov 5, 2021 •

edited by github-actions bot

Loading

vitaliizazmic commented Nov 5, 2021 •

edited by github-actions bot

Loading