Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

🎉 Source Google Directory: migrate to the CDK #7464

Merged
merged 8 commits into from
Nov 2, 2021
Merged

🎉 Source Google Directory: migrate to the CDK #7464

merged 8 commits into from
Nov 2, 2021

Conversation

vitaliizazmic
Copy link
Contributor

@vitaliizazmic vitaliizazmic commented Oct 28, 2021 •
edited
Loading

What

Refactoring Source Drift to use the CDK

Closes #7415

How

This task is part of issue #6164. The Source significantly relies on dead CDK code. So import BaseClient and BaseSource from base_python were replaced by import from deprecated package of the CDK. Gradle file, setup.py, Dockerfile, requirements.txt were updated. main_dev.py was renamed to main.py, updated, and added to .dockerignore. SAT was enabled for Source. Also schema was fixed.

Community member or Airbyter

  • Grant edit access to maintainers (instructions)
  • Secrets in the connector's spec are annotated with airbyte_secret
  • Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
  • Code reviews completed
  • Documentation updated
    • Connector's README.md
    • Connector's bootstrap.md. See description and examples
    • Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
  • PR name follows PR naming conventions
  • Connector version bumped like described here

@github-actions github-actions bot added the area/connectors Connector related issues label Oct 28, 2021
@vitaliizazmic
Copy link
Contributor Author

vitaliizazmic commented Oct 28, 2021 •
edited by github-actions bot
Loading

/test connector=source-google-directory

🕑 source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1396203491
❌ source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1396203491
🐛 https://gradle.com/s/sam3vawyh74p6

@jrhizor jrhizor temporarily deployed to more-secrets October 28, 2021 19:54 Inactive
@vitaliizazmic vitaliizazmic self-assigned this Oct 29, 2021
@vitaliizazmic vitaliizazmic mentioned this pull request Nov 1, 2021
Merged
10 tasks
@vitaliizazmic
Copy link
Contributor Author

vitaliizazmic commented Nov 1, 2021 •
edited by github-actions bot
Loading

/test connector=source-google-directory

🕑 source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1408717291
✅ source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1408717291
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              200     94    53%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  860    419    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                  Stmts   Miss  Cover
	 /actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-google-directory/.venv/lib/python3.8/site-packages/coverage/control.py:761: CoverageWarning: No data was collected. (no-data-collected)
	 ---------------------------------------------------------
	 source_google_directory/__init__.py       2      2     0%
	 source_google_directory/api.py           74     74     0%
	 source_google_directory/client.py        24     24     0%
	 source_google_directory/source.py         4      4     0%
	   self._warn("No data was collected.", slug="no-data-collected")
	 source_google_directory/utils.py          3      3     0%
	 ---------------------------------------------------------
	 TOTAL                                   107    107     0%

@jrhizor jrhizor temporarily deployed to more-secrets November 1, 2021 16:20 Inactive
@vitaliizazmic
Copy link
Contributor Author

vitaliizazmic commented Nov 1, 2021 •
edited by github-actions bot
Loading

/test connector=source-google-directory

🕑 source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1408976945
✅ source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1408976945
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        74      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              200     94    53%
	 source_acceptance_test/tests/test_full_refresh.py       18     11    39%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 47     20    57%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  860    419    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 /actions-runner/_work/airbyte/airbyte/airbyte-integrations/connectors/source-google-directory/.venv/lib/python3.8/site-packages/coverage/control.py:761: CoverageWarning: No data was collected. (no-data-collected)
	 Name                                  Stmts   Miss  Cover
	 ---------------------------------------------------------
	 source_google_directory/__init__.py       2      2     0%
	 source_google_directory/api.py           75     75     0%
	 source_google_directory/client.py        24     24     0%
	 source_google_directory/source.py         4      4     0%
	 source_google_directory/utils.py          3      3     0%
	 ---------------------------------------------------------
	 TOTAL                                   108    108     0%

@vitaliizazmic vitaliizazmic temporarily deployed to more-secrets November 1, 2021 17:34 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets November 1, 2021 17:35 Inactive
@vitaliizazmic vitaliizazmic temporarily deployed to more-secrets November 1, 2021 18:20 Inactive
yevhenii-ldv
yevhenii-ldv reviewed Nov 2, 2021
View reviewed changes
airbyte-integrations/connectors/source-google-directory/build.gradle Outdated
configuredCatalogPath = "sample_files/configured_catalog.json"
}


dependencies {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can remove dependency of "source-acceptance-test" here

yevhenii-ldv
yevhenii-ldv reviewed Nov 2, 2021
View reviewed changes
airbyte-integrations/connectors/source-google-directory/integration_tests/sample_config.json
@@ -0,0 +1,3 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why absolutely different files invalid_config and sample_config?

yevhenii-ldv
yevhenii-ldv approved these changes Nov 2, 2021
View reviewed changes
Copy link
Contributor

@yevhenii-ldv yevhenii-ldv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM in general

@vitaliizazmic vitaliizazmic temporarily deployed to more-secrets November 2, 2021 17:09 Inactive
@vitaliizazmic
Copy link
Contributor Author

vitaliizazmic commented Nov 2, 2021 •
edited by github-actions bot
Loading

/test connector=source-google-directory

🕑 source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1413359033
❌ source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1413359033
🐛 https://gradle.com/s/g4lk4r5g42a7i

@jrhizor jrhizor temporarily deployed to more-secrets November 2, 2021 17:14 Inactive
@vitaliizazmic
Copy link
Contributor Author

vitaliizazmic commented Nov 2, 2021 •
edited by github-actions bot
Loading

/test connector=source-google-directory

🕑 source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1413425704
❌ source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1413425704
🐛 https://gradle.com/s/zvw7xh7byjyj4

@jrhizor jrhizor temporarily deployed to more-secrets November 2, 2021 17:32 Inactive
@vitaliizazmic
Source Google Directory #7415 - added etag and lastLoginTime to ignor…
40d6fef 
…ed fields for full refresh acceptance test
@vitaliizazmic
Copy link
Contributor Author

vitaliizazmic commented Nov 2, 2021 •
edited by github-actions bot
Loading

/test connector=source-google-directory

🕑 source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1413687736
❌ source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1413687736
🐛 https://gradle.com/s/lmedcujpyck5w

@vitaliizazmic vitaliizazmic temporarily deployed to more-secrets November 2, 2021 18:47 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets November 2, 2021 18:48 Inactive
@vitaliizazmic
Copy link
Contributor Author

vitaliizazmic commented Nov 2, 2021 •
edited by github-actions bot
Loading

/test connector=source-google-directory

🕑 source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1413754589
❌ source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1413754589
🐛 https://gradle.com/s/esmcjd474mdie

@vitaliizazmic vitaliizazmic temporarily deployed to more-secrets November 2, 2021 19:06 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets November 2, 2021 19:08 Inactive
@vitaliizazmic
Copy link
Contributor Author

vitaliizazmic commented Nov 2, 2021 •
edited by github-actions bot
Loading

/test connector=source-google-directory

🕑 source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1413823518
✅ source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1413823518
Python tests coverage:

	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	 Name                                                 Stmts   Miss  Cover
	 ------------------------------------------------------------------------
	 source_acceptance_test/__init__.py                       2      0   100%
	 source_acceptance_test/base.py                          10      4    60%
	 source_acceptance_test/config.py                        75      8    89%
	 source_acceptance_test/conftest.py                     108    108     0%
	 source_acceptance_test/plugin.py                        47     47     0%
	 source_acceptance_test/tests/__init__.py                 4      0   100%
	 source_acceptance_test/tests/test_core.py              200     94    53%
	 source_acceptance_test/tests/test_full_refresh.py       33     22    33%
	 source_acceptance_test/tests/test_incremental.py        69     38    45%
	 source_acceptance_test/utils/__init__.py                 6      0   100%
	 source_acceptance_test/utils/asserts.py                 37      2    95%
	 source_acceptance_test/utils/common.py                  41     24    41%
	 source_acceptance_test/utils/compare.py                 62     25    60%
	 source_acceptance_test/utils/connector_runner.py        82     49    40%
	 source_acceptance_test/utils/json_schema_helper.py     115     14    88%
	 ------------------------------------------------------------------------
	 TOTAL                                                  891    435    51%
	 ---------- coverage: platform linux, python 3.8.10-final-0 -----------
	   self._warn("No data was collected.", slug="no-data-collected")
	 Name                                  Stmts   Miss  Cover
	 ---------------------------------------------------------
	 source_google_directory/__init__.py       2      2     0%
	 source_google_directory/api.py           75     75     0%
	 source_google_directory/client.py        24     24     0%
	 source_google_directory/source.py         4      4     0%
	 source_google_directory/utils.py          3      3     0%
	 ---------------------------------------------------------
	 TOTAL                                   108    108     0%

@vitaliizazmic vitaliizazmic temporarily deployed to more-secrets November 2, 2021 19:26 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets November 2, 2021 19:27 Inactive
@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label Nov 2, 2021
@vitaliizazmic
Copy link
Contributor Author

vitaliizazmic commented Nov 2, 2021 •
edited by github-actions bot
Loading

/publish connector=connectors/source-google-directory

🕑 connectors/source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1413878102
✅ connectors/source-google-directory https://github.com/airbytehq/airbyte/actions/runs/1413878102

@vitaliizazmic vitaliizazmic temporarily deployed to more-secrets November 2, 2021 19:44 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets November 2, 2021 19:45 Inactive
@jrhizor jrhizor temporarily deployed to more-secrets November 2, 2021 19:45 Inactive
@vitaliizazmic vitaliizazmic temporarily deployed to more-secrets November 2, 2021 19:54 Inactive
@vitaliizazmic vitaliizazmic merged commit d37e00e into master Nov 2, 2021
@vitaliizazmic vitaliizazmic deleted the vitalii/7415_google_directory_refactor_to_use_the_cdk branch November 2, 2021 20:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yevhenii-ldv yevhenii-ldv yevhenii-ldv approved these changes

@keu keu Awaiting requested review from keu

@eliziario eliziario Awaiting requested review from eliziario

Assignees

@vitaliizazmic vitaliizazmic

Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/source/drift connectors/sources-api
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Source Google Directory: refactor to use the CDK
5 participants
@vitaliizazmic @yevhenii-ldv @jrhizor @VasylLazebnyk @igrankova