Some more cleanup in abstract classes in preparation of splitting #24164

akashkulk · 2023-03-16T19:41:19Z

No description provided.

akashkulk · 2023-03-16T19:41:56Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4441001592
✅ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4441001592
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:509: The previous and actual discovered catalogs are identical.
=================== 62 passed, 5 skipped in 84.01s (0:01:24) ===================

akashkulk · 2023-03-16T19:42:03Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/4441002019
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/4441002019
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
================= 19 passed, 6 skipped, 27 warnings in 17.74s ==================

akashkulk · 2023-03-16T19:42:12Z

/test connector=connectors/source-bigquery

🕑 connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/4441003394
✅ connectors/source-bigquery https://github.com/airbytehq/airbyte/actions/runs/4441003394
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:509: The previous and actual discovered catalogs are identical.
======================== 32 passed, 2 skipped in 44.11s ========================

akashkulk · 2023-03-16T19:42:19Z

/test connector=connectors/source-mssql

🕑 connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/4441004136
✅ connectors/source-mssql https://github.com/airbytehq/airbyte/actions/runs/4441004136
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestConnection.test_check: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestDiscovery.test_discover: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestBasicRead.test_read: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
================= 19 passed, 6 skipped, 27 warnings in 17.76s ==================

akashkulk · 2023-03-16T19:42:38Z

/test connector=connectors/source-mongodb-v2

🕑 connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/4441006580
❌ connectors/source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/4441006580
🐛 https://gradle.com/s/dnhwnej6wfrvg

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestConnection::test_check[inputs0] - AssertionError: as...
FAILED test_core.py::TestDiscovery::test_discover[inputs0] - docker.errors.Co...
ERROR test_core.py::TestDiscovery::test_defined_cursors_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_refs_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-allOf]
ERROR test_core.py::TestDiscovery::test_defined_keyword_exist_in_schema[inputs0-not]
ERROR test_core.py::TestDiscovery::test_primary_keys_exist_in_schema[inputs0]
ERROR test_core.py::TestDiscovery::test_streams_has_sync_modes[inputs0] - doc...
ERROR test_core.py::TestDiscovery::test_additional_properties_is_true[inputs0]
ERROR test_core.py::TestDiscovery::test_backward_compatibility[inputs0] - doc...
ERROR test_core.py::TestBasicRead::test_read[inputs0] - docker.errors.Contain...
ERROR test_full_refresh.py::TestFullRefresh::test_sequential_reads[inputs0]
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
============= 2 failed, 21 passed, 1 skipped, 10 errors in 36.01s ==============

github-actions · 2023-03-16T19:44:18Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (25)

Connector	Version	Changelog	Publish
`source-alloydb`	`2.0.2`	✅	✅
`source-alloydb-strict-encrypt`	`2.0.2`	🔵 (ignored)	🔵 (ignored)
`source-bigquery`	`0.2.3`	✅	✅
`source-clickhouse`	`0.1.16`	✅	✅
`source-clickhouse-strict-encrypt`	`0.1.16`	🔵 (ignored)	🔵 (ignored)
`source-cockroachdb`	`0.1.21`	✅	✅
`source-cockroachdb-strict-encrypt`	`0.1.21`	🔵 (ignored)	🔵 (ignored)
`source-db2`	`0.1.18`	✅	✅
`source-db2-strict-encrypt`	`0.1.18`	🔵 (ignored)	🔵 (ignored)
`source-dynamodb`	`0.1.2`	✅	✅
`source-jdbc`	`0.3.5`	🔵 (ignored)	🔵 (ignored)
`source-mongodb-strict-encrypt`	`0.1.19`	🔵 (ignored)	🔵 (ignored)
`source-mongodb-v2`	`0.1.19`	✅	✅
`source-mssql`	`1.0.3`	✅	✅
`source-mssql-strict-encrypt`	`1.0.3`	🔵 (ignored)	🔵 (ignored)
`source-mysql`	`2.0.3`	✅	✅
`source-mysql-strict-encrypt`	`2.0.3`	🔵 (ignored)	🔵 (ignored)
`source-oracle`	`0.3.23`	✅	✅
`source-oracle-strict-encrypt`	`0.3.23`	🔵 (ignored)	🔵 (ignored)
`source-postgres`	`2.0.3`	✅	✅
`source-postgres-strict-encrypt`	`2.0.3`	🔵 (ignored)	🔵 (ignored)
`source-redshift`	`0.3.16`	✅	✅
`source-scaffold-java-jdbc`	`0.1.0`	🔵 (ignored)	🔵 (ignored)
`source-snowflake`	`0.1.31`	✅	✅
`source-tidb`	`0.2.3`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

✅ Destinations (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

akashkulk · 2023-03-16T20:44:18Z

/test connector=connectors/source-snowflake

🕑 connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/4441477088
✅ connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/4441477088
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:509: The previous and actual discovered catalogs are identical.
=================== 31 passed, 3 skipped in 75.44s (0:01:15) ===================

nguyenaiden · 2023-03-16T21:32:28Z

...ional-db/src/main/java/io/airbyte/integrations/source/relationaldb/DbSourceDiscoverUtil.java


+/**
+ * Contains utilities and helper classes for discovering schemas in database sources.
+ */
 public class DbSourceDiscoverUtil {


I'm good with leaving testing of this class to blackbox at the connector level for now. But we should unit test this at some point.
Also, making these methods static is appropriate here, but I always have run into an issue trying to test static methods.

nguyenaiden

+1 with a small comment.

nguyenaiden · 2023-03-16T21:33:47Z

...ional-db/src/main/java/io/airbyte/integrations/source/relationaldb/DbSourceDiscoverUtil.java

+      final JsonNode catalogSchema = stream.getJsonSchema();
+      if (!catalogSchema.equals(currentJsonSchema)) {
+        LOGGER.warn(
+            "Source schema changed for table  {}! Actual schema: {}. Catalog schema:  {}",


nit: extra space between table {} and Catalog schema: {}

…rbytehq#24164)

akashkulk · 2023-03-20T15:56:59Z

/test connector=connectors/source-postgres

🕑 connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4470237327
❌ connectors/source-postgres https://github.com/airbytehq/airbyte/actions/runs/4470237327
🐛 https://gradle.com/s/vxwmugi5mmxf2

Build Failed

Test summary info:

=========================== short test summary info ============================
FAILED test_core.py::TestBasicRead::test_read[inputs0] - AssertionError: All ...
FAILED test_core.py::TestBasicRead::test_read[inputs1] - AssertionError: All ...
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [2] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:509: The previous and actual discovered catalogs are identical.
============== 2 failed, 60 passed, 5 skipped in 78.74s (0:01:18) ==============

…4164)

Some more cleanup in abstract classes in preparation of splitting

aef6eaf

octavia-squidington-iii added area/connectors Connector related issues connectors/source/relational-db labels Mar 16, 2023

Merge branch 'master' into more_refac

9dd8491

akashkulk marked this pull request as ready for review March 16, 2023 19:48

akashkulk requested a review from rodireich March 16, 2023 20:29

Merge branch 'master' into more_refac

5559547

akashkulk requested a review from nguyenaiden March 16, 2023 21:10

rodireich approved these changes Mar 16, 2023

View reviewed changes

nguyenaiden reviewed Mar 16, 2023

View reviewed changes

nguyenaiden approved these changes Mar 16, 2023

View reviewed changes

nguyenaiden reviewed Mar 16, 2023

View reviewed changes

akashkulk merged commit c772bc2 into master Mar 16, 2023

akashkulk deleted the more_refac branch March 16, 2023 21:56

adriennevermorel pushed a commit to adriennevermorel/airbyte that referenced this pull request Mar 17, 2023

Some more cleanup in abstract classes in preparation of splitting (ai…

fa92e32

…rbytehq#24164)

erohmensing pushed a commit that referenced this pull request Mar 22, 2023

Some more cleanup in abstract classes in preparation of splitting (#2…

965a95b

…4164)

erohmensing pushed a commit that referenced this pull request Mar 22, 2023

Some more cleanup in abstract classes in preparation of splitting (#2…

c835266

…4164)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Some more cleanup in abstract classes in preparation of splitting #24164

Some more cleanup in abstract classes in preparation of splitting #24164

akashkulk commented Mar 16, 2023

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Mar 16, 2023 •

edited

Loading

✅ Sources (25)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

nguyenaiden Mar 16, 2023

nguyenaiden left a comment

nguyenaiden Mar 16, 2023

akashkulk commented Mar 20, 2023 •

edited by github-actions bot

Loading

Some more cleanup in abstract classes in preparation of splitting #24164

Some more cleanup in abstract classes in preparation of splitting #24164

Conversation

akashkulk commented Mar 16, 2023

akashkulk commented Mar 16, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Mar 16, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Mar 16, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Mar 16, 2023 • edited by github-actions bot Loading

Build Passed

akashkulk commented Mar 16, 2023 • edited by github-actions bot Loading

Build Failed

github-actions bot commented Mar 16, 2023 • edited Loading

Affected Connector Report

✅ Sources (25)

✅ Destinations (0)

✅ Other Modules (0)

Actionable Items

akashkulk commented Mar 16, 2023 • edited by github-actions bot Loading

Build Passed

nguyenaiden Mar 16, 2023

Choose a reason for hiding this comment

nguyenaiden left a comment

Choose a reason for hiding this comment

nguyenaiden Mar 16, 2023

Choose a reason for hiding this comment

akashkulk commented Mar 20, 2023 • edited by github-actions bot Loading

Build Failed

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

github-actions bot commented Mar 16, 2023 •

edited

Loading

akashkulk commented Mar 16, 2023 •

edited by github-actions bot

Loading

akashkulk commented Mar 20, 2023 •

edited by github-actions bot

Loading