Normalization: Revert to protocol v0 #22283

edgao · 2023-02-02T00:54:04Z

What

Downgrades main branch to revert protocol v0 normalization changes

Note that the normalization tests failed only on redshift. We've hit the table count limit in there (again >.>) so redshift tests are failing across the board.

This isn't just a naive revert - see https://github.com/airbytehq/airbyte/pull/22283/files/b7af3a858412dfd822fada90c203814b6327a7cf..4d46932c899299c8b1b82b38ce7c651616419e5d for the interesting diff.

…ary data as string (#22165)" This reverts commit 8276d03.

This reverts commit dbe56d6.

This reverts commit c1d7736.

github-actions · 2023-02-02T00:56:19Z

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

Run integration tests
Bump connector or module version
Add changelog
Publish the new version

✅ Sources (0)

Connector	Version	Changelog	Publish

See "Actionable Items" below for how to resolve warnings and errors.

❌ Destinations (16)

Connector	Version	Changelog	Publish
`destination-bigquery`	`1.2.13`	✅	✅
`destination-bigquery-denormalized`	`1.2.12`	✅	❌ (diff seed version)
`destination-clickhouse`	`0.2.2`	❌ (changelog missing)	✅
`destination-clickhouse-strict-encrypt`	`0.2.2`	🔵 (ignored)	🔵 (ignored)
`destination-jdbc`	`0.3.14`	🔵 (ignored)	🔵 (ignored)
`destination-mssql`	`0.1.22`	✅	✅
`destination-mssql-strict-encrypt`	`0.1.22`	🔵 (ignored)	🔵 (ignored)
`destination-mysql`	`0.1.20`	✅	✅
`destination-mysql-strict-encrypt`	❌ `0.1.21` (mismatch: `0.1.20`)	🔵 (ignored)	🔵 (ignored)
`destination-oracle`	`0.1.19`	✅	✅
`destination-oracle-strict-encrypt`	`0.1.19`	🔵 (ignored)	🔵 (ignored)
`destination-postgres`	`0.3.26`	✅	✅
`destination-postgres-strict-encrypt`	`0.3.26`	🔵 (ignored)	🔵 (ignored)
`destination-redshift`	`0.3.56`	✅	✅
`destination-snowflake`	`0.4.47`	✅	✅
`destination-tidb`	`0.1.0`	✅	✅

See "Actionable Items" below for how to resolve warnings and errors.

👀 Other Modules (1)

base-normalization

Actionable Items

(click to expand)

Category	Status	Actionable Item
Version	❌ mismatch	The version of the connector is different from its normal variant. Please bump the version of the connector.
	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
Changelog	⚠ doc not found	The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like `source-jdbc` is not published or documented). Please double-check to make sure that it is not a bug.
	❌ changelog missing	There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish	⚠ not in seed	The connector is not in the seed file (e.g. `source_definitions.yaml`), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
	❌ diff seed version	The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the `/publish` command to publish the latest version.

github-actions · 2023-02-02T01:04:05Z

Airbyte Code Coverage

There is no coverage information present for the Files changed

Total Project Coverage	24.51%	❌

github-actions · 2023-02-02T01:20:59Z

Platform Test Results

  243 files ±0   243 suites ±0 22m 37s ⏱️ -42s
1 663 tests ±0 1 652 ✔️ ±0 11 💤 ±0 0 ❌ ±0
1 682 runs ±0 1 671 ✔️ ±0 11 💤 ±0 0 ❌ ±0

Results for commit e54ccaf. ± Comparison against base commit aa5ed6d.

♻️ This comment has been updated with latest results.

github-actions · 2023-02-02T01:43:09Z

Kube Test Results

  47 files ±0   47 suites ±0 19m 17s ⏱️ + 10m 2s
208 tests ±0 202 ✔️ +30 6 💤 - 30 0 ❌ ±0
226 runs ±0 220 ✔️ +30 6 💤 - 30 0 ❌ ±0

Results for commit e54ccaf. ± Comparison against base commit aa5ed6d.

This pull request skips 3 and un-skips 33 tests.

io.airbyte.test.acceptance.ContainerOrchestratorAcceptanceTests ‑ testCancelSyncWhenCancelledWhenWorkerIsNotRunning()
io.airbyte.test.acceptance.ContainerOrchestratorAcceptanceTests ‑ testCancelSyncWithInterruption()
io.airbyte.test.acceptance.ContainerOrchestratorAcceptanceTests ‑ testDowntimeDuringSync()

io.airbyte.test.acceptance.BasicAcceptanceTests ‑ testCancelSync()
io.airbyte.test.acceptance.BasicAcceptanceTests ‑ testCreateConnection()
io.airbyte.test.acceptance.BasicAcceptanceTests ‑ testCreateDestination()
io.airbyte.test.acceptance.BasicAcceptanceTests ‑ testCronSync()
io.airbyte.test.acceptance.BasicAcceptanceTests ‑ testDeleteConnection()
io.airbyte.test.acceptance.BasicAcceptanceTests ‑ testDestinationCheckConnection()
io.airbyte.test.acceptance.BasicAcceptanceTests ‑ testDiscoverSourceSchema()
io.airbyte.test.acceptance.BasicAcceptanceTests ‑ testFailedGet404()
io.airbyte.test.acceptance.BasicAcceptanceTests ‑ testGetDestinationSpec()
io.airbyte.test.acceptance.BasicAcceptanceTests ‑ testGetSourceSpec()
…

♻️ This comment has been updated with latest results.

edgao · 2023-02-02T03:25:15Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/4070976155
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/4070976155
🐛 https://gradle.com/s/sgbqwprgoe7gu

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 SKIPPED [1] integration_tests/test_drop_scd_overwrite.py:56: DestinationType.ORACLE does not support incremental sync with schema change yet
	 SKIPPED [1] integration_tests/test_drop_scd_overwrite.py:56: DestinationType.TIDB does not support incremental sync with schema change yet
	 SKIPPED [3] integration_tests/test_ephemeral.py:102: ephemeral materialization isn't supported in ClickHouse yet
	 SKIPPED [1] integration_tests/test_ephemeral.py:59: Skipping test for column limit, because in MySQL, the max number of columns is limited by row size (8KB)
	 SKIPPED [1] integration_tests/test_normalization.py:82: Destinations DestinationType.CLICKHOUSE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:144: DestinationType.CLICKHOUSE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:147: DestinationType.MSSQL is disabled as it doesnt fully support schema change in incremental yet
	 SKIPPED [2] integration_tests/test_normalization.py:135: DestinationType.MYSQL does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:82: Destinations DestinationType.ORACLE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:135: DestinationType.ORACLE does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:144: DestinationType.SNOWFLAKE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:144: DestinationType.TIDB is disabled as it doesnt support schema change in incremental yet (column type changes)
	 FAILED integration_tests/test_drop_scd_overwrite.py::test_reset_scd_on_overwrite[DestinationType.REDSHIFT]
	 FAILED integration_tests/test_ephemeral.py::test_destination_supported_limits[DestinationType.REDSHIFT-1000]
	 FAILED integration_tests/test_ephemeral.py::test_destination_failure_over_limits[Redshift-1665-target lists can have at most 1664 entries]
	 FAILED integration_tests/test_ephemeral.py::test_empty_streams[DestinationType.REDSHIFT]
	 FAILED integration_tests/test_ephemeral.py::test_stream_with_1_airbyte_column[DestinationType.REDSHIFT]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.REDSHIFT-test_nested_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.REDSHIFT-test_simple_streams]
	 FAILED integration_tests/test_normalization.py::test_redshift_normalization_migration
	 �[31m============ �[31m�[1m8 failed�[0m, �[32m39 passed�[0m, �[33m15 skipped�[0m�[31m in 2886.31s (0:48:06)�[0m�[31m =============�[0m

edgao · 2023-02-02T03:44:14Z

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/4071063342
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/4071063342
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

edgao · 2023-02-02T04:48:14Z

/test connector=connectors/destination-bigquery

🕑 connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/4071357377
✅ connectors/destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/4071357377
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

edgao added 3 commits February 1, 2023 16:53

Revert "Normalization: handle non-object top-level schemas; treat bin…

7fae32d

…ary data as string (#22165)" This reverts commit 8276d03.

Revert "Normalization: check for ref type existence (#22161)"

804eeaf

This reverts commit dbe56d6.

Revert "🎉Updated normalization to handle new datatypes (#19721)"

4d20c86

This reverts commit c1d7736.

octavia-squidington-iii added area/documentation Improvements or additions to documentation normalization labels Feb 2, 2023

edgao temporarily deployed to more-secrets February 2, 2023 00:56 — with GitHub Actions Inactive

revert dest definitions

b7af3a8

edgao temporarily deployed to more-secrets February 2, 2023 01:00 — with GitHub Actions Inactive

also dockerfile

4abde74

edgao temporarily deployed to more-secrets February 2, 2023 01:05 — with GitHub Actions Inactive

edgao added 2 commits February 1, 2023 19:34

re-add to changelog

f3ae4ae

add comment in dockerfile

866358b

edgao temporarily deployed to more-secrets February 2, 2023 03:37 — with GitHub Actions Inactive

Merge branch 'master' into edgao/revert_normalization_protocol_v1

e54ccaf

edgao temporarily deployed to more-secrets February 2, 2023 03:46 — with GitHub Actions Inactive

Merge branch 'master' into edgao/revert_normalization_protocol_v1

4d46932

edgao marked this pull request as ready for review February 4, 2023 00:24

edgao requested a review from a team as a code owner February 4, 2023 00:24

ryankfu approved these changes Feb 4, 2023

View reviewed changes

etsybaev approved these changes Feb 5, 2023

View reviewed changes

edgao merged commit 517fc6a into master Feb 6, 2023

edgao deleted the edgao/revert_normalization_protocol_v1 branch February 6, 2023 18:14

github-actions bot mentioned this pull request Feb 7, 2023

Bump helm chart version reference to 0.43.28 #22489

Closed

This was referenced Feb 8, 2023

Bump Airbyte version from 0.40.32 to 0.40.33 #22578

Closed

Bump Airbyte version from 0.40.32 to 0.40.33 #22689

Closed

Bump Airbyte version from 0.40.32 to 0.40.33 #22839

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Normalization: Revert to protocol v0 #22283

Normalization: Revert to protocol v0 #22283

edgao commented Feb 2, 2023 •

edited

Loading

github-actions bot commented Feb 2, 2023 •

edited

Loading

✅ Sources (0)

❌ Destinations (16)

👀 Other Modules (1)

Actionable Items

github-actions bot commented Feb 2, 2023 •

edited

Loading

github-actions bot commented Feb 2, 2023 •

edited

Loading

github-actions bot commented Feb 2, 2023 •

edited

Loading

edgao commented Feb 2, 2023 •

edited by github-actions bot

Loading

edgao commented Feb 2, 2023 •

edited by github-actions bot

Loading

edgao commented Feb 2, 2023 •

edited by github-actions bot

Loading

Normalization: Revert to protocol v0 #22283

Normalization: Revert to protocol v0 #22283

Conversation

edgao commented Feb 2, 2023 • edited Loading

What

github-actions bot commented Feb 2, 2023 • edited Loading

Affected Connector Report

✅ Sources (0)

❌ Destinations (16)

👀 Other Modules (1)

Actionable Items

github-actions bot commented Feb 2, 2023 • edited Loading

Airbyte Code Coverage

github-actions bot commented Feb 2, 2023 • edited Loading

Platform Test Results

github-actions bot commented Feb 2, 2023 • edited Loading

Kube Test Results

edgao commented Feb 2, 2023 • edited by github-actions bot Loading

Build Failed

edgao commented Feb 2, 2023 • edited by github-actions bot Loading

Build Passed

edgao commented Feb 2, 2023 • edited by github-actions bot Loading

Build Passed

edgao commented Feb 2, 2023 •

edited

Loading

github-actions bot commented Feb 2, 2023 •

edited

Loading

github-actions bot commented Feb 2, 2023 •

edited

Loading

github-actions bot commented Feb 2, 2023 •

edited

Loading

github-actions bot commented Feb 2, 2023 •

edited

Loading

edgao commented Feb 2, 2023 •

edited by github-actions bot

Loading

edgao commented Feb 2, 2023 •

edited by github-actions bot

Loading

edgao commented Feb 2, 2023 •

edited by github-actions bot

Loading