🐛 Destination ClickHouse: Update Normalization Docker File #19573

genzgd · 2022-11-17T22:30:57Z

What

Update the dbt version and the dbt-clickhouse version in the ClickHouse normalization Dockerfile

🚨 User Impact 🚨

Fixes an issue for ClickHouse normalization containers when run on Apple M1 chips (and possibly on Linux with old kernel versions)

Community member or Airbyter

Community member? Grant edit access to maintainers (instructions)
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

I was unable to get integration tests running locally.

CLAassistant · 2022-11-17T22:31:02Z

All committers have signed the CLA.

genzgd · 2022-11-17T22:40:11Z

@grishick -- Apologies for being unable to get tests running locally.

marcosmarxm · 2022-11-22T20:26:01Z

Hello 👋, first thank you for this amazing contribution.

We really appreciate the effort you've made to improve the project.
We ask you patience for the code review. Last month our team was focused on Hacktoberfest event and that probably left some PR without the proper feedback. And this week, due to the Thanksgiving US Holiday, most our team is out of office with their families. Another important piece of information why code won't be merge this week is: as a safety measure the core team has decided to freeze merging code to main branch to keep the release stable. Next week we'll return to you with the proper code review and update the status of your contribution.

If you have any questions feel free to send me a message in Slack!
Thanks!

marcosmarxm · 2022-12-02T22:39:50Z

/test connector=connectors/destination-clickhouse

🕑 connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/3605602672
✅ connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/3605602672
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

marcosmarxm · 2022-12-05T11:38:28Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/3619658728
✅ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/3619658728
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%
	 Name                                                 Stmts   Miss  Cover   Missing
	 ----------------------------------------------------------------------------------
	 source_acceptance_test/base.py                          12      4    67%   16-19
	 source_acceptance_test/config.py                       140      5    96%   87, 93, 238, 242-243
	 source_acceptance_test/conftest.py                     199     93    53%   35, 41-43, 48, 53, 59, 65, 71, 77-79, 98, 103-105, 111-113, 119-120, 125-126, 131, 137, 146-155, 161-166, 181, 205, 236, 242, 250-255, 263-268, 276-289, 294-300, 307-318, 325-341
	 source_acceptance_test/plugin.py                        69     25    64%   22-23, 31, 36, 120-140, 144-148
	 source_acceptance_test/tests/test_core.py              398    111    72%   53, 58, 87-95, 100-107, 111-112, 116-117, 299, 337-354, 363-371, 375-380, 386, 419-424, 462-469, 512-514, 517, 582-590, 602-605, 610, 666-667, 673, 676, 712-722, 735-760
	 source_acceptance_test/tests/test_incremental.py       158     14    91%   52-59, 64-77, 240
	 source_acceptance_test/utils/asserts.py                 39      2    95%   62-63
	 source_acceptance_test/utils/common.py                  94     10    89%   16-17, 32-38, 72, 75
	 source_acceptance_test/utils/compare.py                 62     23    63%   21-51, 68, 97-99
	 source_acceptance_test/utils/connector_runner.py       112     50    55%   23-26, 32, 36, 39-68, 71-73, 76-78, 81-83, 86-88, 91-93, 96-114, 148-150
	 source_acceptance_test/utils/json_schema_helper.py     107     13    88%   30-31, 38, 41, 65-68, 96, 120, 192-194
	 ----------------------------------------------------------------------------------
	 TOTAL                                                 1569    350    78%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%
Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/utils.py                             51      1    98%
normalization/transform_catalog/destination_name_transformer.py     166      5    97%
normalization/transform_catalog/stream_processor.py                 595     35    94%
normalization/transform_catalog/catalog_processor.py                147     12    92%
normalization/transform_catalog/dbt_macro.py                         22      3    86%
normalization/transform_catalog/table_name_registry.py              174     51    71%
normalization/transform_catalog/transform.py                         61     22    64%
normalization/transform_config/transform.py                         189     77    59%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    206    86%

Build Passed

Test summary info:

	 =========================== short test summary info ============================
	 SKIPPED [1] integration_tests/test_drop_scd_overwrite.py:56: DestinationType.ORACLE does not support incremental sync with schema change yet
	 SKIPPED [1] integration_tests/test_drop_scd_overwrite.py:56: DestinationType.TIDB does not support incremental sync with schema change yet
	 SKIPPED [3] integration_tests/test_ephemeral.py:102: ephemeral materialization isn't supported in ClickHouse yet
	 SKIPPED [1] integration_tests/test_ephemeral.py:59: Skipping test for column limit, because in MySQL, the max number of columns is limited by row size (8KB)
	 SKIPPED [1] integration_tests/test_normalization.py:82: Destinations DestinationType.CLICKHOUSE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:144: DestinationType.CLICKHOUSE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:147: DestinationType.MSSQL is disabled as it doesnt fully support schema change in incremental yet
	 SKIPPED [2] integration_tests/test_normalization.py:135: DestinationType.MYSQL does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:82: Destinations DestinationType.ORACLE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:135: DestinationType.ORACLE does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:144: DestinationType.SNOWFLAKE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:144: DestinationType.TIDB is disabled as it doesnt support schema change in incremental yet (column type changes)
	 �[32m================= �[32m�[1m47 passed�[0m, �[33m15 skipped�[0m�[32m in 5093.26s (1:24:53)�[0m�[32m ==================�[0m
	 =========================== short test summary info ============================
	 SKIPPED [1] integration_tests/test_drop_scd_overwrite.py:56: DestinationType.ORACLE does not support incremental sync with schema change yet
	 SKIPPED [1] integration_tests/test_drop_scd_overwrite.py:56: DestinationType.TIDB does not support incremental sync with schema change yet
	 SKIPPED [3] integration_tests/test_ephemeral.py:102: ephemeral materialization isn't supported in ClickHouse yet
	 SKIPPED [1] integration_tests/test_ephemeral.py:59: Skipping test for column limit, because in MySQL, the max number of columns is limited by row size (8KB)
	 SKIPPED [1] integration_tests/test_normalization.py:144: DestinationType.CLICKHOUSE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:82: Destinations DestinationType.CLICKHOUSE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:147: DestinationType.MSSQL is disabled as it doesnt fully support schema change in incremental yet
	 SKIPPED [2] integration_tests/test_normalization.py:135: DestinationType.MYSQL does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:135: DestinationType.ORACLE does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:82: Destinations DestinationType.ORACLE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:144: DestinationType.SNOWFLAKE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:144: DestinationType.TIDB is disabled as it doesnt support schema change in incremental yet (column type changes)
	 �[32m================= �[32m�[1m47 passed�[0m, �[33m15 skipped�[0m�[32m in 4307.08s (1:11:47)�[0m�[32m ==================�[0m

marcosmarxm · 2022-12-05T15:09:41Z

/publish connector=bases/base-normalization

🕑 Publishing the following connectors:
bases/base-normalization
https://github.com/airbytehq/airbyte/actions/runs/3621333274

Connector	Did it publish?	Were definitions generated?
bases/base-normalization	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

marcosmarxm · 2022-12-06T17:03:26Z

/publish connector=bases/base-normalization run-tests=false

🕑 Publishing the following connectors:
bases/base-normalization
https://github.com/airbytehq/airbyte/actions/runs/3631706478

Connector	Did it publish?	Were definitions generated?
bases/base-normalization	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

marcosmarxm · 2022-12-06T17:18:18Z

/publish connector=connectors/destination-clickhouse

🕑 Publishing the following connectors:
connectors/destination-clickhouse
https://github.com/airbytehq/airbyte/actions/runs/3631832879

Connector	Did it publish?	Were definitions generated?
connectors/destination-clickhouse	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

marcosmarxm

Thanks @genzgd

Update ClickHouse normalization docker file

f80a98d

genzgd requested a review from a team as a code owner November 17, 2022 22:30

octavia-squidington-iv added normalization community labels Nov 17, 2022

Merge branch 'master' into clickhouse_dbt_1_3

1b335c1

Merge branch 'master' into clickhouse_dbt_1_3

464f5c6

bump destination and norm version

33bcbee

marcosmarxm requested a review from a team as a code owner December 5, 2022 15:09

octavia-squidington-iv added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Dec 5, 2022

octavia-squidington-iv added bounty connectors/destination/clickhouse labels Dec 5, 2022

marcosmarxm mentioned this pull request Dec 5, 2022

Update dbt-clickhouse version from 1.1.8 to 1.2.2 #18370

Closed

37 tasks

sajarin added the internal label Dec 5, 2022

octavia-squidington-iv removed the bounty label Dec 5, 2022

Merge branch 'master' into clickhouse_dbt_1_3

1f413b6

octavia-squidington-iii and others added 2 commits December 6, 2022 17:47

auto-bump connector version

fd4f075

update doc

2ffef72

marcosmarxm approved these changes Dec 6, 2022

View reviewed changes

marcosmarxm merged commit b7816f4 into airbytehq:master Dec 6, 2022

octavia-squidington-iii mentioned this pull request Dec 6, 2022

Bump Airbyte version from 0.40.23 to 0.40.24 #20161

Closed

edgao mentioned this pull request Dec 6, 2022

Destination Clickhouse: Publish strict encrypt variant #20171

Merged

octavia-squidington-iii mentioned this pull request Dec 7, 2022

Bump Airbyte version from 0.40.23 to 0.40.24 #20206

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Destination ClickHouse: Update Normalization Docker File #19573

🐛 Destination ClickHouse: Update Normalization Docker File #19573

genzgd commented Nov 17, 2022

CLAassistant commented Nov 17, 2022 •

edited

Loading

genzgd commented Nov 17, 2022

marcosmarxm commented Nov 22, 2022

marcosmarxm commented Dec 2, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Dec 5, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Dec 5, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Dec 6, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Dec 6, 2022 •

edited by github-actions bot

Loading

marcosmarxm left a comment

🐛 Destination ClickHouse: Update Normalization Docker File #19573

🐛 Destination ClickHouse: Update Normalization Docker File #19573

Conversation

genzgd commented Nov 17, 2022

What

🚨 User Impact 🚨

Community member or Airbyter

Airbyter

CLAassistant commented Nov 17, 2022 • edited Loading

genzgd commented Nov 17, 2022

marcosmarxm commented Nov 22, 2022

marcosmarxm commented Dec 2, 2022 • edited by github-actions bot Loading

Build Passed

marcosmarxm commented Dec 5, 2022 • edited by github-actions bot Loading

Build Passed

marcosmarxm commented Dec 5, 2022 • edited by github-actions bot Loading

marcosmarxm commented Dec 6, 2022 • edited by github-actions bot Loading

marcosmarxm commented Dec 6, 2022 • edited by github-actions bot Loading

marcosmarxm left a comment

Choose a reason for hiding this comment

CLAassistant commented Nov 17, 2022 •

edited

Loading

marcosmarxm commented Dec 2, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Dec 5, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Dec 5, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Dec 6, 2022 •

edited by github-actions bot

Loading

marcosmarxm commented Dec 6, 2022 •

edited by github-actions bot

Loading