🎉 Destination ClickHouse: bump dbt-clickhouse to v1.4.0 #23023

mshustov · 2023-02-14T18:35:25Z

What

Updates DBT-clickhouse to v1.4.0

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

marcosmarxm · 2023-02-14T22:02:13Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/4178464944
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/4178464944
🐛 https://gradle.com/s/jgtuxetph5hog

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 SKIPPED [1] integration_tests/test_drop_scd_overwrite.py:56: DestinationType.ORACLE does not support incremental sync with schema change yet
	 SKIPPED [1] integration_tests/test_drop_scd_overwrite.py:56: DestinationType.TIDB does not support incremental sync with schema change yet
	 SKIPPED [3] integration_tests/test_ephemeral.py:102: ephemeral materialization isn't supported in ClickHouse yet
	 SKIPPED [1] integration_tests/test_ephemeral.py:59: Skipping test for column limit, because in MySQL, the max number of columns is limited by row size (8KB)
	 SKIPPED [1] integration_tests/test_normalization.py:82: Destinations DestinationType.CLICKHOUSE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:152: DestinationType.MSSQL is disabled as it doesnt fully support schema change in incremental yet
	 SKIPPED [2] integration_tests/test_normalization.py:135: DestinationType.MYSQL does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:135: DestinationType.ORACLE does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:82: Destinations DestinationType.ORACLE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:149: DestinationType.SNOWFLAKE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:149: DestinationType.TIDB is disabled as it doesnt support schema change in incremental yet (column type changes)
	 ERROR integration_tests/test_normalization.py::test_redshift_normalization_migration
	 FAILED integration_tests/test_drop_scd_overwrite.py::test_reset_scd_on_overwrite[DestinationType.BIGQUERY]
	 FAILED integration_tests/test_drop_scd_overwrite.py::test_reset_scd_on_overwrite[DestinationType.CLICKHOUSE]
	 FAILED integration_tests/test_drop_scd_overwrite.py::test_reset_scd_on_overwrite[DestinationType.DUCKDB]
	 FAILED integration_tests/test_ephemeral.py::test_destination_supported_limits[DestinationType.BIGQUERY-1000]
	 FAILED integration_tests/test_ephemeral.py::test_destination_supported_limits[DestinationType.DUCKDB-1000]
	 FAILED integration_tests/test_ephemeral.py::test_destination_failure_over_limits[BigQuery-3000-The view is too large.]
	 FAILED integration_tests/test_ephemeral.py::test_empty_streams[DestinationType.BIGQUERY]
	 FAILED integration_tests/test_ephemeral.py::test_empty_streams[DestinationType.DUCKDB]
	 FAILED integration_tests/test_ephemeral.py::test_stream_with_1_airbyte_column[DestinationType.BIGQUERY]
	 FAILED integration_tests/test_ephemeral.py::test_stream_with_1_airbyte_column[DestinationType.DUCKDB]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.BIGQUERY-test_simple_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.BIGQUERY-test_nested_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.CLICKHOUSE-test_simple_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.DUCKDB-test_simple_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.DUCKDB-test_nested_streams]
	 �[31m======= �[31m�[1m15 failed�[0m, �[32m39 passed�[0m, �[33m14 skipped�[0m, �[31m�[1m1 error�[0m�[31m in 3135.42s (0:52:15)�[0m�[31m ========�[0m

mshustov · 2023-02-15T09:47:01Z

@marcosmarxm @edgao integration tests for many unrelated destinations (BigQuery, DuckDB) failed. It's unlikely to be caused by the changes in the current PR. How should I proceed?

marcosmarxm · 2023-02-15T13:34:33Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/4184411237
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/4184411237
🐛 https://gradle.com/s/yk73wrtkxigaa

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 SKIPPED [1] integration_tests/test_drop_scd_overwrite.py:56: DestinationType.ORACLE does not support incremental sync with schema change yet
	 SKIPPED [1] integration_tests/test_drop_scd_overwrite.py:56: DestinationType.TIDB does not support incremental sync with schema change yet
	 SKIPPED [3] integration_tests/test_ephemeral.py:102: ephemeral materialization isn't supported in ClickHouse yet
	 SKIPPED [1] integration_tests/test_ephemeral.py:59: Skipping test for column limit, because in MySQL, the max number of columns is limited by row size (8KB)
	 SKIPPED [1] integration_tests/test_normalization.py:82: Destinations DestinationType.CLICKHOUSE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:152: DestinationType.MSSQL is disabled as it doesnt fully support schema change in incremental yet
	 SKIPPED [2] integration_tests/test_normalization.py:135: DestinationType.MYSQL does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:82: Destinations DestinationType.ORACLE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:135: DestinationType.ORACLE does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:149: DestinationType.SNOWFLAKE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:149: DestinationType.TIDB is disabled as it doesnt support schema change in incremental yet (column type changes)
	 ERROR integration_tests/test_normalization.py::test_redshift_normalization_migration
	 FAILED integration_tests/test_drop_scd_overwrite.py::test_reset_scd_on_overwrite[DestinationType.BIGQUERY]
	 FAILED integration_tests/test_drop_scd_overwrite.py::test_reset_scd_on_overwrite[DestinationType.CLICKHOUSE]
	 FAILED integration_tests/test_drop_scd_overwrite.py::test_reset_scd_on_overwrite[DestinationType.DUCKDB]
	 FAILED integration_tests/test_ephemeral.py::test_destination_supported_limits[DestinationType.BIGQUERY-1000]
	 FAILED integration_tests/test_ephemeral.py::test_destination_supported_limits[DestinationType.DUCKDB-1000]
	 FAILED integration_tests/test_ephemeral.py::test_destination_failure_over_limits[BigQuery-3000-The view is too large.]
	 FAILED integration_tests/test_ephemeral.py::test_empty_streams[DestinationType.BIGQUERY]
	 FAILED integration_tests/test_ephemeral.py::test_empty_streams[DestinationType.DUCKDB]
	 FAILED integration_tests/test_ephemeral.py::test_stream_with_1_airbyte_column[DestinationType.BIGQUERY]
	 FAILED integration_tests/test_ephemeral.py::test_stream_with_1_airbyte_column[DestinationType.DUCKDB]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.BIGQUERY-test_nested_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.BIGQUERY-test_simple_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.CLICKHOUSE-test_simple_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.DUCKDB-test_nested_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.DUCKDB-test_simple_streams]
	 �[31m======= �[31m�[1m15 failed�[0m, �[32m39 passed�[0m, �[33m14 skipped�[0m, �[31m�[1m1 error�[0m�[31m in 3489.45s (0:58:09)�[0m�[31m ========�[0m

edgao · 2023-02-15T15:52:41Z

cherry picked 18e6f01 and 3b9d192, I think that should help get tests passing (I completely forgot that these weren't merged to master yet >.> )

edgao · 2023-02-15T15:52:55Z

/test connector=bases/base-normalization

🕑 bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/4185674062
❌ bases/base-normalization https://github.com/airbytehq/airbyte/actions/runs/4185674062
🐛 https://gradle.com/s/o6kyhoaa76iy6

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 SKIPPED [1] integration_tests/test_drop_scd_overwrite.py:56: DestinationType.ORACLE does not support incremental sync with schema change yet
	 SKIPPED [1] integration_tests/test_drop_scd_overwrite.py:56: DestinationType.TIDB does not support incremental sync with schema change yet
	 SKIPPED [3] integration_tests/test_ephemeral.py:102: ephemeral materialization isn't supported in ClickHouse yet
	 SKIPPED [1] integration_tests/test_ephemeral.py:59: Skipping test for column limit, because in MySQL, the max number of columns is limited by row size (8KB)
	 SKIPPED [1] integration_tests/test_normalization.py:149: DestinationType.CLICKHOUSE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:82: Destinations DestinationType.CLICKHOUSE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:152: DestinationType.MSSQL is disabled as it doesnt fully support schema change in incremental yet
	 SKIPPED [2] integration_tests/test_normalization.py:135: DestinationType.MYSQL does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:135: DestinationType.ORACLE does not support incremental yet
	 SKIPPED [1] integration_tests/test_normalization.py:82: Destinations DestinationType.ORACLE does not support nested streams
	 SKIPPED [1] integration_tests/test_normalization.py:149: DestinationType.SNOWFLAKE is disabled as it doesnt support schema change in incremental yet (column type changes)
	 SKIPPED [1] integration_tests/test_normalization.py:149: DestinationType.TIDB is disabled as it doesnt support schema change in incremental yet (column type changes)
	 FAILED integration_tests/test_drop_scd_overwrite.py::test_reset_scd_on_overwrite[DestinationType.BIGQUERY]
	 FAILED integration_tests/test_ephemeral.py::test_destination_supported_limits[DestinationType.BIGQUERY-1000]
	 FAILED integration_tests/test_ephemeral.py::test_destination_failure_over_limits[BigQuery-3000-The view is too large.]
	 FAILED integration_tests/test_ephemeral.py::test_empty_streams[DestinationType.BIGQUERY]
	 FAILED integration_tests/test_ephemeral.py::test_stream_with_1_airbyte_column[DestinationType.BIGQUERY]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.BIGQUERY-test_simple_streams]
	 FAILED integration_tests/test_normalization.py::test_normalization[DestinationType.BIGQUERY-test_nested_streams]
	 �[31m============ �[31m�[1m7 failed�[0m, �[32m40 passed�[0m, �[33m15 skipped�[0m�[31m in 3289.93s (0:54:49)�[0m�[31m =============�[0m

jbfbell · 2023-02-15T17:11:10Z

airbyte-integrations/bases/base-normalization/integration_tests/dbt_integration_test.py

@@ -476,7 +476,7 @@ def run_check_dbt_command(self, normalization_image: str, command: str, cwd: str
        """
        Run dbt subprocess while checking and counting for "ERROR", "FAIL" or "WARNING" printed in its outputs
        """
-        if normalization_image.startswith("airbyte/normalization-oracle"):
+        if normalization_image.startswith("airbyte/normalization-oracle") or normalization_image.startswith("airbyte/normalization-clickhouse"):


I think two is fine to combine with the or clause, but if we need to add another it might be worth switching to a collection

if any([normalization_image.startswith(x) for x in ["airbyte/normalization-oracle", "airbyte/normalization-clickhouse"]])

once these live as connectors code it won't be a problem 😉 (but also yes. this entire section is kind of iffy. It kind of looks like recent dbt versions removed the --event-buffer-size option entirely)

edgao · 2023-02-15T21:15:42Z

@grishick - base-normalization tests are passing on all destinations except bigquery, because of an upstream dependency bump. This PR (and your PR) both only touch clickhouse - I think we should be OK to publish skipping tests?

the deps issue:

protocol-models bumped google-api-client to 2.2.0
destination-bigquery depends on protocol-models, so now gradle is resolving it to use 2.2.0
normalization tests fail because the destination connector fails - You are currently running with version 2.2.0 of google-api-client. You need at least version 1.31.1 of google-api-client to run version 1.31.0 of the BigQuery API library.
- i.e. it fails before normalization even runs

edgao · 2023-02-15T22:13:13Z

/publish connector=bases/base-normalization run-tests=false

🕑 Publishing the following connectors:
bases/base-normalization
https://github.com/airbytehq/airbyte/actions/runs/4188607713

Connector	Did it publish?	Were definitions generated?
bases/base-normalization	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

edgao · 2023-02-15T22:15:09Z

running with run-tests=false. normalizatoin dockerfile is already at 0.2.26.

c-p-b · 2023-02-17T15:46:32Z

FYI, this was merged because it qualifies as a P1 issue per this comment: https://github.com/airbytehq/oncall/issues/1474#issuecomment-1422777303

bump dbt-clickhouse to 1.4.0

8a2662c

mshustov requested a review from a team as a code owner February 14, 2023 18:35

octavia-squidington-iii added normalization community labels Feb 14, 2023

Merge branch 'master' into dbt-clickhouse-140

4d4c7f6

marcosmarxm self-assigned this Feb 14, 2023

edgao mentioned this pull request Feb 14, 2023

Publish normalization for clickhouse SSL fix #23034

Closed

Merge branch 'master' into dbt-clickhouse-140

1744d97

edgao added 2 commits February 15, 2023 07:49

fix clickhouse integration test

4ccc0bf

exclude duckdb from tests

d79a25c

jbfbell approved these changes Feb 15, 2023

View reviewed changes

Merge branch 'master' into dbt-clickhouse-140

f9f0957

add to changelog

7cba790

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Feb 15, 2023

bump normalization version in definitions

2bb487b

edgao added the ready-to-ship label Feb 15, 2023

marcosmarxm changed the title ~~🎉ClickHouse: bump dbt-clickhouse to v1.4.0~~ 🎉 Destination ClickHouse: bump dbt-clickhouse to v1.4.0 Feb 16, 2023

edgao mentioned this pull request Feb 16, 2023

[DNM] Destination ClickHouse: bump dbt-clickhouse to v1.4.0 #23177

Closed

edgao temporarily deployed to more-secrets February 16, 2023 23:21 — with GitHub Actions Inactive

edgao temporarily deployed to more-secrets February 17, 2023 00:12 — with GitHub Actions Inactive

grishick merged commit 2ce3c17 into airbytehq:master Feb 17, 2023

mshustov deleted the dbt-clickhouse-140 branch February 17, 2023 10:35

grishick mentioned this pull request Feb 17, 2023

Update clickhouse destination changelog #23219

Closed

This was referenced Apr 1, 2023

Destination Clickhouse: Normalization should accept non-ssl-tunnel configs #22046

Closed

Fix clickhouse normalization non-ssl error #22387

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Destination ClickHouse: bump dbt-clickhouse to v1.4.0 #23023

🎉 Destination ClickHouse: bump dbt-clickhouse to v1.4.0 #23023

mshustov commented Feb 14, 2023 •

edited

Loading

marcosmarxm commented Feb 14, 2023 •

edited by github-actions bot

Loading

mshustov commented Feb 15, 2023

marcosmarxm commented Feb 15, 2023 •

edited by github-actions bot

Loading

edgao commented Feb 15, 2023 •

edited

Loading

edgao commented Feb 15, 2023 •

edited by github-actions bot

Loading

jbfbell Feb 15, 2023

edgao Feb 15, 2023

edgao commented Feb 15, 2023

edgao commented Feb 15, 2023 •

edited by github-actions bot

Loading

edgao commented Feb 15, 2023

c-p-b commented Feb 17, 2023

🎉 Destination ClickHouse: bump dbt-clickhouse to v1.4.0 #23023

🎉 Destination ClickHouse: bump dbt-clickhouse to v1.4.0 #23023

Conversation

mshustov commented Feb 14, 2023 • edited Loading

What

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

marcosmarxm commented Feb 14, 2023 • edited by github-actions bot Loading

Build Failed

mshustov commented Feb 15, 2023

marcosmarxm commented Feb 15, 2023 • edited by github-actions bot Loading

Build Failed

edgao commented Feb 15, 2023 • edited Loading

edgao commented Feb 15, 2023 • edited by github-actions bot Loading

Build Failed

jbfbell Feb 15, 2023

Choose a reason for hiding this comment

edgao Feb 15, 2023

Choose a reason for hiding this comment

edgao commented Feb 15, 2023

edgao commented Feb 15, 2023 • edited by github-actions bot Loading

edgao commented Feb 15, 2023

c-p-b commented Feb 17, 2023

mshustov commented Feb 14, 2023 •

edited

Loading

marcosmarxm commented Feb 14, 2023 •

edited by github-actions bot

Loading

marcosmarxm commented Feb 15, 2023 •

edited by github-actions bot

Loading

edgao commented Feb 15, 2023 •

edited

Loading

edgao commented Feb 15, 2023 •

edited by github-actions bot

Loading

edgao commented Feb 15, 2023 •

edited by github-actions bot

Loading