Greg/clickhouse polishing #17483

grishick · 2022-09-30T22:15:56Z

This PR is co-authored with @mshustov via #16970

What

ClickHouse configuration in Airbyte requires both HTTP and Native port to be specified.
The HTTP port is used for JDBC driver to communicate with ClickHouse, the native port is used by dbt-clickhouse during transformation step.
The native port is not a requirement for DBT-clickhouse anymore since the last the last versions of dbt-clickhouse support both HTTP and native protocols. We want to switch to HTTP by default everywhere since it reduces the technical complexity of the integration.

How

This PR removes requirements for Native port to be provided to reduce the configuration size and improve UX. I removed all the mentions of TCP port in the codebase introduced in #9340 and #14783

🚨 User Impact 🚨

The number of required fields in ClickHouse configuration was reduced. I didn't manage to run Airbyte locally (neither master branch nor the current one) to take a screenshot. In short, it should remove native port field:

SUB_BUILD=PLATFORM ./gradlew build fails with

DockerProcessFactoryTest > testEnvMapSet() FAILED
    io.airbyte.workers.exception.WorkerException: Could not find image: busybox

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

…icklhouse-polishing

github-actions · 2022-09-30T22:16:48Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

destination-jdbc
destination-oracle
destination-bigquery
destination-postgres-strict-encrypt
destination-mysql
destination-mssql-strict-encrypt
destination-clickhouse-strict-encrypt
destination-oracle-strict-encrypt
destination-redshift
destination-snowflake
destination-mssql
destination-postgres
destination-clickhouse
destination-bigquery-denormalized
destination-mysql-strict-encrypt
destination-tidb

grishick · 2022-09-30T22:16:50Z

/test connector=connectors/destination-clickhouse

🕑 connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/3162000209
✅ connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/3162000209
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1435    623    57%

Build Passed

Test summary info:

All Passed

github-actions · 2022-09-30T23:38:49Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

destination-clickhouse-strict-encrypt
destination-tidb
destination-clickhouse
destination-bigquery
destination-mysql-strict-encrypt
destination-redshift
destination-mssql-strict-encrypt
destination-jdbc
destination-mssql
destination-postgres
destination-snowflake
destination-mysql
destination-oracle-strict-encrypt
destination-postgres-strict-encrypt
destination-oracle
destination-bigquery-denormalized

grishick · 2022-10-01T01:07:41Z

/publish connector=connectors/destination-clickhouse

🕑 Publishing the following connectors:
connectors/destination-clickhouse
https://github.com/airbytehq/airbyte/actions/runs/3162613062

Connector	Did it publish?	Were definitions generated?
connectors/destination-clickhouse	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

grishick · 2022-10-01T01:09:04Z

/publish connector=connectors/destination-clickhouse-strict-encrypt auto-bump-version=false

🕑 Publishing the following connectors:
connectors/destination-clickhouse-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/3162617893

Connector	Did it publish?	Were definitions generated?
connectors/destination-clickhouse-strict-encrypt	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

grishick · 2022-10-01T02:47:50Z

/publish connector=connectors/destination-clickhouse-strict-encrypt auto-bump-version=false

🕑 Publishing the following connectors:
connectors/destination-clickhouse-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/3162898590

Connector	Did it publish?	Were definitions generated?
connectors/destination-clickhouse-strict-encrypt	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

grishick · 2022-10-01T04:02:50Z

/test connector=connectors/destination-clickhouse-strict-encrypt

🕑 connectors/destination-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3163089076
❌ connectors/destination-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3163089076
🐛 https://gradle.com/s/vaor7lqnjn4gm

Build Failed

Test summary info:

Could not find result summary

grishick · 2022-10-01T04:17:35Z

/test connector=connectors/destination-clickhouse-strict-encrypt

🕑 connectors/destination-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3163136614
❌ connectors/destination-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3163136614
🐛 https://gradle.com/s/tcdfzhgzdhjdu

Build Failed

Test summary info:

	 =========================== short test summary info ============================
	 FAILED unit_tests/test_transform_config.py::TestTransformConfig::test_transform_clickhouse
	 �[31m======================== �[31m�[1m1 failed�[0m, �[32m183 passed�[0m�[31m in 2.16s�[0m�[31m =========================�[0m

github-actions · 2022-10-01T04:18:08Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

destination-mssql
destination-snowflake
destination-bigquery-denormalized
destination-redshift
destination-mysql
destination-oracle
destination-jdbc
destination-postgres
destination-mssql-strict-encrypt
destination-mysql-strict-encrypt
destination-bigquery
destination-clickhouse
destination-tidb
destination-postgres-strict-encrypt
destination-clickhouse-strict-encrypt
destination-oracle-strict-encrypt

grishick · 2022-10-01T06:38:43Z

/test connector=connectors/destination-clickhouse-strict-encrypt

🕑 connectors/destination-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3163509348
❌ connectors/destination-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3163509348
🐛 https://gradle.com/s/marf7otp6mvzq

Build Failed

Test summary info:

Could not find result summary

github-actions · 2022-10-01T06:39:33Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

destination-postgres-strict-encrypt
destination-tidb
destination-mysql
destination-bigquery-denormalized
destination-mysql-strict-encrypt
destination-oracle-strict-encrypt
destination-redshift
destination-bigquery
destination-clickhouse-strict-encrypt
destination-clickhouse
destination-mssql
destination-jdbc
destination-oracle
destination-mssql-strict-encrypt
destination-snowflake
destination-postgres

grishick · 2022-10-01T07:29:14Z

/test connector=connectors/destination-clickhouse-strict-encrypt

🕑 connectors/destination-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3163654774
✅ connectors/destination-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/3163654774
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 589    394    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1435    623    57%

Build Passed

Test summary info:

All Passed

github-actions · 2022-10-01T07:29:42Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

destination-postgres-strict-encrypt
destination-clickhouse
destination-mysql
destination-jdbc
destination-postgres
destination-redshift
destination-snowflake
destination-bigquery-denormalized
destination-mssql-strict-encrypt
destination-mssql
destination-mysql-strict-encrypt
destination-oracle-strict-encrypt
destination-oracle
destination-bigquery
destination-tidb
destination-clickhouse-strict-encrypt

grishick · 2022-10-01T19:10:25Z

/publish connector=connectors/destination-clickhouse-strict-encrypt auto-bump-version=false

🕑 Publishing the following connectors:
connectors/destination-clickhouse-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/3165801944

Connector	Did it publish?	Were definitions generated?
connectors/destination-clickhouse-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

…ations * master: (75 commits) source-sentry: migrate to per-stream states (#17466) Greg/clickhouse polishing (#17483) upgrade debezium version to 1.9.6 (#17459) 🐛 Source Twilio: Lookback_window config (#17478) hide S3 source connector from catalog (#17472) 🪟 Migrate styles for Connection-related Components (#17339) Added new title (#17480) Refactor & convert `PageTitle` to SCSS (#17139) updated releaseStage for zendesk-talk (#17477) [low-code] Apply log level to stream loggers (#17284) 🐛 Source Salesforce: filter out objects not supported by the Bulk API (#17453) Source Marketo: certify GA (#17445) Update greenhouse paginator (#17429) Add some services start validation to acceptance_tests.sh (#17425) 📖 Removes $ from terminal commands to allow direct copying. (#17467) migrate source GA connectors to per-stream states (2) (#17410) Source Klaviyo: bump CDK dependency (#17422) Source Pinterest: change releaseStage to GA (#17045) Source Pinterest: Set start_date dynamically based on API restrictions for lookup (#17387) updated releaseStage to generally_available (#17374) ...

* master: source-sentry: migrate to per-stream states (#17466) Greg/clickhouse polishing (#17483) upgrade debezium version to 1.9.6 (#17459) 🐛 Source Twilio: Lookback_window config (#17478) hide S3 source connector from catalog (#17472) 🪟 Migrate styles for Connection-related Components (#17339) Added new title (#17480) Refactor & convert `PageTitle` to SCSS (#17139) updated releaseStage for zendesk-talk (#17477) [low-code] Apply log level to stream loggers (#17284) 🐛 Source Salesforce: filter out objects not supported by the Bulk API (#17453) Source Marketo: certify GA (#17445) Update greenhouse paginator (#17429) Add some services start validation to acceptance_tests.sh (#17425) 📖 Removes $ from terminal commands to allow direct copying. (#17467)

* add icon for clickhouse in destination folder * use http port only in clickhouse * declare driver: http for dbt explicitly * bump destination clickhouse version Co-authored-by: restrry <restrry@gmail.com> Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

mshustov and others added 12 commits September 21, 2022 13:33

add icon for clickhouse in destination folder

32d32ea

use http port only in clickhouse

bb47de8

Merge branch 'master' into clicklhouse-polishing

aaa4b75

Merge remote-tracking branch 'origin' into clicklhouse-polishing

8695a84

fix a typo in clickhouse icon

cd23430

Merge remote-tracking branch 'upstream/clicklhouse-polishing' into cl…

e60e297

…icklhouse-polishing

declare driver: http for dbt explicitly

ec94b2f

Merge remote-tracking branch 'origin' into clicklhouse-polishing

37425a1

Merge remote-tracking branch 'origin' into clicklhouse-polishing

9874d5f

roll back changes to auto genereates files

cf69749

bump destination clickhouse version

e05cc2d

fix unit test

5dcb619

grishick requested review from a team as code owners September 30, 2022 22:15

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation normalization labels Sep 30, 2022

octavia-squidington-iv added connectors/destination/clickhouse connectors/destination/clickhouse-strict-encrypt labels Sep 30, 2022

grishick temporarily deployed to more-secrets September 30, 2022 22:17 Inactive

tuliren approved these changes Sep 30, 2022

View reviewed changes

grishick mentioned this pull request Sep 30, 2022

Destination ClickHouse: remove tcp port spec parameter #16970

Closed

37 tasks

grishick temporarily deployed to more-secrets September 30, 2022 22:50 Inactive

grishick temporarily deployed to more-secrets September 30, 2022 23:15 Inactive

Merge branch 'master' into greg/clickhouse-polishing

ef14ab9

grishick temporarily deployed to more-secrets September 30, 2022 23:40 Inactive

auto-bump connector version [ci skip]

f33261a

grishick temporarily deployed to more-secrets October 1, 2022 04:19 Inactive

grishick temporarily deployed to more-secrets October 1, 2022 06:41 Inactive

Fix normalization test for Clickhouse Strict Encrypt

d50192b

grishick force-pushed the greg/clickhouse-polishing branch from 7f2b744 to d50192b Compare October 1, 2022 07:28

grishick temporarily deployed to more-secrets October 1, 2022 07:31 Inactive

grishick merged commit 5f25d2d into master Oct 1, 2022

grishick deleted the greg/clickhouse-polishing branch October 1, 2022 19:40

grishick added the team/destinations Destinations team's backlog label Oct 1, 2022

octavia-squidington-iii mentioned this pull request Oct 5, 2022

Bump Airbyte version from 0.40.10 to 0.40.11 #17610

Merged

edgao mentioned this pull request Oct 12, 2022

Destination Clickhouse: Publish normalization after removing native port #17896

Merged

tuliren mentioned this pull request Oct 12, 2022

Connector dependency reporter does not include normalization changes #17897

Closed

mshustov mentioned this pull request Nov 10, 2022

Destination ClickHouse: doesn't have a logo #19283

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Greg/clickhouse polishing #17483

Greg/clickhouse polishing #17483

grishick commented Sep 30, 2022 •

edited

Loading

github-actions bot commented Sep 30, 2022

grishick commented Sep 30, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Sep 30, 2022

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Oct 1, 2022

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Oct 1, 2022

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Oct 1, 2022

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

Greg/clickhouse polishing #17483

Greg/clickhouse polishing #17483

Conversation

grishick commented Sep 30, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

github-actions bot commented Sep 30, 2022

grishick commented Sep 30, 2022 • edited by github-actions bot Loading

Build Passed

github-actions bot commented Sep 30, 2022

grishick commented Oct 1, 2022 • edited by github-actions bot Loading

grishick commented Oct 1, 2022 • edited by github-actions bot Loading

grishick commented Oct 1, 2022 • edited by github-actions bot Loading

grishick commented Oct 1, 2022 • edited by github-actions bot Loading

Build Failed

grishick commented Oct 1, 2022 • edited by github-actions bot Loading

Build Failed

github-actions bot commented Oct 1, 2022

grishick commented Oct 1, 2022 • edited by github-actions bot Loading

Build Failed

github-actions bot commented Oct 1, 2022

grishick commented Oct 1, 2022 • edited by github-actions bot Loading

Build Passed

github-actions bot commented Oct 1, 2022

grishick commented Oct 1, 2022 • edited by github-actions bot Loading

grishick commented Sep 30, 2022 •

edited

Loading

grishick commented Sep 30, 2022 •

edited by github-actions bot

Loading

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading

grishick commented Oct 1, 2022 •

edited by github-actions bot

Loading