Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Staging destinations: Fail fast on error during periodic checkpoint #24671

Merged
merged 7 commits into from
Mar 29, 2023
Merged

Staging destinations: Fail fast on error during periodic checkpoint #24671

merged 7 commits into from
Mar 29, 2023

Conversation

edgao
Copy link
Contributor

@edgao edgao commented Mar 29, 2023

What

Don't swallow exception in periodic buffer flush - #23931 (comment)

How

just rethrow it

@ryankfu - which connectors do I need to publishd? I'm guessing snowflake/bigquery/redshift?

@edgao edgao requested a review from a team as a code owner March 29, 2023 18:48
@edgao
Copy link
Contributor Author

edgao commented Mar 29, 2023 •
edited by github-actions bot
Loading

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/4556859008
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/4556859008
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 15      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    18      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     171     10    94%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         195     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         65     39    40%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1461    632    57%

Build Passed

Test summary info:

All Passed

ryankfu
ryankfu approved these changes Mar 29, 2023
View reviewed changes
Copy link
Contributor

@ryankfu ryankfu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, publishing for Snowflake, Redshift, and BigQuery are the primary connectors

@github-actions
Copy link
Contributor

github-actions bot commented Mar 29, 2023 •
edited
Loading

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

  • Run integration tests
  • Bump connector or module version
  • Add changelog
  • Publish the new version

✅ Sources (31)
Connector Version Changelog Publish
source-alloydb 2.0.13 ✅ ✅
source-alloydb-strict-encrypt 2.0.13 🔵
(ignored)		 🔵
(ignored)
source-bigquery 0.2.3 ✅ ✅
source-clickhouse 0.1.17 ✅ ✅
source-clickhouse-strict-encrypt 0.1.17 🔵
(ignored)		 🔵
(ignored)
source-cockroachdb 0.1.22 ✅ ✅
source-cockroachdb-strict-encrypt 0.1.22 🔵
(ignored)		 🔵
(ignored)
source-db2 0.1.19 ✅ ✅
source-db2-strict-encrypt 0.1.19 🔵
(ignored)		 🔵
(ignored)
source-dynamodb 0.1.2 ✅ ✅
source-e2e-test 2.1.4 ✅ ✅
source-e2e-test-cloud 2.1.4 🔵
(ignored)		 🔵
(ignored)
source-elasticsearch 0.1.1 ✅ ✅
source-jdbc 0.3.5 🔵
(ignored)		 🔵
(ignored)
source-kafka 0.2.3 ✅ ✅
source-mongodb-strict-encrypt 0.1.19 🔵
(ignored)		 🔵
(ignored)
source-mongodb-v2 0.1.19 ✅ ✅
source-mssql 1.0.8 ✅ ✅
source-mssql-strict-encrypt 1.0.8 🔵
(ignored)		 🔵
(ignored)
source-mysql 2.0.11 ✅ ✅
source-mysql-strict-encrypt 2.0.11 🔵
(ignored)		 🔵
(ignored)
source-oracle 0.3.24 ✅ ✅
source-oracle-strict-encrypt 0.3.24 🔵
(ignored)		 🔵
(ignored)
source-postgres 2.0.13 ✅ ✅
source-postgres-strict-encrypt 2.0.13 🔵
(ignored)		 🔵
(ignored)
source-redshift 0.3.16 ✅ ✅
source-relational-db 0.3.1 🔵
(ignored)		 🔵
(ignored)
source-scaffold-java-jdbc 0.1.0 🔵
(ignored)		 🔵
(ignored)
source-sftp 0.1.2 ✅ ✅
source-snowflake 0.1.33 ✅ ✅
source-tidb 0.2.4 ✅ ✅
  • See "Actionable Items" below for how to resolve warnings and errors.

❌ Destinations (48)
Connector Version Changelog Publish
destination-azure-blob-storage 0.2.0 ✅ ✅
destination-bigquery 1.2.19 ✅ ✅
destination-bigquery-denormalized 1.2.19 ✅ ✅
destination-cassandra 0.1.4 ✅ ✅
destination-clickhouse 0.2.2 ✅ ✅
destination-clickhouse-strict-encrypt 0.2.2 🔵
(ignored)		 🔵
(ignored)
destination-csv 1.0.0 ✅ ✅
destination-databricks 1.0.0 ✅ ✅
destination-dev-null 0.2.7 🔵
(ignored)		 🔵
(ignored)
destination-doris 0.1.0 ✅ ✅
destination-dynamodb 0.1.7 ✅ ✅
destination-e2e-test 0.2.4 ✅ ✅
destination-elasticsearch 0.1.6 ✅ ✅
destination-elasticsearch-strict-encrypt 0.1.6 🔵
(ignored)		 🔵
(ignored)
destination-exasol 0.1.1 ✅ ✅
destination-gcs 0.2.16 ✅ ✅
destination-iceberg 0.1.0 ✅ ✅
destination-jdbc 0.3.14 🔵
(ignored)		 🔵
(ignored)
destination-kafka 0.1.10 ✅ ✅
destination-keen 0.2.4 ✅ ✅
destination-kinesis 0.1.5 ✅ ✅
destination-local-json 0.2.11 ✅ ✅
destination-mariadb-columnstore 0.1.7 ✅ ✅
destination-mongodb 0.1.9 ✅ ✅
destination-mongodb-strict-encrypt 0.1.9 🔵
(ignored)		 🔵
(ignored)
destination-mqtt 0.1.3 ✅ ✅
destination-mssql 0.1.22 ✅ ✅
destination-mssql-strict-encrypt 0.1.22 🔵
(ignored)		 🔵
(ignored)
destination-mysql 0.1.20 ✅ ✅
destination-mysql-strict-encrypt ❌ 0.1.21
(mismatch: 0.1.20)		 🔵
(ignored)		 🔵
(ignored)
destination-oracle 0.1.19 ✅ ✅
destination-oracle-strict-encrypt 0.1.19 🔵
(ignored)		 🔵
(ignored)
destination-postgres 0.3.26 ✅ ✅
destination-postgres-strict-encrypt 0.3.26 🔵
(ignored)		 🔵
(ignored)
destination-pubsub 0.2.0 ✅ ✅
destination-pulsar 0.1.3 ✅ ✅
destination-r2 0.1.0 ✅ ✅
destination-redis 0.1.4 ✅ ✅
destination-redpanda 0.1.0 ✅ ✅
destination-redshift 0.4.4 ✅ ✅
destination-rockset 0.1.4 ✅ ✅
destination-s3 0.3.22 ✅ ✅
destination-s3-glue 0.1.4 ✅ ✅
destination-scylla 0.1.3 ✅ ✅
destination-snowflake 0.4.59 ✅ ✅
destination-teradata 0.1.1 ✅ ✅
destination-tidb 0.1.0 ✅ ✅
destination-yugabytedb 0.1.1 ✅ ✅
  • See "Actionable Items" below for how to resolve warnings and errors.

👀 Other Modules (1)
  • base-normalization

Actionable Items

(click to expand)
Category Status Actionable Item
Version ❌
mismatch		 The version of the connector is different from its normal variant. Please bump the version of the connector.
⚠
doc not found		 The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
Changelog ⚠
doc not found		 The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
❌
changelog missing		 There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish ⚠
not in seed		 The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.
❌
diff seed version		 The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version.

@edgao edgao requested a review from a team as a code owner March 29, 2023 19:45
@edgao edgao changed the title rethrow exception for fail fast Staging destinations: Fail fast on error during periodic checkpoint Mar 29, 2023
@edgao
Copy link
Contributor Author

edgao commented Mar 29, 2023 •
edited by github-actions bot
Loading

/publish parallel=true connector=connectors/destination-bigquery,connectors/destination-bigquery-denormalized,connectors/destination-snowflake,connectors/destination-redshift

🕑 Publishing the following connectors:
connectors/destination-bigquery,connectors/destination-bigquery-denormalized,connectors/destination-snowflake,connectors/destination-redshift
https://github.com/airbytehq/airbyte/actions/runs/4557340239


Connector Did it publish? Were definitions generated?
connectors/destination-redshift ✅ ✅
connectors/destination-bigquery ✅ ❌
connectors/destination-bigquery-denormalized ✅ ❌
connectors/destination-snowflake ✅ ❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

@edgao edgao enabled auto-merge (squash) March 29, 2023 23:24
@edgao edgao merged commit 0586006 into master Mar 29, 2023
@edgao edgao deleted the edgao/periodic_flush_fail_fast branch March 29, 2023 23:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ryankfu ryankfu ryankfu approved these changes

Assignees
No one assigned
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/bigquery connectors/destination/bigquery-denormalized connectors/destination/redshift connectors/destination/snowflake
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@edgao @ryankfu @octavia-squidington-iii