Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

test for behavior when a column is removed in an incremental sync #20033

Merged
merged 10 commits into from
Jan 9, 2023

Conversation

davinchia
Copy link
Contributor

@davinchia davinchia commented Dec 2, 2022

What

Adds a destination acceptance test for removing a column in an incremental sync.

The goal is to verify that it's safe to go ahead with column selection, particularly for some concerning destinations.

@github-actions
Copy link
Contributor

github-actions bot commented Dec 2, 2022

Affected Connector Report

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to do the following as needed:

  • Run integration tests
  • Bump connector or module version
  • Add changelog
  • Publish the new version

✅ Sources (0)

Connector Version Changelog Publish
  • See "Actionable Items" below for how to resolve warnings and errors.

❌ Destinations (47)

Connector Version Changelog Publish
destination-aws-datalake 0.1.1
destination-azure-blob-storage 0.1.6
destination-bigquery 1.2.9
destination-bigquery-denormalized 1.2.10
destination-cassandra 0.1.4
destination-clickhouse 0.2.1
destination-clickhouse-strict-encrypt 0.2.1 🔵
(ignored)
🔵
(ignored)
destination-csv 0.2.10
destination-databricks 0.3.1
destination-dev-null 0.2.7 🔵
(ignored)
🔵
(ignored)
destination-doris 0.1.0
destination-dynamodb 0.1.7
destination-e2e-test 0.2.4
destination-elasticsearch 0.1.6
destination-elasticsearch-strict-encrypt 0.1.6 🔵
(ignored)
🔵
(ignored)
destination-gcs 0.2.12
destination-iceberg 0.1.0
destination-jdbc 0.3.14 🔵
(ignored)
🔵
(ignored)
destination-kafka 0.1.10
destination-keen 0.2.4
destination-kinesis 0.1.5
destination-local-json 0.2.11
destination-mariadb-columnstore 0.1.7
destination-mongodb 0.1.9
destination-mongodb-strict-encrypt 0.1.9 🔵
(ignored)
🔵
(ignored)
destination-mqtt 0.1.3
destination-mssql 0.1.22
destination-mssql-strict-encrypt 0.1.22 🔵
(ignored)
🔵
(ignored)
destination-mysql 0.1.20
destination-mysql-strict-encrypt 0.1.21
(mismatch: 0.1.20)
🔵
(ignored)
🔵
(ignored)
destination-oracle 0.1.19
destination-oracle-strict-encrypt 0.1.19 🔵
(ignored)
🔵
(ignored)
destination-postgres 0.3.26
destination-postgres-strict-encrypt 0.3.26 🔵
(ignored)
🔵
(ignored)
destination-pubsub 0.2.0
destination-pulsar 0.1.3
destination-r2 0.1.0
destination-redis 0.1.4
destination-redpanda 0.1.0
destination-redshift 0.3.53
destination-rockset 0.1.4
destination-s3 0.3.18
destination-s3-glue 0.1.1
destination-scylla 0.1.3
destination-snowflake 0.4.41
destination-tidb 0.1.0
destination-yugabytedb 0.1.0
  • See "Actionable Items" below for how to resolve warnings and errors.

✅ Other Modules (0)

Actionable Items

(click to expand)

Category Status Actionable Item
Version
mismatch
The version of the connector is different from its normal variant. Please bump the version of the connector.

doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.
Changelog
doc not found
The connector does not seem to have a documentation file. This can be normal (e.g. basic connector like source-jdbc is not published or documented). Please double-check to make sure that it is not a bug.

changelog missing
There is no chnagelog for the current version of the connector. If you are the author of the current version, please add a changelog.
Publish
not in seed
The connector is not in the seed file (e.g. source_definitions.yaml), so its publication status cannot be checked. This can be normal (e.g. some connectors are cloud-specific, and only listed in the cloud seed file). Please double-check to make sure that it is not a bug.

diff seed version
The connector exists in the seed file, but the latest version is not listed there. This usually means that the latest version is not published. Please use the /publish command to publish the latest version.

// remove one field
final JsonNode jsonSchema = configuredCatalog.getStreams().get(0).getStream().getJsonSchema();
((ObjectNode) jsonSchema.findValue("properties")).remove("HKD");
configuredCatalog.getStreams().get(0).getStream().setJsonSchema(jsonSchema);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: this isn't technically necessary, since you're modifying the ObjectNode in-place

@@ -561,6 +564,7 @@ public void specDBTValueShouldBeCorrect() throws WorkerException {
* append records to the datastore instead of overwriting the previous run.
*/
@Test
@Disabled
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a note - don't forget that these need to be un-disabled!

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 2, 2022

/test connector=destination-snowflake

🕑 destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/3605685638
❌ destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/3605685638
🐛 https://gradle.com/s/aiecr3adfc7sy

Build Failed

Test summary info:

Could not find result summary

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 2, 2022

/test connector=destination-bigquery

🕑 destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3605694006
❌ destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3605694006
🐛 https://gradle.com/s/cltgpuwu766w6

Build Failed

Test summary info:

Could not find result summary

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 2, 2022

/test connector=destination-bigquery-denormalized

🕑 destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/3605694609
✅ destination-bigquery-denormalized https://github.com/airbytehq/airbyte/actions/runs/3605694609
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 2, 2022

/test connector=destination-aws-datalake

🕑 destination-aws-datalake https://github.com/airbytehq/airbyte/actions/runs/3605694956
❌ destination-aws-datalake https://github.com/airbytehq/airbyte/actions/runs/3605694956
🐛 https://gradle.com/s/zowvl2unpraba

Build Failed

Test summary info:

Could not find result summary

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 2, 2022

/test connector=destination-databricks

🕑 destination-databricks https://github.com/airbytehq/airbyte/actions/runs/3605696985
✅ destination-databricks https://github.com/airbytehq/airbyte/actions/runs/3605696985
No Python unittests run

Build Passed

Test summary info:

All Passed

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 2, 2022

/test connector=destination-s3

🕑 destination-s3 https://github.com/airbytehq/airbyte/actions/runs/3605697634
✅ destination-s3 https://github.com/airbytehq/airbyte/actions/runs/3605697634
No Python unittests run

Build Passed

Test summary info:

All Passed

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 2, 2022

/test connector=destination-postgres

🕑 destination-postgres https://github.com/airbytehq/airbyte/actions/runs/3605698611
❌ destination-postgres https://github.com/airbytehq/airbyte/actions/runs/3605698611
🐛 https://gradle.com/s/ltyj5sz4m56fe

Build Failed

Test summary info:

Could not find result summary

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 9, 2022

/test connector=destination-postgres

🕑 destination-postgres https://github.com/airbytehq/airbyte/actions/runs/3657821105
❌ destination-postgres https://github.com/airbytehq/airbyte/actions/runs/3657821105
🐛

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 9, 2022

/test connector=destination-aws-datalake

🕑 destination-aws-datalake https://github.com/airbytehq/airbyte/actions/runs/3657823051
❌ destination-aws-datalake https://github.com/airbytehq/airbyte/actions/runs/3657823051
🐛

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 9, 2022

/test connector=destination-bigquery

🕑 destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3657825104
❌ destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3657825104
🐛

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 9, 2022

/test connector=destination-snowflake

🕑 destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/3657830259
❌ destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/3657830259
🐛

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 11, 2022

/test connector=destination-postgres

🕑 destination-postgres https://github.com/airbytehq/airbyte/actions/runs/3670842351
❌ destination-postgres https://github.com/airbytehq/airbyte/actions/runs/3670842351
🐛 https://gradle.com/s/zcxshcfw7kn6u

Build Failed

Test summary info:

Could not find result summary

🕑 destination-postgres https://github.com/airbytehq/airbyte/actions/runs/3670842351
❌ destination-postgres https://github.com/airbytehq/airbyte/actions/runs/3670842351
🐛 https://gradle.com/s/jimvfusaunafe

Build Failed

Test summary info:

Could not find result summary

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 11, 2022

/test connector=destination-aws-datalake

🕑 destination-aws-datalake https://github.com/airbytehq/airbyte/actions/runs/3670843500
❌ destination-aws-datalake https://github.com/airbytehq/airbyte/actions/runs/3670843500
🐛 https://gradle.com/s/qkzukoqe6lo7e

Build Failed

Test summary info:

Could not find result summary

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 11, 2022

/test connector=destination-bigquery

🕑 destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3670844386
❌ destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3670844386
🐛 https://gradle.com/s/4wjj5lhmzbi5q

Build Failed

Test summary info:

Could not find result summary

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 11, 2022

/test connector=destination-snowflake

🕑 destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/3670845552
❌ destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/3670845552
🐛 https://gradle.com/s/s67xpif6522m6

Build Failed

Test summary info:

Could not find result summary

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 12, 2022

/test connector=destination-postgres

🕑 destination-postgres https://github.com/airbytehq/airbyte/actions/runs/3677761129
❌ destination-postgres https://github.com/airbytehq/airbyte/actions/runs/3677761129
🐛 https://gradle.com/s/vatnil6bqhax2

Build Failed

Test summary info:

Could not find result summary

@mfsiega-airbyte
Copy link
Contributor

/test connector=destination-postgres

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 12, 2022

/test connector=destination-postgres

🕑 destination-postgres https://github.com/airbytehq/airbyte/actions/runs/3678466374
✅ destination-postgres https://github.com/airbytehq/airbyte/actions/runs/3678466374
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 12, 2022

/test connector=destination-snowflake

🕑 destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/3678746207
✅ destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/3678746207
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 12, 2022

/test connector=destination-bigquery

🕑 destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3678747092
✅ destination-bigquery https://github.com/airbytehq/airbyte/actions/runs/3678747092
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 14      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    14      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     166      8    95%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_config/transform.py                         189     48    75%
normalization/transform_catalog/utils.py                             51     14    73%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 595    400    33%
-------------------------------------------------------------------------------------
TOTAL                                                              1441    629    56%

Build Passed

Test summary info:

All Passed

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 12, 2022

/test connector=destination-aws-datalake

🕑 destination-aws-datalake https://github.com/airbytehq/airbyte/actions/runs/3678749453
❌ destination-aws-datalake https://github.com/airbytehq/airbyte/actions/runs/3678749453
🐛 https://gradle.com/s/sinopocsi4fli

Build Failed

Test summary info:

Could not find result summary

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 12, 2022

/test connector=destination-aws-datalake

🕑 destination-aws-datalake https://github.com/airbytehq/airbyte/actions/runs/3679047759
❌ destination-aws-datalake https://github.com/airbytehq/airbyte/actions/runs/3679047759
🐛 https://gradle.com/s/j6gkqfcwin6bk

Build Failed

Test summary info:

Could not find result summary

@mfsiega-airbyte
Copy link
Contributor

mfsiega-airbyte commented Dec 12, 2022

/test connector=destination-aws-datalake

🕑 destination-aws-datalake https://github.com/airbytehq/airbyte/actions/runs/3679309487
❌ destination-aws-datalake https://github.com/airbytehq/airbyte/actions/runs/3679309487
🐛 https://gradle.com/s/4elhomwp6xy62

Build Failed

Test summary info:

Could not find result summary

@mfsiega-airbyte
Copy link
Contributor

Quick summary: passing tests for:

  1. BigQuery
  2. Snowflake
  3. Postgres
  4. BigQuery Denormalized
  5. Databricks
  6. S3

No green test for AWS Datalake but from the test run, the actual new test case is passing and it appears that the failure is an unrelated test dating back to July, so I think we can ignore that one.

@mfsiega-airbyte mfsiega-airbyte marked this pull request as ready for review December 13, 2022 01:10
Copy link
Contributor

@edgao edgao left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks good! one comment about simplifying the test structure but :shipit: once that's resolved

final AirbyteCatalog catalog = Jsons.deserialize(MoreResources.readResource(catalogFilename),
AirbyteCatalog.class);

if (!catalog.getStreams().get(0).getName().equals("exchange_rate")) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rather than using @ParameterizedTest, you can just reference DataArgumentsProvider.EXCHANGE_RATE_CONFIG directly

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mfsiega-airbyte what about this comment?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davinchia me wanting to but not getting a chance to spend the time to address this comment is why this PR isn't already merged :) it's not a trivial context switch, so I haven't found time.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this entire if statement can be removed now?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'll send a follow-up with that cleanup.

@davinchia davinchia closed this Dec 28, 2022
@davinchia davinchia reopened this Dec 28, 2022
@mfsiega-airbyte mfsiega-airbyte merged commit cae0e3a into master Jan 9, 2023
@mfsiega-airbyte mfsiega-airbyte deleted the msiega/dat-edit-tests branch January 9, 2023 20:49
@@ -1462,7 +1529,6 @@ public String toString() {
* your_containers_id" (ex. docker container attach 18cc929f44c8) to see the container's output
*/
@Test
@Disabled
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Enabling this test may break some connectors and will keep lots of trash in DB. Basically, it wasn't even a test, but just a script for manual load emulation.

jbfbell pushed a commit that referenced this pull request Jan 13, 2023
…0033)

* test for behavior when a column is removed in an incremental sync

* fixes in dat test for dropping a column

* only run drop-one-column test for the exchange rates dataset

* re-enable tests that were disabled during development

* remove unused import

* update test to new method for checking spec capabilities

* use config directly instead of parameterized test

Co-authored-by: Michael Siega <michael@airbyte.io>
Co-authored-by: Michael Siega <109092231+mfsiega-airbyte@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants