-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
DAT tearDown
should cleanup *all* schemas used
#29312
Conversation
Before Merging a Connector Pull RequestWow! What a great pull request you have here! 🎉 To merge this PR, ensure the following has been done/considered for each connector added or updated:
If the checklist is complete, but the CI check is failing,
|
destination-kafka test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-kafka/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-kafka docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-kafka:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-kafka test
destination-keen test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-keen/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-keen docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-keen:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-keen test
destination-tidb test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-tidb/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-tidb docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization-tidb:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-tidb:integrationTest | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-tidb test
destination-redpanda test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-redpanda/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-redpanda docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-redpanda:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-redpanda test
destination-gcs test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-gcs/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-gcs docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-gcs:integrationTest | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-gcs test
destination-snowflake test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-snowflake docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization-snowflake:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-snowflake test
destination-doris test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-doris/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ❌ |
Build connector tar | ✅ |
Build destination-doris docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-doris:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-doris test
destination-postgres-strict-encrypt test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-postgres-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-postgres-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-postgres-strict-encrypt:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-postgres-strict-encrypt test
destination-oracle test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-oracle/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-oracle docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization-oracle:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-oracle:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-oracle test
destination-kinesis test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-kinesis/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-kinesis docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-kinesis:integrationTest | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-kinesis test
destination-databricks test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-databricks/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-databricks docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-databricks:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-databricks test
destination-mongodb-strict-encrypt test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-mongodb-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-mongodb-strict-encrypt docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-mongodb-strict-encrypt:integrationTest | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mongodb-strict-encrypt test
destination-bigquery test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-bigquery docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-bigquery test
destination-teradata test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-teradata/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-teradata docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-teradata:integrationTest | ✅ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-teradata test
destination-e2e-test test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-e2e-test/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-e2e-test docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-e2e-test:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-e2e-test test
destination-mssql-strict-encrypt test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-mssql-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-mssql-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization-mssql:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-mssql-strict-encrypt:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mssql-strict-encrypt test
destination-oracle-strict-encrypt test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-oracle-strict-encrypt/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-oracle-strict-encrypt docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization-oracle:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-oracle-strict-encrypt:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-oracle-strict-encrypt test
destination-redshift test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-redshift docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization-redshift:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-redshift:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-redshift test
destination-starburst-galaxy test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-starburst-galaxy/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-starburst-galaxy docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-starburst-galaxy:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-starburst-galaxy test
destination-mssql test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-mssql/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-mssql docker image for platform linux/x86_64 | ✅ |
Build airbyte/normalization-mssql:dev | ✅ |
./gradlew :airbyte-integrations:connectors:destination-mssql:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mssql test
destination-rockset test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-rockset/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ❌ |
Build connector tar | ✅ |
Build destination-rockset docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-rockset:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-rockset test
destination-scylla test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-scylla/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-scylla docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-scylla:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-scylla test
There are too many connectors to test here. I'm merging this because:
|
destination-mariadb-columnstore test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-mariadb-columnstore/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ✅ |
Build connector tar | ✅ |
Build destination-mariadb-columnstore docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-mariadb-columnstore:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-mariadb-columnstore test
destination-iceberg test report (commit
|
Step | Result |
---|---|
Validate airbyte-integrations/connectors/destination-iceberg/metadata.yaml | ✅ |
Connector version semver check | ✅ |
QA checks | ❌ |
Build connector tar | ✅ |
Build destination-iceberg docker image for platform linux/x86_64 | ✅ |
./gradlew :airbyte-integrations:connectors:destination-iceberg:integrationTest | ❌ |
☁️ View runs for commit in Dagger Cloud
Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command
airbyte-ci connectors --name=destination-iceberg test
* DAT `tearDown` should cleanup *all* schemas used * redshift too * push all method signature changes * Automated Commit - Format and Process Resources Changes * use hashset * TEST_SCHEMAS on setup as well * fix call * Automated Commit - Format and Process Resources Changes --------- Co-authored-by: evantahler <evantahler@users.noreply.github.com>
This PR investigates why we have so many test schemas lingering in our databases. It looks like we create a few extra schemas in DAT so we can test writing to other namespaces. However, the test doesn't clean up after itself. To address this, we:
TEST_SCHEMAS
DestinationAcceptanceTest.tearDown
to now also pass TEST_SCHEMAS to implementors. DBs with persistent servers (Snowflake, BigQuery, Redshift) can now choose to delete all these schemas on teardown.Note: this is still brittle to the test suite crashing (we won't cleanup) but should be a lot cleaner than what we are doing now, which is nothing.
Reviewers:
Start with DestinationAcceptanceTest.java and then check out the teardown changes in RedshiftStagingS3DestinationAcceptanceTest.java and SnowflakeInsertDestinationAcceptanceTest.java. BigQuery DB tests seem ok because we create and then destroy a whole dataset each time, and all other changes are just the method signature.