✨ Destination BigQuery - Add v1v2 Migration #28962

jbfbell · 2023-08-02T00:07:00Z

Add Destination V1 -> V2 Migration

Start with the DestinationV1V2Migrator this interface contains all of the logic to determine whether a migration is needed.

Next, in the SqlGenerator interface, there is a new method to generate the SQL required to perform the migration.

The rest is either testing or shimming into the appropriate place.

The shim is in the TyperDeduper because this class calls a prepareFinalTables method prior to any other operations, and the migrator must be run first.

… work

github-actions · 2023-08-02T00:07:25Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

octavia-squidington-iii · 2023-08-02T02:48:20Z

destination-redshift test report (commit `6ac8c469b7`) - ❌

⏲️ Total pipeline duration: 27mn16s

Step	Result
Validate airbyte-integrations/connectors/destination-redshift/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build destination-redshift docker image for platform linux/x86_64	✅
Build airbyte/normalization-redshift:dev	✅
./gradlew :airbyte-integrations:connectors:destination-redshift:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-redshift test

octavia-squidington-iii · 2023-08-02T02:59:31Z

destination-dynamodb test report (commit `6ac8c469b7`) - ❌

⏲️ Total pipeline duration: 10mn57s

Step	Result
Validate airbyte-integrations/connectors/destination-dynamodb/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build destination-dynamodb docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-dynamodb:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-dynamodb test

octavia-squidington-iii · 2023-08-02T03:43:36Z

destination-databricks test report (commit `6ac8c469b7`) - ❌

⏲️ Total pipeline duration: 43mn51s

Step	Result
Validate airbyte-integrations/connectors/destination-databricks/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build destination-databricks docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-databricks:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-databricks test

octavia-squidington-iii · 2023-08-02T03:56:48Z

destination-gcs test report (commit `6ac8c469b7`) - ❌

⏲️ Total pipeline duration: 12mn58s

Step	Result
Validate airbyte-integrations/connectors/destination-gcs/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build destination-gcs docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:destination-gcs:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-gcs test

octavia-squidington-iii · 2023-08-02T09:03:24Z

destination-oracle test report (commit `6ac8c469b7`) - ❌

⏲️ Total pipeline duration: 306mn20s

Step	Result
Validate airbyte-integrations/connectors/destination-oracle/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build destination-oracle docker image for platform linux/x86_64	✅
Build airbyte/normalization-oracle:dev	✅
./gradlew :airbyte-integrations:connectors:destination-oracle:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-oracle test

octavia-squidington-iii · 2023-08-05T01:27:51Z

destination-bigquery test report (commit `3c38e432fa`) - ✅

⏲️ Total pipeline duration: 32mn53s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

octavia-squidington-iii · 2023-08-05T01:31:27Z

destination-snowflake test report (commit `3c38e432fa`) - ❌

⏲️ Total pipeline duration: 03mn27s

Step	Result
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Build connector tar	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

octavia-squidington-iii · 2023-08-05T01:37:19Z

destination-snowflake test report (commit `c8a2025efa`) - ❌

⏲️ Total pipeline duration: 03mn51s

Step	Result
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Build connector tar	❌

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

octavia-squidington-iii · 2023-08-05T01:41:35Z

destination-bigquery test report (commit `c8a2025efa`) - ✅

⏲️ Total pipeline duration: 03mn56s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

...o/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.java

edgao · 2023-08-07T15:17:12Z

snowflake compile error:

SnowflakeSqlGenerator is not abstract and does not override abstract method migrateFromV1toV2(StreamId,String,String) in SqlGenerator
public class SnowflakeSqlGenerator implements SqlGenerator {
       ^

I guess just add an empty method + tag the snowflake sqlgenerator test class with @Disabled to keep the build green and I can fill this in?

…java/io/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.java Co-authored-by: Edward Gao <edward.gao@airbyte.io>

octavia-squidington-iii · 2023-08-07T20:57:31Z

destination-snowflake test report (commit `df2bb3fc4d`) - ❌

⏲️ Total pipeline duration: 44mn14s

Step	Result
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Build airbyte/normalization-snowflake:dev	✅
./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

octavia-squidington-iii · 2023-08-07T21:37:17Z

destination-bigquery test report (commit `df2bb3fc4d`) - ✅

⏲️ Total pipeline duration: 39mn35s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

octavia-squidington-iii · 2023-08-08T00:15:08Z

destination-snowflake test report (commit `c78ff467e2`) - ✅

⏲️ Total pipeline duration: 44mn04s

Step	Result
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Build airbyte/normalization-snowflake:dev	✅
./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

octavia-squidington-iii · 2023-08-08T00:48:00Z

destination-bigquery test report (commit `c78ff467e2`) - ✅

⏲️ Total pipeline duration: 32mn42s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

octavia-squidington-iii · 2023-08-08T16:47:18Z

destination-snowflake test report (commit `684dab5c7e`) - ✅

⏲️ Total pipeline duration: 47mn10s

Step	Result
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Build airbyte/normalization-snowflake:dev	✅
./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

octavia-squidington-iii · 2023-08-08T17:18:40Z

destination-bigquery test report (commit `684dab5c7e`) - ✅

⏲️ Total pipeline duration: 31mn11s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

octavia-squidington-iii · 2023-08-08T18:05:59Z

destination-bigquery test report (commit `4ca01fc7c5`) - ✅

⏲️ Total pipeline duration: 32mn01s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

octavia-squidington-iii · 2023-08-08T18:49:39Z

destination-snowflake test report (commit `4ca01fc7c5`) - ✅

⏲️ Total pipeline duration: 43mn29s

Step	Result
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Build airbyte/normalization-snowflake:dev	✅
./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

octavia-squidington-iii · 2023-08-09T19:30:57Z

destination-bigquery test report (commit `82061fba4d`) - ✅

⏲️ Total pipeline duration: 31mn51s

Step	Result
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Build airbyte/normalization:dev	✅
./gradlew :airbyte-integrations:connectors:destination-bigquery:integrationTest	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

octavia-squidington-iii · 2023-08-09T20:12:08Z

destination-snowflake test report (commit `82061fba4d`) - ✅

⏲️ Total pipeline duration: 41mn00s

Step	Result
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	✅
QA checks	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Build airbyte/normalization-snowflake:dev	✅
./gradlew :airbyte-integrations:connectors:destination-snowflake:integrationTest	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

* Add everything for BQ but migrate, refactor interface after practical work * Make new default methods, refactor to single implemented method * MigrationInterface and BQ impl created * Trying to integrate with standard inserts * remove unnecessary NameAndNamespacePair class * Shimmed in * Java Docs * Initial Testing Setup * Tests! * Move Migrator into TyperDeduper * Functional Migration * Add Integration Test * Pr updates * bump version * bump version * version bump * Update to airbyte-ci-internal (airbytehq#29026) * 🐛 Source Github, Instagram, Zendesk-support, Zendesk-talk: fix CAT tests fail on `spec` (airbytehq#28910) * connectors-ci: better modified connectors detection logic (airbytehq#28855) * connectors-ci: report path should always start with `airbyte-ci/` (airbytehq#29030) * make report path always start with airbyte-ci * revert report path in orchestrator * add more test cases * bump version * Updated docs (airbytehq#29019) * CDK: Embedded reader utils (airbytehq#28873) * relax pydantic dep * Automated Commit - Format and Process Resources Changes * wip * wrap up base integration * add init file * introduce CDK runner and improve error message * make state param optional * update protocol models * review comments * always run incremental if possible * fix --------- Co-authored-by: flash1293 <flash1293@users.noreply.github.com> * 🤖 Bump minor version of Airbyte CDK * 🚨🚨 Low code CDK: Decouple SimpleRetriever and HttpStream (airbytehq#28657) * fix tests * format * review comments * Automated Commit - Formatting Changes * review comments * review comments * review comments * log all messages * log all message * review comments * review comments * Automated Commit - Formatting Changes * add comment --------- Co-authored-by: flash1293 <flash1293@users.noreply.github.com> * 🤖 Bump minor version of Airbyte CDK * 🐛 Source Github, Instagram, Zendesk Support / Talk - revert `spec` changes and improve (airbytehq#29031) * Source oauth0: new streams and fix incremental (airbytehq#29001) * Add new streams Organizations,OrganizationMembers,OrganizationMemberRoles * relax schema definition to allow additional fields * Bump image tag version * revert some changes to the old schemas * Format python so gradle can pass * update incremental * remove unused print * fix unit test --------- Co-authored-by: Vasilis Gavriilidis <vasilis.gavriilidis@orfium.com> * 🐛 Source Mongo: Fix failing acceptance tests (airbytehq#28816) * Fix failing acceptance tests * Fix failing strict acceptance tests * Source-Greenhouse: Fix unit tests for new CDK version (airbytehq#28969) Fix unit tests * Add CSV options to the CSV parser (airbytehq#28491) * remove invalid legacy option * remove unused option * the tests pass but this is quite messy * very slight clean up * Add skip options to csv format * fix some of the typing issues * fixme comment * remove extra log message * fix typing issues * skip before header * skip after header * format * add another test * Automated Commit - Formatting Changes * auto generate column names * delete dead code * update title and description * true and false values * Update the tests * Add comment * missing test * rename * update expected spec * move to method * Update comment * fix typo * remove unused import * Add a comment * None records do not pass the WaitForDiscoverPolicy * format * remove second branch to ensure we always go through the same processing * Raise an exception if the record is None * reset * Update tests * handle unquoted newlines * Automated Commit - Formatting Changes * Update test case so the quoting is explicit * Update comment * Automated Commit - Formatting Changes * Fail validation if skipping rows before header and header is autogenerated * always fail if a record cannot be parsed * format * set write line_no in error message * remove none check * Automated Commit - Formatting Changes * enable autogenerate test * remove duplicate test * missing unit tests * Update * remove branching * remove unused none check * Update tests * remove branching * format * extract to function * comment * missing type * type annotation * use set * Document that the strings are case-sensitive * public -> private * add unit test * newline --------- Co-authored-by: girarda <girarda@users.noreply.github.com> * Dagster: Add sentry logging (airbytehq#28822) * Add sentry * add sentry decorator * Add traces * Use sentry trace * Improve duplicate logging * Add comments * DNC * Fix up issues * Move to scopes * Remove breadcrumb * Update lock * ✨Source Shortio: Migrate Python CDK to Low-code CDK (airbytehq#28950) * Migrate Shortio to Low-Code * Update abnormal state * Format * Update Docs * Fix metadata.yaml * Add pagination * Add incremental sync * add incremental parameters * update metadata * rollback update version * release date --------- Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> * Update to new verbiage (airbytehq#29051) * [skip ci] Metadata: Remove leading underscore (airbytehq#29024) * DNC * Add test models * Add model test * Remove underscore from metadata files * Regenerate models * Add test to check for key transformation * Allow additional fields on metadata * Delete transform * Proof of concept parallel source stream reading implementation for MySQL (airbytehq#26580) * Proof of concept parallel source stream reading implementation for MySQL * Automated Change * Add read method that supports concurrent execution to Source interface * Remove parallel iterator * Ensure that executor service is stopped * Automated Commit - Format and Process Resources Changes * Expose method to fix compilation issue * Use concurrent map to avoid access issues * Automated Commit - Format and Process Resources Changes * Ensure concurrent streams finish before closing source * Fix compile issue * Formatting * Exclude concurrent stream threads from orphan thread watcher * Automated Commit - Format and Process Resources Changes * Refactor orphaned thread logic to account for concurrent execution * PR feedback * Implement readStreams in wrapper source * Automated Commit - Format and Process Resources Changes * Add readStream override * Automated Commit - Format and Process Resources Changes * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * Debug logging * Reduce logging level * Replace synchronized calls to System.out.println when concurrent * Close consumer * Flush before close * Automated Commit - Format and Process Resources Changes * Remove charset * Use ASCII and flush periodically for parallel streams * Test performance harness patch * Automated Commit - Format and Process Resources Changes * Cleanup * Logging to identify concurrent read enabled * Mark parameter as final --------- Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com> Co-authored-by: rodireich <rodireich@users.noreply.github.com> * connectors-ci: disable dependency scanning (airbytehq#29033) * updates (airbytehq#29059) * Metadata: skip breaking change validation on prerelease (airbytehq#29017) * skip breaking change validation * Move ValidatorOpts higher in call * Add prerelease test * Fix test * ✨ Source MongoDB Internal POC: Generate Test Data (airbytehq#29049) * Add script to generate test data * Fix prose * Update credentials example * PR feedback * Bump Airbyte version from 0.50.12 to 0.50.13 * Bump versions for mssql strict-encrypt (airbytehq#28964) * Bump versions for mssql strict-encrypt * Fix failing test * Fix failing test * 🎨 Improve replication method selection UX (airbytehq#28882) * update replication method in MySQL source * bump version * update expected specs * update registries * bump strict encrypt version * make password always_show * change url * update registries * 🐛 Avoid writing records to log (airbytehq#29047) * Avoid writing records to log * Update version * Rollout ctid cdc (airbytehq#28708) * source-postgres: enable ctid+cdc implementation * 100% ctid rollout for cdc * remove CtidFeatureFlags * fix CdcPostgresSourceAcceptanceTest * Bump versions and release notes * Fix compilation error due to previous merge --------- Co-authored-by: subodh <subodh1810@gmail.com> * connectors-ci: fix `unhashable type 'set'` (airbytehq#29064) * Add Slack Alert lifecycle to Dagster for Metadata publish (airbytehq#28759) * DNC * Add slack lifecycle logging * Update to use slack * Update slack to use resource and bot * Improve markdown * Improve log * Add sensor logging * Extend sensor time * merge conflict * PR Refactoring * Make the tests work * remove unnecessary classes, pr feedback * more merging * Update airbyte-integrations/bases/base-typing-deduping-test/src/main/java/io/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.java Co-authored-by: Edward Gao <edward.gao@airbyte.io> * snowflake updates --------- Co-authored-by: Ben Church <ben@airbyte.io> Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com> Co-authored-by: Augustin <augustin@airbyte.io> Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com> Co-authored-by: Joe Reuter <joe@airbyte.io> Co-authored-by: flash1293 <flash1293@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Vasilis Gavriilidis <vasilis.gavriilidis@orfium.com> Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io> Co-authored-by: Alexandre Girard <alexandre@airbyte.io> Co-authored-by: girarda <girarda@users.noreply.github.com> Co-authored-by: btkcodedev <btk.codedev@gmail.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com> Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com> Co-authored-by: rodireich <rodireich@users.noreply.github.com> Co-authored-by: Alexandre Cuoci <Hesperide@users.noreply.github.com> Co-authored-by: terencecho <terencecho@users.noreply.github.com> Co-authored-by: Lake Mossman <lake@airbyte.io> Co-authored-by: Benoit Moriceau <benoit@airbyte.io> Co-authored-by: subodh <subodh1810@gmail.com> Co-authored-by: Edward Gao <edward.gao@airbyte.io>

jbfbell added 13 commits July 24, 2023 16:36

Add everything for BQ but migrate, refactor interface after practical…

f65f92b

… work

Make new default methods, refactor to single implemented method

3542aac

MigrationInterface and BQ impl created

d34e755

Trying to integrate with standard inserts

ab19eaa

remove unnecessary NameAndNamespacePair class

e49ef2a

Shimmed in

1edc723

Java Docs

347c647

Initial Testing Setup

81ad6b9

Tests!

c525728

Merge branch 'master' into joseph.bell/28065/dest-v2-bq-migration

fdfc952

Move Migrator into TyperDeduper

864e99a

Functional Migration

312c4af

Add Integration Test

41b6f40

octavia-squidington-iii added area/connectors Connector related issues connectors/destination/bigquery connectors/destination/snowflake labels Aug 2, 2023

jbfbell added 2 commits August 1, 2023 17:28

Pr updates

41057bf

Merge branch 'master' into joseph.bell/28065/dest-v2-bq-migration

9a91dea

jbfbell changed the title ~~Destination BigQuery - Add v1v2 Migration~~ ✨ Destination BigQuery - Add v1v2 Migration Aug 2, 2023

jbfbell added 2 commits August 1, 2023 19:18

bump version

bcac78e

Merge branch 'master' into joseph.bell/28065/dest-v2-bq-migration

6ac8c46

octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Aug 2, 2023

jbfbell marked this pull request as ready for review August 2, 2023 02:19

jbfbell requested a review from a team as a code owner August 2, 2023 02:19

edgao approved these changes Aug 7, 2023

View reviewed changes

...o/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.java Outdated Show resolved Hide resolved

jbfbell and others added 3 commits August 7, 2023 11:59

Update airbyte-integrations/bases/base-typing-deduping-test/src/main/…

45b62fb

…java/io/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.java Co-authored-by: Edward Gao <edward.gao@airbyte.io>

merge conflicts

68188ed

snowflake updates

df2bb3f

jbfbell enabled auto-merge (squash) August 7, 2023 20:44

jbfbell added 2 commits August 7, 2023 16:26

Merge branch 'master' into joseph.bell/28065/dest-v2-bq-migration

05f03e8

bump snowflake version

c78ff46

Merge branch 'master' into joseph.bell/28065/dest-v2-bq-migration

684dab5

jbfbell and others added 3 commits August 8, 2023 10:30

fix test import

91b2e19

Merge branch 'master' into joseph.bell/28065/dest-v2-bq-migration

4ca01fc

Automated Commit - Format and Process Resources Changes

0a705c9

mergin

82061fb

jbfbell merged commit 6e9cdc8 into master Aug 9, 2023

jbfbell deleted the joseph.bell/28065/dest-v2-bq-migration branch August 9, 2023 20:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ Destination BigQuery - Add v1v2 Migration #28962

✨ Destination BigQuery - Add v1v2 Migration #28962

jbfbell commented Aug 2, 2023 •

edited

Loading

github-actions bot commented Aug 2, 2023 •

edited by jbfbell

Loading

octavia-squidington-iii commented Aug 2, 2023

octavia-squidington-iii commented Aug 2, 2023

octavia-squidington-iii commented Aug 2, 2023

octavia-squidington-iii commented Aug 2, 2023

octavia-squidington-iii commented Aug 2, 2023

octavia-squidington-iii commented Aug 5, 2023

octavia-squidington-iii commented Aug 5, 2023

octavia-squidington-iii commented Aug 5, 2023

octavia-squidington-iii commented Aug 5, 2023

edgao commented Aug 7, 2023 •

edited

Loading

octavia-squidington-iii commented Aug 7, 2023

octavia-squidington-iii commented Aug 7, 2023

octavia-squidington-iii commented Aug 8, 2023

octavia-squidington-iii commented Aug 8, 2023

octavia-squidington-iii commented Aug 8, 2023

octavia-squidington-iii commented Aug 8, 2023

octavia-squidington-iii commented Aug 8, 2023

octavia-squidington-iii commented Aug 8, 2023

octavia-squidington-iii commented Aug 9, 2023

octavia-squidington-iii commented Aug 9, 2023

✨ Destination BigQuery - Add v1v2 Migration #28962

✨ Destination BigQuery - Add v1v2 Migration #28962

Conversation

jbfbell commented Aug 2, 2023 • edited Loading

Add Destination V1 -> V2 Migration

github-actions bot commented Aug 2, 2023 • edited by jbfbell Loading

Before Merging a Connector Pull Request

octavia-squidington-iii commented Aug 2, 2023

destination-redshift test report (commit 6ac8c469b7) - ❌

octavia-squidington-iii commented Aug 2, 2023

destination-dynamodb test report (commit 6ac8c469b7) - ❌

octavia-squidington-iii commented Aug 2, 2023

destination-databricks test report (commit 6ac8c469b7) - ❌

octavia-squidington-iii commented Aug 2, 2023

destination-gcs test report (commit 6ac8c469b7) - ❌

octavia-squidington-iii commented Aug 2, 2023

destination-oracle test report (commit 6ac8c469b7) - ❌

octavia-squidington-iii commented Aug 5, 2023

destination-bigquery test report (commit 3c38e432fa) - ✅

octavia-squidington-iii commented Aug 5, 2023

destination-snowflake test report (commit 3c38e432fa) - ❌

octavia-squidington-iii commented Aug 5, 2023

destination-snowflake test report (commit c8a2025efa) - ❌

octavia-squidington-iii commented Aug 5, 2023

destination-bigquery test report (commit c8a2025efa) - ✅

edgao commented Aug 7, 2023 • edited Loading

octavia-squidington-iii commented Aug 7, 2023

destination-snowflake test report (commit df2bb3fc4d) - ❌

octavia-squidington-iii commented Aug 7, 2023

destination-bigquery test report (commit df2bb3fc4d) - ✅

octavia-squidington-iii commented Aug 8, 2023

destination-snowflake test report (commit c78ff467e2) - ✅

octavia-squidington-iii commented Aug 8, 2023

destination-bigquery test report (commit c78ff467e2) - ✅

octavia-squidington-iii commented Aug 8, 2023

destination-snowflake test report (commit 684dab5c7e) - ✅

octavia-squidington-iii commented Aug 8, 2023

destination-bigquery test report (commit 684dab5c7e) - ✅

octavia-squidington-iii commented Aug 8, 2023

destination-bigquery test report (commit 4ca01fc7c5) - ✅

octavia-squidington-iii commented Aug 8, 2023

destination-snowflake test report (commit 4ca01fc7c5) - ✅

octavia-squidington-iii commented Aug 9, 2023

destination-bigquery test report (commit 82061fba4d) - ✅

octavia-squidington-iii commented Aug 9, 2023

destination-snowflake test report (commit 82061fba4d) - ✅

jbfbell commented Aug 2, 2023 •

edited

Loading

github-actions bot commented Aug 2, 2023 •

edited by jbfbell

Loading

destination-redshift test report (commit `6ac8c469b7`) - ❌

destination-dynamodb test report (commit `6ac8c469b7`) - ❌

destination-databricks test report (commit `6ac8c469b7`) - ❌

destination-gcs test report (commit `6ac8c469b7`) - ❌

destination-oracle test report (commit `6ac8c469b7`) - ❌

destination-bigquery test report (commit `3c38e432fa`) - ✅

destination-snowflake test report (commit `3c38e432fa`) - ❌

destination-snowflake test report (commit `c8a2025efa`) - ❌

destination-bigquery test report (commit `c8a2025efa`) - ✅

edgao commented Aug 7, 2023 •

edited

Loading

destination-snowflake test report (commit `df2bb3fc4d`) - ❌

destination-bigquery test report (commit `df2bb3fc4d`) - ✅

destination-snowflake test report (commit `c78ff467e2`) - ✅

destination-bigquery test report (commit `c78ff467e2`) - ✅

destination-snowflake test report (commit `684dab5c7e`) - ✅

destination-bigquery test report (commit `684dab5c7e`) - ✅

destination-bigquery test report (commit `4ca01fc7c5`) - ✅

destination-snowflake test report (commit `4ca01fc7c5`) - ✅

destination-bigquery test report (commit `82061fba4d`) - ✅

destination-snowflake test report (commit `82061fba4d`) - ✅