🐛 Source Mongo: Fix failing acceptance tests #28816

jdpgrailsdev · 2023-07-27T20:00:35Z

What

Re-apply fixes to failing acceptance tests for the source-mongodb-v2 connector

How

Ensure that the configuration is properly set for Java-based acceptance tests

source-mongodb-v2 test report (commit `5bb3c2a52f`) - ❌

⏲️ Total pipeline duration: 16mn40s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-v2/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mongodb-v2 docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-v2:integrationTest	❌
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-v2 test

octavia-squidington-iii · 2023-07-27T20:39:26Z

source-mongodb-strict-encrypt test report (commit `5bb3c2a52f`) - ❌

⏲️ Total pipeline duration: 06mn19s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build source-mongodb-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-strict-encrypt:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-strict-encrypt test

octavia-squidington-iii · 2023-07-31T12:55:41Z

source-mongodb-strict-encrypt test report (commit `674c70630f`) - ❌

⏲️ Total pipeline duration: 06mn03s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build source-mongodb-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-strict-encrypt:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-strict-encrypt test

octavia-squidington-iii · 2023-07-31T13:12:37Z

source-mongodb-v2 test report (commit `674c70630f`) - ❌

⏲️ Total pipeline duration: 16mn43s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-v2/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mongodb-v2 docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-v2:integrationTest	❌
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-v2 test

octavia-squidington-iii · 2023-07-31T13:31:47Z

source-mongodb-strict-encrypt test report (commit `66b4a4abb1`) - ❌

⏲️ Total pipeline duration: 06mn45s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build source-mongodb-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-strict-encrypt:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-strict-encrypt test

octavia-squidington-iii · 2023-07-31T13:47:21Z

source-mongodb-v2 test report (commit `66b4a4abb1`) - ❌

⏲️ Total pipeline duration: 15mn21s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-v2/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mongodb-v2 docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-v2:integrationTest	❌
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-v2 test

jdpgrailsdev · 2023-07-31T14:10:18Z

/legacy-test connector=source-mongodb-v2

🕑 source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/5715747504
✅ source-mongodb-v2 https://github.com/airbytehq/airbyte/actions/runs/5715747504
No Python unittests run

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestIncremental.test_two_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:105: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:628: The previous and actual discovered catalogs are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:725: This tests currently leads to too much failures. We need to fix the connectors at scale first.
================== 35 passed, 4 skipped in 133.28s (0:02:13) ===================

jdpgrailsdev · 2023-07-31T14:22:28Z

The source-mongodb-v2 test failures are not reproducible locally. I have executed them both via the command line (./gradlew :airbyte-integrations:connectors:source-mongodb-v2:integrationTest) and via Intellij and they pass. I have also run them on this PR using the legacy runner and all of the tests pass, so I assume that there is an issue with the current test execution apparatus.

jdpgrailsdev · 2023-07-31T14:22:49Z

@colesnodgrass Could you please re-review, as I had to fix a few issues with the strict encrypt tests. Thanks!

octavia-squidington-iii · 2023-07-31T20:07:12Z

source-mongodb-strict-encrypt test report (commit `5e827dd583`) - ❌

⏲️ Total pipeline duration: 02mn22s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build source-mongodb-strict-encrypt docker image for platform linux/x86_64	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-strict-encrypt test

octavia-squidington-iii · 2023-07-31T20:09:44Z

source-mongodb-v2 test report (commit `5e827dd583`) - ❌

⏲️ Total pipeline duration: 02mn20s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-v2/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mongodb-v2 docker image for platform linux/x86_64	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-v2 test

octavia-squidington-iii · 2023-08-01T13:39:17Z

source-mongodb-strict-encrypt test report (commit `c0a6fec651`) - ❌

⏲️ Total pipeline duration: 06mn42s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build source-mongodb-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-strict-encrypt:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-strict-encrypt test

octavia-squidington-iii · 2023-08-01T13:51:30Z

source-mongodb-v2 test report (commit `c0a6fec651`) - ❌

⏲️ Total pipeline duration: 12mn03s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-v2/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mongodb-v2 docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-v2:integrationTest	❌
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-v2 test

octavia-squidington-iii · 2023-08-01T16:04:59Z

source-mongodb-strict-encrypt test report (commit `0ece0612f7`) - ❌

⏲️ Total pipeline duration: 02mn58s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build source-mongodb-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-strict-encrypt:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-strict-encrypt test

octavia-squidington-iii · 2023-08-01T16:14:27Z

source-mongodb-v2 test report (commit `0ece0612f7`) - ❌

⏲️ Total pipeline duration: 09mn14s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-v2/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mongodb-v2 docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-v2:integrationTest	❌
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-v2 test

jdpgrailsdev · 2023-08-02T13:10:52Z

@colesnodgrass @jpefaur Could I get a re-review of this PR so that we can close the door on the V2 work? The acceptance tests are failing via the new automatic execution runner on the PR, but they pass when I run them manually using the legacy slash command.

octavia-squidington-iii · 2023-08-02T13:20:18Z

source-mongodb-strict-encrypt test report (commit `da3a4061c7`) - ❌

⏲️ Total pipeline duration: 08mn07s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build source-mongodb-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-strict-encrypt:integrationTest	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-strict-encrypt test

octavia-squidington-iii · 2023-08-02T13:30:05Z

source-mongodb-v2 test report (commit `da3a4061c7`) - ❌

⏲️ Total pipeline duration: 09mn33s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-v2/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build source-mongodb-v2 docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-v2:integrationTest	❌
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-v2 test

octavia-squidington-iii · 2023-08-02T20:22:17Z

source-mongodb-strict-encrypt test report (commit `8ba431c10d`) - ❌

⏲️ Total pipeline duration: 05mn36s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	✅
Build source-mongodb-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-strict-encrypt:integrationTest	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-strict-encrypt test

octavia-squidington-iii · 2023-08-02T20:34:36Z

source-mongodb-v2 test report (commit `8ba431c10d`) - ❌

⏲️ Total pipeline duration: 12mn04s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-v2/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mongodb-v2 docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-v2:integrationTest	❌
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-v2 test

octavia-squidington-iii · 2023-08-02T20:51:57Z

source-mongodb-strict-encrypt test report (commit `8ba431c10d`) - ❌

⏲️ Total pipeline duration: 24.84s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	❌
Build connector tar	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-strict-encrypt test

octavia-squidington-iii · 2023-08-02T21:03:42Z

source-mongodb-v2 test report (commit `8ba431c10d`) - ❌

⏲️ Total pipeline duration: 11mn34s

Step	Result
Validate airbyte-integrations/connectors/source-mongodb-v2/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mongodb-v2 docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mongodb-v2:integrationTest	❌
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-v2 test

octavia-squidington-iii · 2023-08-03T14:10:35Z

source-mongodb-v2 test report (commit `48c6059aba`) - ❌

⏲️ Total pipeline duration: 19.71s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-v2 test

octavia-squidington-iii · 2023-08-03T14:10:49Z

source-mongodb-strict-encrypt test report (commit `48c6059aba`) - ❌

⏲️ Total pipeline duration: 4.50s

Step	Result

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mongodb-strict-encrypt test

jdpgrailsdev · 2023-08-03T14:13:02Z

/approve-and-merge reason="Fixes currently failing acceptance tests"

octavia-approvington · 2023-08-03T14:13:35Z

Send it!

* Fix failing acceptance tests * Fix failing strict acceptance tests

* Add everything for BQ but migrate, refactor interface after practical work * Make new default methods, refactor to single implemented method * MigrationInterface and BQ impl created * Trying to integrate with standard inserts * remove unnecessary NameAndNamespacePair class * Shimmed in * Java Docs * Initial Testing Setup * Tests! * Move Migrator into TyperDeduper * Functional Migration * Add Integration Test * Pr updates * bump version * bump version * version bump * Update to airbyte-ci-internal (#29026) * 🐛 Source Github, Instagram, Zendesk-support, Zendesk-talk: fix CAT tests fail on `spec` (#28910) * connectors-ci: better modified connectors detection logic (#28855) * connectors-ci: report path should always start with `airbyte-ci/` (#29030) * make report path always start with airbyte-ci * revert report path in orchestrator * add more test cases * bump version * Updated docs (#29019) * CDK: Embedded reader utils (#28873) * relax pydantic dep * Automated Commit - Format and Process Resources Changes * wip * wrap up base integration * add init file * introduce CDK runner and improve error message * make state param optional * update protocol models * review comments * always run incremental if possible * fix --------- Co-authored-by: flash1293 <flash1293@users.noreply.github.com> * 🤖 Bump minor version of Airbyte CDK * 🚨🚨 Low code CDK: Decouple SimpleRetriever and HttpStream (#28657) * fix tests * format * review comments * Automated Commit - Formatting Changes * review comments * review comments * review comments * log all messages * log all message * review comments * review comments * Automated Commit - Formatting Changes * add comment --------- Co-authored-by: flash1293 <flash1293@users.noreply.github.com> * 🤖 Bump minor version of Airbyte CDK * 🐛 Source Github, Instagram, Zendesk Support / Talk - revert `spec` changes and improve (#29031) * Source oauth0: new streams and fix incremental (#29001) * Add new streams Organizations,OrganizationMembers,OrganizationMemberRoles * relax schema definition to allow additional fields * Bump image tag version * revert some changes to the old schemas * Format python so gradle can pass * update incremental * remove unused print * fix unit test --------- Co-authored-by: Vasilis Gavriilidis <vasilis.gavriilidis@orfium.com> * 🐛 Source Mongo: Fix failing acceptance tests (#28816) * Fix failing acceptance tests * Fix failing strict acceptance tests * Source-Greenhouse: Fix unit tests for new CDK version (#28969) Fix unit tests * Add CSV options to the CSV parser (#28491) * remove invalid legacy option * remove unused option * the tests pass but this is quite messy * very slight clean up * Add skip options to csv format * fix some of the typing issues * fixme comment * remove extra log message * fix typing issues * skip before header * skip after header * format * add another test * Automated Commit - Formatting Changes * auto generate column names * delete dead code * update title and description * true and false values * Update the tests * Add comment * missing test * rename * update expected spec * move to method * Update comment * fix typo * remove unused import * Add a comment * None records do not pass the WaitForDiscoverPolicy * format * remove second branch to ensure we always go through the same processing * Raise an exception if the record is None * reset * Update tests * handle unquoted newlines * Automated Commit - Formatting Changes * Update test case so the quoting is explicit * Update comment * Automated Commit - Formatting Changes * Fail validation if skipping rows before header and header is autogenerated * always fail if a record cannot be parsed * format * set write line_no in error message * remove none check * Automated Commit - Formatting Changes * enable autogenerate test * remove duplicate test * missing unit tests * Update * remove branching * remove unused none check * Update tests * remove branching * format * extract to function * comment * missing type * type annotation * use set * Document that the strings are case-sensitive * public -> private * add unit test * newline --------- Co-authored-by: girarda <girarda@users.noreply.github.com> * Dagster: Add sentry logging (#28822) * Add sentry * add sentry decorator * Add traces * Use sentry trace * Improve duplicate logging * Add comments * DNC * Fix up issues * Move to scopes * Remove breadcrumb * Update lock * ✨Source Shortio: Migrate Python CDK to Low-code CDK (#28950) * Migrate Shortio to Low-Code * Update abnormal state * Format * Update Docs * Fix metadata.yaml * Add pagination * Add incremental sync * add incremental parameters * update metadata * rollback update version * release date --------- Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> * Update to new verbiage (#29051) * [skip ci] Metadata: Remove leading underscore (#29024) * DNC * Add test models * Add model test * Remove underscore from metadata files * Regenerate models * Add test to check for key transformation * Allow additional fields on metadata * Delete transform * Proof of concept parallel source stream reading implementation for MySQL (#26580) * Proof of concept parallel source stream reading implementation for MySQL * Automated Change * Add read method that supports concurrent execution to Source interface * Remove parallel iterator * Ensure that executor service is stopped * Automated Commit - Format and Process Resources Changes * Expose method to fix compilation issue * Use concurrent map to avoid access issues * Automated Commit - Format and Process Resources Changes * Ensure concurrent streams finish before closing source * Fix compile issue * Formatting * Exclude concurrent stream threads from orphan thread watcher * Automated Commit - Format and Process Resources Changes * Refactor orphaned thread logic to account for concurrent execution * PR feedback * Implement readStreams in wrapper source * Automated Commit - Format and Process Resources Changes * Add readStream override * Automated Commit - Format and Process Resources Changes * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * Debug logging * Reduce logging level * Replace synchronized calls to System.out.println when concurrent * Close consumer * Flush before close * Automated Commit - Format and Process Resources Changes * Remove charset * Use ASCII and flush periodically for parallel streams * Test performance harness patch * Automated Commit - Format and Process Resources Changes * Cleanup * Logging to identify concurrent read enabled * Mark parameter as final --------- Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com> Co-authored-by: rodireich <rodireich@users.noreply.github.com> * connectors-ci: disable dependency scanning (#29033) * updates (#29059) * Metadata: skip breaking change validation on prerelease (#29017) * skip breaking change validation * Move ValidatorOpts higher in call * Add prerelease test * Fix test * ✨ Source MongoDB Internal POC: Generate Test Data (#29049) * Add script to generate test data * Fix prose * Update credentials example * PR feedback * Bump Airbyte version from 0.50.12 to 0.50.13 * Bump versions for mssql strict-encrypt (#28964) * Bump versions for mssql strict-encrypt * Fix failing test * Fix failing test * 🎨 Improve replication method selection UX (#28882) * update replication method in MySQL source * bump version * update expected specs * update registries * bump strict encrypt version * make password always_show * change url * update registries * 🐛 Avoid writing records to log (#29047) * Avoid writing records to log * Update version * Rollout ctid cdc (#28708) * source-postgres: enable ctid+cdc implementation * 100% ctid rollout for cdc * remove CtidFeatureFlags * fix CdcPostgresSourceAcceptanceTest * Bump versions and release notes * Fix compilation error due to previous merge --------- Co-authored-by: subodh <subodh1810@gmail.com> * connectors-ci: fix `unhashable type 'set'` (#29064) * Add Slack Alert lifecycle to Dagster for Metadata publish (#28759) * DNC * Add slack lifecycle logging * Update to use slack * Update slack to use resource and bot * Improve markdown * Improve log * Add sensor logging * Extend sensor time * merge conflict * PR Refactoring * Make the tests work * remove unnecessary classes, pr feedback * more merging * Update airbyte-integrations/bases/base-typing-deduping-test/src/main/java/io/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.java Co-authored-by: Edward Gao <edward.gao@airbyte.io> * snowflake updates --------- Co-authored-by: Ben Church <ben@airbyte.io> Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com> Co-authored-by: Augustin <augustin@airbyte.io> Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com> Co-authored-by: Joe Reuter <joe@airbyte.io> Co-authored-by: flash1293 <flash1293@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Vasilis Gavriilidis <vasilis.gavriilidis@orfium.com> Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io> Co-authored-by: Alexandre Girard <alexandre@airbyte.io> Co-authored-by: girarda <girarda@users.noreply.github.com> Co-authored-by: btkcodedev <btk.codedev@gmail.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com> Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com> Co-authored-by: rodireich <rodireich@users.noreply.github.com> Co-authored-by: Alexandre Cuoci <Hesperide@users.noreply.github.com> Co-authored-by: terencecho <terencecho@users.noreply.github.com> Co-authored-by: Lake Mossman <lake@airbyte.io> Co-authored-by: Benoit Moriceau <benoit@airbyte.io> Co-authored-by: subodh <subodh1810@gmail.com> Co-authored-by: Edward Gao <edward.gao@airbyte.io>

* Add everything for BQ but migrate, refactor interface after practical work * Make new default methods, refactor to single implemented method * MigrationInterface and BQ impl created * Trying to integrate with standard inserts * remove unnecessary NameAndNamespacePair class * Shimmed in * Java Docs * Initial Testing Setup * Tests! * Move Migrator into TyperDeduper * Functional Migration * Add Integration Test * Pr updates * bump version * bump version * version bump * Update to airbyte-ci-internal (airbytehq#29026) * 🐛 Source Github, Instagram, Zendesk-support, Zendesk-talk: fix CAT tests fail on `spec` (airbytehq#28910) * connectors-ci: better modified connectors detection logic (airbytehq#28855) * connectors-ci: report path should always start with `airbyte-ci/` (airbytehq#29030) * make report path always start with airbyte-ci * revert report path in orchestrator * add more test cases * bump version * Updated docs (airbytehq#29019) * CDK: Embedded reader utils (airbytehq#28873) * relax pydantic dep * Automated Commit - Format and Process Resources Changes * wip * wrap up base integration * add init file * introduce CDK runner and improve error message * make state param optional * update protocol models * review comments * always run incremental if possible * fix --------- Co-authored-by: flash1293 <flash1293@users.noreply.github.com> * 🤖 Bump minor version of Airbyte CDK * 🚨🚨 Low code CDK: Decouple SimpleRetriever and HttpStream (airbytehq#28657) * fix tests * format * review comments * Automated Commit - Formatting Changes * review comments * review comments * review comments * log all messages * log all message * review comments * review comments * Automated Commit - Formatting Changes * add comment --------- Co-authored-by: flash1293 <flash1293@users.noreply.github.com> * 🤖 Bump minor version of Airbyte CDK * 🐛 Source Github, Instagram, Zendesk Support / Talk - revert `spec` changes and improve (airbytehq#29031) * Source oauth0: new streams and fix incremental (airbytehq#29001) * Add new streams Organizations,OrganizationMembers,OrganizationMemberRoles * relax schema definition to allow additional fields * Bump image tag version * revert some changes to the old schemas * Format python so gradle can pass * update incremental * remove unused print * fix unit test --------- Co-authored-by: Vasilis Gavriilidis <vasilis.gavriilidis@orfium.com> * 🐛 Source Mongo: Fix failing acceptance tests (airbytehq#28816) * Fix failing acceptance tests * Fix failing strict acceptance tests * Source-Greenhouse: Fix unit tests for new CDK version (airbytehq#28969) Fix unit tests * Add CSV options to the CSV parser (airbytehq#28491) * remove invalid legacy option * remove unused option * the tests pass but this is quite messy * very slight clean up * Add skip options to csv format * fix some of the typing issues * fixme comment * remove extra log message * fix typing issues * skip before header * skip after header * format * add another test * Automated Commit - Formatting Changes * auto generate column names * delete dead code * update title and description * true and false values * Update the tests * Add comment * missing test * rename * update expected spec * move to method * Update comment * fix typo * remove unused import * Add a comment * None records do not pass the WaitForDiscoverPolicy * format * remove second branch to ensure we always go through the same processing * Raise an exception if the record is None * reset * Update tests * handle unquoted newlines * Automated Commit - Formatting Changes * Update test case so the quoting is explicit * Update comment * Automated Commit - Formatting Changes * Fail validation if skipping rows before header and header is autogenerated * always fail if a record cannot be parsed * format * set write line_no in error message * remove none check * Automated Commit - Formatting Changes * enable autogenerate test * remove duplicate test * missing unit tests * Update * remove branching * remove unused none check * Update tests * remove branching * format * extract to function * comment * missing type * type annotation * use set * Document that the strings are case-sensitive * public -> private * add unit test * newline --------- Co-authored-by: girarda <girarda@users.noreply.github.com> * Dagster: Add sentry logging (airbytehq#28822) * Add sentry * add sentry decorator * Add traces * Use sentry trace * Improve duplicate logging * Add comments * DNC * Fix up issues * Move to scopes * Remove breadcrumb * Update lock * ✨Source Shortio: Migrate Python CDK to Low-code CDK (airbytehq#28950) * Migrate Shortio to Low-Code * Update abnormal state * Format * Update Docs * Fix metadata.yaml * Add pagination * Add incremental sync * add incremental parameters * update metadata * rollback update version * release date --------- Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> * Update to new verbiage (airbytehq#29051) * [skip ci] Metadata: Remove leading underscore (airbytehq#29024) * DNC * Add test models * Add model test * Remove underscore from metadata files * Regenerate models * Add test to check for key transformation * Allow additional fields on metadata * Delete transform * Proof of concept parallel source stream reading implementation for MySQL (airbytehq#26580) * Proof of concept parallel source stream reading implementation for MySQL * Automated Change * Add read method that supports concurrent execution to Source interface * Remove parallel iterator * Ensure that executor service is stopped * Automated Commit - Format and Process Resources Changes * Expose method to fix compilation issue * Use concurrent map to avoid access issues * Automated Commit - Format and Process Resources Changes * Ensure concurrent streams finish before closing source * Fix compile issue * Formatting * Exclude concurrent stream threads from orphan thread watcher * Automated Commit - Format and Process Resources Changes * Refactor orphaned thread logic to account for concurrent execution * PR feedback * Implement readStreams in wrapper source * Automated Commit - Format and Process Resources Changes * Add readStream override * Automated Commit - Format and Process Resources Changes * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * 🤖 Auto format source-mysql code [skip ci] * Debug logging * Reduce logging level * Replace synchronized calls to System.out.println when concurrent * Close consumer * Flush before close * Automated Commit - Format and Process Resources Changes * Remove charset * Use ASCII and flush periodically for parallel streams * Test performance harness patch * Automated Commit - Format and Process Resources Changes * Cleanup * Logging to identify concurrent read enabled * Mark parameter as final --------- Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com> Co-authored-by: rodireich <rodireich@users.noreply.github.com> * connectors-ci: disable dependency scanning (airbytehq#29033) * updates (airbytehq#29059) * Metadata: skip breaking change validation on prerelease (airbytehq#29017) * skip breaking change validation * Move ValidatorOpts higher in call * Add prerelease test * Fix test * ✨ Source MongoDB Internal POC: Generate Test Data (airbytehq#29049) * Add script to generate test data * Fix prose * Update credentials example * PR feedback * Bump Airbyte version from 0.50.12 to 0.50.13 * Bump versions for mssql strict-encrypt (airbytehq#28964) * Bump versions for mssql strict-encrypt * Fix failing test * Fix failing test * 🎨 Improve replication method selection UX (airbytehq#28882) * update replication method in MySQL source * bump version * update expected specs * update registries * bump strict encrypt version * make password always_show * change url * update registries * 🐛 Avoid writing records to log (airbytehq#29047) * Avoid writing records to log * Update version * Rollout ctid cdc (airbytehq#28708) * source-postgres: enable ctid+cdc implementation * 100% ctid rollout for cdc * remove CtidFeatureFlags * fix CdcPostgresSourceAcceptanceTest * Bump versions and release notes * Fix compilation error due to previous merge --------- Co-authored-by: subodh <subodh1810@gmail.com> * connectors-ci: fix `unhashable type 'set'` (airbytehq#29064) * Add Slack Alert lifecycle to Dagster for Metadata publish (airbytehq#28759) * DNC * Add slack lifecycle logging * Update to use slack * Update slack to use resource and bot * Improve markdown * Improve log * Add sensor logging * Extend sensor time * merge conflict * PR Refactoring * Make the tests work * remove unnecessary classes, pr feedback * more merging * Update airbyte-integrations/bases/base-typing-deduping-test/src/main/java/io/airbyte/integrations/base/destination/typing_deduping/BaseSqlGeneratorIntegrationTest.java Co-authored-by: Edward Gao <edward.gao@airbyte.io> * snowflake updates --------- Co-authored-by: Ben Church <ben@airbyte.io> Co-authored-by: Baz <oleksandr.bazarnov@globallogic.com> Co-authored-by: Augustin <augustin@airbyte.io> Co-authored-by: Serhii Lazebnyi <53845333+lazebnyi@users.noreply.github.com> Co-authored-by: Joe Reuter <joe@airbyte.io> Co-authored-by: flash1293 <flash1293@users.noreply.github.com> Co-authored-by: Marcos Marx <marcosmarxm@users.noreply.github.com> Co-authored-by: Vasilis Gavriilidis <vasilis.gavriilidis@orfium.com> Co-authored-by: Jonathan Pearlin <jonathan@airbyte.io> Co-authored-by: Alexandre Girard <alexandre@airbyte.io> Co-authored-by: girarda <girarda@users.noreply.github.com> Co-authored-by: btkcodedev <btk.codedev@gmail.com> Co-authored-by: marcosmarxm <marcosmarxm@gmail.com> Co-authored-by: Natalie Kwong <38087517+nataliekwong@users.noreply.github.com> Co-authored-by: jdpgrailsdev <jdpgrailsdev@users.noreply.github.com> Co-authored-by: octavia-squidington-iii <octavia-squidington-iii@users.noreply.github.com> Co-authored-by: Rodi Reich Zilberman <867491+rodireich@users.noreply.github.com> Co-authored-by: rodireich <rodireich@users.noreply.github.com> Co-authored-by: Alexandre Cuoci <Hesperide@users.noreply.github.com> Co-authored-by: terencecho <terencecho@users.noreply.github.com> Co-authored-by: Lake Mossman <lake@airbyte.io> Co-authored-by: Benoit Moriceau <benoit@airbyte.io> Co-authored-by: subodh <subodh1810@gmail.com> Co-authored-by: Edward Gao <edward.gao@airbyte.io>

Fix failing acceptance tests

5bb3c2a

jdpgrailsdev requested review from colesnodgrass, jpefaur and a team July 27, 2023 20:00

octavia-squidington-iii added area/connectors Connector related issues connectors/source/mongodb-v2 labels Jul 27, 2023

colesnodgrass approved these changes Jul 27, 2023

View reviewed changes

Merge branch 'master' into jonathan/fix-mongo-acceptance-tests

674c706

Fix failing strict acceptance tests

66b4a4a

octavia-squidington-iii added the connectors/source/mongodb-strict-encrypt label Jul 31, 2023

jdpgrailsdev requested a review from colesnodgrass July 31, 2023 14:22

Merge branch 'master' into jonathan/fix-mongo-acceptance-tests

5e827dd

Merge branch 'master' into jonathan/fix-mongo-acceptance-tests

c0a6fec

Merge branch 'master' into jonathan/fix-mongo-acceptance-tests

0ece061

Merge branch 'master' into jonathan/fix-mongo-acceptance-tests

da3a406

jdpgrailsdev added 2 commits August 2, 2023 13:56

Merge branch 'master' into jonathan/fix-mongo-acceptance-tests

988257f

Merge branch 'master' into jonathan/fix-mongo-acceptance-tests

8ba431c

jpefaur approved these changes Aug 2, 2023

View reviewed changes

Merge branch 'master' into jonathan/fix-mongo-acceptance-tests

48c6059

octavia-approvington approved these changes Aug 3, 2023

View reviewed changes

octavia-approvington merged commit 4540eab into master Aug 3, 2023

octavia-approvington deleted the jonathan/fix-mongo-acceptance-tests branch August 3, 2023 14:13

jbfbell pushed a commit that referenced this pull request Aug 5, 2023

🐛 Source Mongo: Fix failing acceptance tests (#28816)

ef95c9b

* Fix failing acceptance tests * Fix failing strict acceptance tests

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Source Mongo: Fix failing acceptance tests #28816

🐛 Source Mongo: Fix failing acceptance tests #28816

jdpgrailsdev commented Jul 27, 2023 •

edited

Loading

github-actions bot commented Jul 27, 2023 •

edited by jdpgrailsdev

Loading

octavia-squidington-iii commented Jul 27, 2023

octavia-squidington-iii commented Jul 27, 2023

octavia-squidington-iii commented Jul 31, 2023

octavia-squidington-iii commented Jul 31, 2023

octavia-squidington-iii commented Jul 31, 2023

octavia-squidington-iii commented Jul 31, 2023

jdpgrailsdev commented Jul 31, 2023 •

edited by github-actions bot

Loading

jdpgrailsdev commented Jul 31, 2023

jdpgrailsdev commented Jul 31, 2023

octavia-squidington-iii commented Jul 31, 2023

octavia-squidington-iii commented Jul 31, 2023

octavia-squidington-iii commented Aug 1, 2023

octavia-squidington-iii commented Aug 1, 2023

octavia-squidington-iii commented Aug 1, 2023

octavia-squidington-iii commented Aug 1, 2023

jdpgrailsdev commented Aug 2, 2023

octavia-squidington-iii commented Aug 2, 2023

octavia-squidington-iii commented Aug 2, 2023

octavia-squidington-iii commented Aug 2, 2023

octavia-squidington-iii commented Aug 2, 2023

octavia-squidington-iii commented Aug 2, 2023

octavia-squidington-iii commented Aug 2, 2023

octavia-squidington-iii commented Aug 3, 2023

octavia-squidington-iii commented Aug 3, 2023

jdpgrailsdev commented Aug 3, 2023

octavia-approvington commented Aug 3, 2023

🐛 Source Mongo: Fix failing acceptance tests #28816

🐛 Source Mongo: Fix failing acceptance tests #28816

Conversation

jdpgrailsdev commented Jul 27, 2023 • edited Loading

What

How

Recommended reading order

github-actions bot commented Jul 27, 2023 • edited by jdpgrailsdev Loading

Before Merging a Connector Pull Request

octavia-squidington-iii commented Jul 27, 2023

source-mongodb-v2 test report (commit 5bb3c2a52f) - ❌

octavia-squidington-iii commented Jul 27, 2023

source-mongodb-strict-encrypt test report (commit 5bb3c2a52f) - ❌

octavia-squidington-iii commented Jul 31, 2023

source-mongodb-strict-encrypt test report (commit 674c70630f) - ❌

octavia-squidington-iii commented Jul 31, 2023

source-mongodb-v2 test report (commit 674c70630f) - ❌

octavia-squidington-iii commented Jul 31, 2023

source-mongodb-strict-encrypt test report (commit 66b4a4abb1) - ❌

octavia-squidington-iii commented Jul 31, 2023

source-mongodb-v2 test report (commit 66b4a4abb1) - ❌

jdpgrailsdev commented Jul 31, 2023 • edited by github-actions bot Loading

Build Passed

jdpgrailsdev commented Jul 31, 2023

jdpgrailsdev commented Jul 31, 2023

octavia-squidington-iii commented Jul 31, 2023

source-mongodb-strict-encrypt test report (commit 5e827dd583) - ❌

octavia-squidington-iii commented Jul 31, 2023

source-mongodb-v2 test report (commit 5e827dd583) - ❌

octavia-squidington-iii commented Aug 1, 2023

source-mongodb-strict-encrypt test report (commit c0a6fec651) - ❌

octavia-squidington-iii commented Aug 1, 2023

source-mongodb-v2 test report (commit c0a6fec651) - ❌

octavia-squidington-iii commented Aug 1, 2023

source-mongodb-strict-encrypt test report (commit 0ece0612f7) - ❌

octavia-squidington-iii commented Aug 1, 2023

source-mongodb-v2 test report (commit 0ece0612f7) - ❌

jdpgrailsdev commented Aug 2, 2023

octavia-squidington-iii commented Aug 2, 2023

source-mongodb-strict-encrypt test report (commit da3a4061c7) - ❌

octavia-squidington-iii commented Aug 2, 2023

source-mongodb-v2 test report (commit da3a4061c7) - ❌

octavia-squidington-iii commented Aug 2, 2023

source-mongodb-strict-encrypt test report (commit 8ba431c10d) - ❌

octavia-squidington-iii commented Aug 2, 2023

source-mongodb-v2 test report (commit 8ba431c10d) - ❌

octavia-squidington-iii commented Aug 2, 2023

source-mongodb-strict-encrypt test report (commit 8ba431c10d) - ❌

octavia-squidington-iii commented Aug 2, 2023

source-mongodb-v2 test report (commit 8ba431c10d) - ❌

octavia-squidington-iii commented Aug 3, 2023

source-mongodb-v2 test report (commit 48c6059aba) - ❌

octavia-squidington-iii commented Aug 3, 2023

source-mongodb-strict-encrypt test report (commit 48c6059aba) - ❌

jdpgrailsdev commented Aug 3, 2023

octavia-approvington commented Aug 3, 2023

jdpgrailsdev commented Jul 27, 2023 •

edited

Loading

github-actions bot commented Jul 27, 2023 •

edited by jdpgrailsdev

Loading

source-mongodb-v2 test report (commit `5bb3c2a52f`) - ❌

source-mongodb-strict-encrypt test report (commit `5bb3c2a52f`) - ❌

source-mongodb-strict-encrypt test report (commit `674c70630f`) - ❌

source-mongodb-v2 test report (commit `674c70630f`) - ❌

source-mongodb-strict-encrypt test report (commit `66b4a4abb1`) - ❌

source-mongodb-v2 test report (commit `66b4a4abb1`) - ❌

jdpgrailsdev commented Jul 31, 2023 •

edited by github-actions bot

Loading

source-mongodb-strict-encrypt test report (commit `5e827dd583`) - ❌

source-mongodb-v2 test report (commit `5e827dd583`) - ❌

source-mongodb-strict-encrypt test report (commit `c0a6fec651`) - ❌

source-mongodb-v2 test report (commit `c0a6fec651`) - ❌

source-mongodb-strict-encrypt test report (commit `0ece0612f7`) - ❌

source-mongodb-v2 test report (commit `0ece0612f7`) - ❌

source-mongodb-strict-encrypt test report (commit `da3a4061c7`) - ❌

source-mongodb-v2 test report (commit `da3a4061c7`) - ❌

source-mongodb-strict-encrypt test report (commit `8ba431c10d`) - ❌

source-mongodb-v2 test report (commit `8ba431c10d`) - ❌

source-mongodb-strict-encrypt test report (commit `8ba431c10d`) - ❌

source-mongodb-v2 test report (commit `8ba431c10d`) - ❌

source-mongodb-v2 test report (commit `48c6059aba`) - ❌

source-mongodb-strict-encrypt test report (commit `48c6059aba`) - ❌