Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make exclusive containers first class citizens #34892

Merged

Conversation

stephane-airbyte
Copy link
Contributor

Very heavily lifted from #34865

Originally, the db-source's ContainerFactory test fixture was built under the assumption that testcontainers could always be shared by multiple tests. In practice it turns out that that's not always the case, especially for CDC tests where the CDC state is at the RDBMS level. For instance, in mysql, there's only one binlog.

Recently we struggled to debug an issue because an unshared testcontainer didn't forward its logs to the logging back-end. This is done by default for shared containers, but was forgotten for an unshared container in a CDC test.

Rather than peppering these tests with the suitable logging incantations, this PR adds an exclusive method to ContainerFactory which is like the existing shared except that, well, the container is not shared. This way testcontainers are provisioned in the same way regardless of whether they're shared or not.
While doing that, I took the opportunity to simplify the existing code

Copy link

vercel bot commented Feb 6, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment
Name Status Preview Comments Updated (UTC)
airbyte-docs ⬜️ Ignored (Inspect) Visit Preview Feb 7, 2024 8:46pm

@octavia-squidington-iii octavia-squidington-iii added area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit labels Feb 6, 2024
Copy link
Contributor Author

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @stephane-airbyte and the rest of your teammates on Graphite Graphite

Copy link
Contributor

github-actions bot commented Feb 6, 2024

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

  • PR name follows PR naming conventions
  • Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
  • Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
  • You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
  • Secrets in the connector's spec are annotated with airbyte_secret
  • All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
  • Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
  • Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
  • If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

  1. Check for hidden checklists in your PR description

  2. Toggle the github label checklist-action-run on/off to re-run the checklist CI.

@postamar
Copy link
Contributor

postamar commented Feb 6, 2024

seems simpler this way

@stephane-airbyte stephane-airbyte force-pushed the stephane/02-05-make_exclusive_containers_first_class_citizens branch 2 times, most recently from 0499a70 to 707f5dd Compare February 6, 2024 02:58
@stephane-airbyte stephane-airbyte force-pushed the stephane/02-05-make_exclusive_containers_first_class_citizens branch from 707f5dd to b56578e Compare February 6, 2024 03:42
Copy link
Contributor

github-actions bot commented Feb 6, 2024

Coverage report for source-postgres

There is no coverage information present for the Files changed

Total Project Coverage 69.34% 🍏

@@ -22,97 +24,106 @@
* ContainerFactory is the companion interface to {@link TestDatabase} for providing it with
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

comment needs adjusting

Copy link
Contributor

@postamar postamar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is much better, thanks for doing this. Considering that mssql is still broken on master feel free to adopt these changes in another source in order to get this to merge.

Can you port over the log message colouring from my PR also? It's actually useful. The commit is 22702c6

@stephane-airbyte stephane-airbyte force-pushed the stephane/02-05-make_exclusive_containers_first_class_citizens branch from b56578e to df37742 Compare February 6, 2024 17:25
@postamar postamar marked this pull request as ready for review February 7, 2024 20:24
@postamar postamar requested a review from a team as a code owner February 7, 2024 20:24
@postamar postamar force-pushed the stephane/02-05-make_exclusive_containers_first_class_citizens branch from 93e2258 to 559087f Compare February 7, 2024 20:40
@stephane-airbyte stephane-airbyte force-pushed the stephane/02-05-make_exclusive_containers_first_class_citizens branch from 559087f to dd40f34 Compare February 7, 2024 20:46
@stephane-airbyte
Copy link
Contributor Author

stephane-airbyte commented Feb 7, 2024

/publish-java-cdk

🕑 https://github.com/airbytehq/airbyte/actions/runs/7821132796
✅ Successfully published Java CDK version=0.16.6!

@postamar postamar merged commit e7dc82c into master Feb 7, 2024
24 checks passed
@postamar postamar deleted the stephane/02-05-make_exclusive_containers_first_class_citizens branch February 7, 2024 22:00
xiaohansong pushed a commit that referenced this pull request Feb 13, 2024
Co-authored-by: Marius Posta <marius@airbyte.io>
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 21, 2024
Co-authored-by: Marius Posta <marius@airbyte.io>
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
Co-authored-by: Marius Posta <marius@airbyte.io>
jatinyadav-cc pushed a commit to ollionorg/datapipes-airbyte that referenced this pull request Feb 26, 2024
Co-authored-by: Marius Posta <marius@airbyte.io>
xiaohansong pushed a commit that referenced this pull request Feb 27, 2024
Co-authored-by: Marius Posta <marius@airbyte.io>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation CDK Connector Development Kit connectors/source/mssql connectors/source/postgres
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants