🎉 Source MySql: Added SSL certificates to MySql Source #15044

andriikorotkov · 2022-07-26T18:03:30Z

What

How

Describe the solution

🚨 User Impact 🚨

Are there any breaking changes? What is the end result perceived by the user? If yes, please merge this PR with the 🚨🚨 emoji so changelog authors can further highlight this if needed.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

…ts SSL certificates

…4599_mysql_source_certificates � Conflicts: � airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java

…4599_mysql_source_certificates

…or CA certificate

andriikorotkov · 2022-08-09T15:45:48Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2826542443
❌ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2826542443
🐛 https://gradle.com/s/pqbtszbbjeigk

Build Failed

Test summary info:

Could not find result summary

andriikorotkov · 2022-08-14T20:38:51Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2857100445
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2857100445
No Python unittests run

Build Passed

Test summary info:

All Passed

…4599_mysql_source_certificates

andriikorotkov · 2022-08-15T13:48:28Z

/test connector=connectors/source-mysql-strict-encrypt

🕑 connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2861063000
❌ connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2861063000
🐛 https://gradle.com/s/ddureakvby2vk

Build Failed

Test summary info:

Could not find result summary

…me format

andriikorotkov · 2022-08-15T14:10:39Z

/test connector=connectors/source-mysql-strict-encrypt

🕑 connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2861200711
✅ connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2861200711
No Python unittests run

Build Passed

Test summary info:

All Passed

andriikorotkov · 2022-08-15T14:12:46Z

@subodh1810 I added the necessary property for CDC to work properly with SSL certificates. Also, I added the necessary test for this. If you think this pull request is fine, please approve it.

andriikorotkov · 2022-08-15T14:16:55Z

@subodh1810 also, the changes that were added to Source MySql in your pull request - #15504, I added to source-mysql-strict-encrypt, to the MySqlStrictEncryptJdbcSourceAcceptanceTest class. Now these tests work correctly.

andriikorotkov · 2022-08-15T14:19:40Z

@rodireich I fixed all remarks which you pointed out. Please see this pull request again.

rodireich · 2022-08-15T18:13:33Z

Thanks Andrii
Just to make sure I get it, this change also covers CDC connections done through debezium.
right?

andriikorotkov · 2022-08-16T09:16:41Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2866743074
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2866743074
No Python unittests run

Build Passed

Test summary info:

All Passed

andriikorotkov · 2022-08-16T09:16:52Z

/test connector=connectors/source-mysql-strict-encrypt

🕑 connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2866744327
✅ connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2866744327
No Python unittests run

Build Passed

Test summary info:

All Passed

andriikorotkov · 2022-08-16T09:40:38Z

Thanks Andrii Just to make sure I get it, this change also covers CDC connections done through debezium. right?

@rodireich Oops, sorry, I didn't push (commit only) a small piece of code where I add additional properties for CDC. Now everything works as it should. Please take another look at this pull request.

rodireich · 2022-08-16T18:13:25Z

/publish connector=connectors/source-mysql,connectors/source-mysql-strict-encrypt parallel=true

🕑 Publishing the following connectors:
connectors/source-mysql,connectors/source-mysql-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2870135371

Connector	Did it publish?	Were definitions generated?
connectors/source-mysql-strict-encrypt	✅	❌
connectors/source-mysql	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

* updated mysql source specification and added field for root and clients SSL certificates * added SSL mode for mysql source * fixed code style * updated run process timeout * updated method for create keystore and updated tests * updated normalization version for postgres destination * updated normalization version for postgres destination * added tests for connection with certificates * updated tests for connection with full certificates and added tests for CA certificate * updated tests * updated source-mysql-strict-encrypt and updated versions * updated code style * updated doc * updated specs * fixed minor remarks * fixed minor remarks * updated tests * fixed remarks and updated specification * fixed mysql sources connectors version * added CDC + SSL Certificates tests * added property for CDC and added tests for test SSL with CDC together * fixed MySqlStrictEncryptJdbcSourceAcceptanceTest for work with datetime format * added property for CDC and added tests for test SSL with CDC together * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

andriikorotkov added 4 commits July 21, 2022 17:30

updated mysql source specification and added field for root and clien…

92b8b8f

…ts SSL certificates

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

93f7913

…4599_mysql_source_certificates � Conflicts: � airbyte-integrations/connectors/source-mysql/src/main/java/io/airbyte/integrations/source/mysql/MySqlSource.java

added SSL mode for mysql source

7178243

fixed code style

f5bd946

andriikorotkov requested review from kimerinn and DoNotPanicUA July 26, 2022 18:03

octavia-squidington-iii added the connectors/source/mysql label Jul 26, 2022

github-actions bot added the area/connectors Connector related issues label Jul 26, 2022

updated run process timeout

cfe583d

andriikorotkov temporarily deployed to more-secrets July 27, 2022 07:32 Inactive

andriikorotkov added 2 commits July 28, 2022 10:54

updated method for create keystore and updated tests

6e2dbb3

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

b872f43

…4599_mysql_source_certificates

andriikorotkov temporarily deployed to more-secrets July 28, 2022 08:16 Inactive

subodh1810 mentioned this pull request Aug 1, 2022

Support SSL for CDC in database source connectors #15167

Closed

andriikorotkov added 2 commits August 9, 2022 12:37

updated normalization version for postgres destination

ea08e2b

updated normalization version for postgres destination

ac0f651

andriikorotkov temporarily deployed to more-secrets August 9, 2022 09:40 Inactive

andriikorotkov added 2 commits August 9, 2022 15:09

added tests for connection with certificates

a10374d

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

67d67ae

…4599_mysql_source_certificates

andriikorotkov temporarily deployed to more-secrets August 9, 2022 12:12 Inactive

updated tests for connection with full certificates and added tests f…

9cc861d

…or CA certificate

andriikorotkov requested review from etsybaev and sashaNeshcheret August 9, 2022 15:19

andriikorotkov temporarily deployed to more-secrets August 9, 2022 15:21 Inactive

andriikorotkov linked an issue Aug 9, 2022 that may be closed by this pull request

Source MySQL: SSL authentication certificate import #4599

Closed

updated tests

2575317

andriikorotkov temporarily deployed to more-secrets August 10, 2022 07:55 Inactive

updated source-mysql-strict-encrypt and updated versions

bd1fcb5

github-actions bot added the area/documentation Improvements or additions to documentation label Aug 10, 2022

added CDC + SSL Certificates tests

132c7ce

andriikorotkov temporarily deployed to more-secrets August 14, 2022 20:41 Inactive

added property for CDC and added tests for test SSL with CDC together

5503174

andriikorotkov temporarily deployed to more-secrets August 15, 2022 11:50 Inactive

Merge branch 'master' of github.com:airbytehq/airbyte into akorotkov/…

5f2dad7

…4599_mysql_source_certificates

andriikorotkov temporarily deployed to more-secrets August 15, 2022 13:40 Inactive

fixed MySqlStrictEncryptJdbcSourceAcceptanceTest for work with dateti…

9e732d7

…me format

andriikorotkov temporarily deployed to more-secrets August 15, 2022 14:13 Inactive

andriikorotkov requested a review from subodh1810 August 15, 2022 14:17

rodireich approved these changes Aug 15, 2022

View reviewed changes

added property for CDC and added tests for test SSL with CDC together

5c2d5aa

andriikorotkov temporarily deployed to more-secrets August 16, 2022 09:18 Inactive

rodireich approved these changes Aug 16, 2022

View reviewed changes

auto-bump connector version [ci skip]

099f89d

rodireich merged commit e5098e8 into master Aug 16, 2022

rodireich deleted the akorotkov/4599_mysql_source_certificates branch August 16, 2022 20:34

octavia-squidington-iii mentioned this pull request Aug 19, 2022

Bump Airbyte version from 0.40.0-alpha to 0.40.1-alpha #15775

Closed

evantahler mentioned this pull request Aug 22, 2022

Bump Airbyte version from 0.40.0-alpha to 0.40.1 #15857

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 Source MySql: Added SSL certificates to MySql Source #15044

🎉 Source MySql: Added SSL certificates to MySql Source #15044

andriikorotkov commented Jul 26, 2022 •

edited

Loading

andriikorotkov commented Aug 9, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 14, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022

andriikorotkov commented Aug 15, 2022

andriikorotkov commented Aug 15, 2022

rodireich commented Aug 15, 2022

andriikorotkov commented Aug 16, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 16, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 16, 2022 •

edited

Loading

rodireich commented Aug 16, 2022 •

edited by github-actions bot

Loading

🎉 Source MySql: Added SSL certificates to MySql Source #15044

🎉 Source MySql: Added SSL certificates to MySql Source #15044

Conversation

andriikorotkov commented Jul 26, 2022 • edited Loading

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

andriikorotkov commented Aug 9, 2022 • edited by github-actions bot Loading

Build Failed

andriikorotkov commented Aug 14, 2022 • edited by github-actions bot Loading

Build Passed

andriikorotkov commented Aug 15, 2022 • edited by github-actions bot Loading

Build Failed

andriikorotkov commented Aug 15, 2022 • edited by github-actions bot Loading

Build Passed

andriikorotkov commented Aug 15, 2022

andriikorotkov commented Aug 15, 2022

andriikorotkov commented Aug 15, 2022

rodireich commented Aug 15, 2022

andriikorotkov commented Aug 16, 2022 • edited by github-actions bot Loading

Build Passed

andriikorotkov commented Aug 16, 2022 • edited by github-actions bot Loading

Build Passed

andriikorotkov commented Aug 16, 2022 • edited Loading

rodireich commented Aug 16, 2022 • edited by github-actions bot Loading

andriikorotkov commented Jul 26, 2022 •

edited

Loading

andriikorotkov commented Aug 9, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 14, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 15, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 16, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 16, 2022 •

edited by github-actions bot

Loading

andriikorotkov commented Aug 16, 2022 •

edited

Loading

rodireich commented Aug 16, 2022 •

edited by github-actions bot

Loading