mysql-cdc: implementation to construct initial debezium state #28561

subodh1810 · 2023-07-21T13:45:31Z

Issue : #28238

github-actions · 2023-07-21T13:45:53Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

octavia-squidington-iii · 2023-07-24T17:57:19Z

source-mysql-strict-encrypt test report (commit `a13baec425`) - ❌

⏲️ Total pipeline duration: 17mn10s

Step	Result
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mysql-strict-encrypt:integrationTest	❌
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

octavia-squidington-iii · 2023-07-24T18:05:33Z

source-alloydb test report (commit `a13baec425`) - ❌

⏲️ Total pipeline duration: 08mn00s

Step	Result
Validate airbyte-integrations/connectors/source-alloydb/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-alloydb docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-alloydb:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb test

octavia-squidington-iii · 2023-07-24T18:28:07Z

source-mysql test report (commit `a13baec425`) - ❌

⏲️ Total pipeline duration: 22mn19s

Step	Result
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest	❌
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

octavia-squidington-iii · 2023-07-24T18:36:01Z

source-alloydb-strict-encrypt test report (commit `a13baec425`) - ❌

⏲️ Total pipeline duration: 07mn40s

Step	Result
Validate airbyte-integrations/connectors/source-alloydb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-alloydb-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-alloydb-strict-encrypt:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb-strict-encrypt test

octavia-squidington-iii · 2023-07-24T18:54:44Z

source-postgres test report (commit `a13baec425`) - ✅

⏲️ Total pipeline duration: 18mn27s

Step	Result
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-postgres:integrationTest	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

octavia-squidington-iii · 2023-07-24T19:05:17Z

source-postgres-strict-encrypt test report (commit `a13baec425`) - ❌

⏲️ Total pipeline duration: 10mn19s

Step	Result
Validate airbyte-integrations/connectors/source-postgres-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-postgres-strict-encrypt:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

akashkulk · 2023-07-24T20:50:03Z

...m/src/main/java/io/airbyte/integrations/debezium/internals/mysql/MySqlDebeziumStateUtil.java

+  public static final String MYSQL_CDC_OFFSET = "mysql_cdc_offset";
+  public static final String MYSQL_DB_HISTORY = "mysql_db_history";
+
+  public JsonNode constructInitialDebeziumState(final Properties properties,


So this method is basically constructing the state from spinning up the DBZ engine and grabbing the state assc with the first record?

Yes and no! As discussed over call, we construct the offset by ourselves but in order to construct the db schema we need Debezium and thus we give Debezium the offset that we generated and run it with schema_only_recovery mode and it generates the schema history on its own and we use it.

akashkulk · 2023-07-25T21:48:52Z

...m/src/main/java/io/airbyte/integrations/debezium/internals/mysql/MySqlDebeziumStateUtil.java

+  public JsonNode constructInitialDebeziumState(final Properties properties,
+                                                final ConfiguredAirbyteCatalog catalog,
+                                                final JdbcDatabase database) {
+    // https://debezium.io/documentation/reference/2.2/connectors/mysql.html#mysql-property-snapshot-mode


Can you just add a quick comment on the schema_only_recovery and why we use it?

akashkulk · 2023-07-25T21:53:00Z

...m/src/main/java/io/airbyte/integrations/debezium/internals/mysql/MySqlDebeziumStateUtil.java

+    return jsonNode;
+  }
+
+  private MysqlDebeziumStateAttributes getStateAttributesFromDB(final JdbcDatabase database) {


tiny nit : let's make this method public. I'm going to need it in the initial snapshotting code, so that we can attach the binlog_file + position for each record emitted in the initial pk sync

akashkulk

minor comments, looks good otherwise

subodh1810 · 2023-07-26T11:17:43Z

/approve-and-merge reason="change is not going to impact any current running connector cause its not being used"

octavia-approvington · 2023-07-26T11:18:21Z

All in!!

octavia-squidington-iii · 2023-07-26T20:34:07Z

source-mssql-strict-encrypt test report (commit `b0516ec1f9`) - ❌

⏲️ Total pipeline duration: 18mn46s

Step	Result
Validate airbyte-integrations/connectors/source-mssql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mssql-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mssql-strict-encrypt:integrationTest	❌
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mssql-strict-encrypt test

octavia-squidington-iii · 2023-07-26T20:48:15Z

source-postgres-strict-encrypt test report (commit `b0516ec1f9`) - ❌

⏲️ Total pipeline duration: 13mn54s

Step	Result
Validate airbyte-integrations/connectors/source-postgres-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-postgres-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-postgres-strict-encrypt:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres-strict-encrypt test

octavia-squidington-iii · 2023-07-26T20:57:44Z

source-alloydb-strict-encrypt test report (commit `b0516ec1f9`) - ❌

⏲️ Total pipeline duration: 09mn13s

Step	Result
Validate airbyte-integrations/connectors/source-alloydb-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-alloydb-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-alloydb-strict-encrypt:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb-strict-encrypt test

octavia-squidington-iii · 2023-07-26T21:17:53Z

source-mysql-strict-encrypt test report (commit `b0516ec1f9`) - ❌

⏲️ Total pipeline duration: 19mn54s

Step	Result
Validate airbyte-integrations/connectors/source-mysql-strict-encrypt/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mysql-strict-encrypt docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mysql-strict-encrypt:integrationTest	❌
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql-strict-encrypt test

octavia-squidington-iii · 2023-07-26T21:43:10Z

source-mysql test report (commit `b0516ec1f9`) - ❌

⏲️ Total pipeline duration: 25mn02s

Step	Result
Validate airbyte-integrations/connectors/source-mysql/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mysql docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mysql:integrationTest	❌
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mysql test

octavia-squidington-iii · 2023-07-26T22:03:09Z

source-postgres test report (commit `b0516ec1f9`) - ✅

⏲️ Total pipeline duration: 19mn42s

Step	Result
Validate airbyte-integrations/connectors/source-postgres/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-postgres docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-postgres:integrationTest	✅
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-postgres test

octavia-squidington-iii · 2023-07-26T22:13:44Z

source-alloydb test report (commit `b0516ec1f9`) - ❌

⏲️ Total pipeline duration: 10mn17s

Step	Result
Validate airbyte-integrations/connectors/source-alloydb/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-alloydb docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-alloydb:integrationTest	✅
Acceptance tests	❌

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-alloydb test

octavia-squidington-iii · 2023-07-26T22:38:11Z

source-mssql test report (commit `b0516ec1f9`) - ❌

⏲️ Total pipeline duration: 24mn09s

Step	Result
Validate airbyte-integrations/connectors/source-mssql/metadata.yaml	✅
Connector version semver check	✅
QA checks	✅
Build connector tar	✅
Build source-mssql docker image for platform linux/x86_64	✅
./gradlew :airbyte-integrations:connectors:source-mssql:integrationTest	❌
Acceptance tests	✅

🔗 View the logs here

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=source-mssql test

subodh1810 added 3 commits July 18, 2023 20:13

wip

2c78464

Merge branch 'master' into mysql-cdc-initial-debezium-state

96620f6

wip 2

8e2f021

subodh1810 self-assigned this Jul 21, 2023

octavia-squidington-iii added area/connectors Connector related issues connectors/source/mysql labels Jul 21, 2023

subodh1810 and others added 4 commits July 24, 2023 19:18

add tests + finalise implementation

71e3ee6

Merge branch 'master' into mysql-cdc-initial-debezium-state

7aee46d

undo unwanted changes

95c4d0e

Automated Commit - Format and Process Resources Changes

a13baec

subodh1810 marked this pull request as ready for review July 24, 2023 17:38

subodh1810 requested a review from a team as a code owner July 24, 2023 17:38

subodh1810 requested a review from akashkulk July 24, 2023 17:38

akashkulk reviewed Jul 24, 2023

View reviewed changes

akashkulk reviewed Jul 25, 2023

View reviewed changes

akashkulk approved these changes Jul 25, 2023

View reviewed changes

subodh1810 added 4 commits July 26, 2023 16:37

minor refactor

f2af115

Merge branch 'master' into mysql-cdc-initial-debezium-state

593faf2

add comment

0a2f0ec

make method public

98551ba

octavia-approvington approved these changes Jul 26, 2023

View reviewed changes

octavia-approvington merged commit bcb5a4d into master Jul 26, 2023

octavia-approvington deleted the mysql-cdc-initial-debezium-state branch July 26, 2023 11:18

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

mysql-cdc: implementation to construct initial debezium state #28561

mysql-cdc: implementation to construct initial debezium state #28561

subodh1810 commented Jul 21, 2023 •

edited

Loading

github-actions bot commented Jul 21, 2023

octavia-squidington-iii commented Jul 24, 2023

octavia-squidington-iii commented Jul 24, 2023

octavia-squidington-iii commented Jul 24, 2023

octavia-squidington-iii commented Jul 24, 2023

octavia-squidington-iii commented Jul 24, 2023

octavia-squidington-iii commented Jul 24, 2023

akashkulk Jul 24, 2023

subodh1810 Jul 26, 2023

akashkulk Jul 25, 2023 •

edited

Loading

akashkulk Jul 25, 2023

akashkulk left a comment

subodh1810 commented Jul 26, 2023

octavia-approvington commented Jul 26, 2023

octavia-squidington-iii commented Jul 26, 2023

octavia-squidington-iii commented Jul 26, 2023

octavia-squidington-iii commented Jul 26, 2023

octavia-squidington-iii commented Jul 26, 2023

octavia-squidington-iii commented Jul 26, 2023

octavia-squidington-iii commented Jul 26, 2023

octavia-squidington-iii commented Jul 26, 2023

octavia-squidington-iii commented Jul 26, 2023

mysql-cdc: implementation to construct initial debezium state #28561

mysql-cdc: implementation to construct initial debezium state #28561

Conversation

subodh1810 commented Jul 21, 2023 • edited Loading

github-actions bot commented Jul 21, 2023

Before Merging a Connector Pull Request

octavia-squidington-iii commented Jul 24, 2023

source-mysql-strict-encrypt test report (commit a13baec425) - ❌

octavia-squidington-iii commented Jul 24, 2023

source-alloydb test report (commit a13baec425) - ❌

octavia-squidington-iii commented Jul 24, 2023

source-mysql test report (commit a13baec425) - ❌

octavia-squidington-iii commented Jul 24, 2023

source-alloydb-strict-encrypt test report (commit a13baec425) - ❌

octavia-squidington-iii commented Jul 24, 2023

source-postgres test report (commit a13baec425) - ✅

octavia-squidington-iii commented Jul 24, 2023

source-postgres-strict-encrypt test report (commit a13baec425) - ❌

akashkulk Jul 24, 2023

Choose a reason for hiding this comment

subodh1810 Jul 26, 2023

Choose a reason for hiding this comment

akashkulk Jul 25, 2023 • edited Loading

Choose a reason for hiding this comment

akashkulk Jul 25, 2023

Choose a reason for hiding this comment

akashkulk left a comment

Choose a reason for hiding this comment

subodh1810 commented Jul 26, 2023

octavia-approvington commented Jul 26, 2023

octavia-squidington-iii commented Jul 26, 2023

source-mssql-strict-encrypt test report (commit b0516ec1f9) - ❌

octavia-squidington-iii commented Jul 26, 2023

source-postgres-strict-encrypt test report (commit b0516ec1f9) - ❌

octavia-squidington-iii commented Jul 26, 2023

source-alloydb-strict-encrypt test report (commit b0516ec1f9) - ❌

octavia-squidington-iii commented Jul 26, 2023

source-mysql-strict-encrypt test report (commit b0516ec1f9) - ❌

octavia-squidington-iii commented Jul 26, 2023

source-mysql test report (commit b0516ec1f9) - ❌

octavia-squidington-iii commented Jul 26, 2023

source-postgres test report (commit b0516ec1f9) - ✅

octavia-squidington-iii commented Jul 26, 2023

source-alloydb test report (commit b0516ec1f9) - ❌

octavia-squidington-iii commented Jul 26, 2023

source-mssql test report (commit b0516ec1f9) - ❌

subodh1810 commented Jul 21, 2023 •

edited

Loading

source-mysql-strict-encrypt test report (commit `a13baec425`) - ❌

source-alloydb test report (commit `a13baec425`) - ❌

source-mysql test report (commit `a13baec425`) - ❌

source-alloydb-strict-encrypt test report (commit `a13baec425`) - ❌

source-postgres test report (commit `a13baec425`) - ✅

source-postgres-strict-encrypt test report (commit `a13baec425`) - ❌

akashkulk Jul 25, 2023 •

edited

Loading

source-mssql-strict-encrypt test report (commit `b0516ec1f9`) - ❌

source-postgres-strict-encrypt test report (commit `b0516ec1f9`) - ❌

source-alloydb-strict-encrypt test report (commit `b0516ec1f9`) - ❌

source-mysql-strict-encrypt test report (commit `b0516ec1f9`) - ❌

source-mysql test report (commit `b0516ec1f9`) - ❌

source-postgres test report (commit `b0516ec1f9`) - ✅

source-alloydb test report (commit `b0516ec1f9`) - ❌

source-mssql test report (commit `b0516ec1f9`) - ❌