MySQL Source : Standardize spec.json for DB connectors that support log-based CDC replication #16216

VitaliiMaltsev · 2022-09-01T10:53:47Z

What

Analytics team looking for a way to track CDC vs. non-CDC source statistics. For more information, refer to https://github.com/airbytehq/airbyte-warehouse/issues/50.

When trying to expose the fields in the warehouse, we realized there's a mismatch in spec.json files between Postgres and MySQL, for example. The difference in specs makes it difficult to track the fields, so we would like to standardize them.

How

The spec.json of connectors such as MySql should use a oneOf called "replication_method" where the method can be selected via a const value of CDC instead

🚨 User Impact 🚨

To avoid backward compatibility issue user config for existing source connections should be updated

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

…og-based CDC replication

github-actions · 2022-09-01T10:54:36Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql-strict-encrypt
source-mysql

github-actions · 2022-09-01T11:46:04Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql-strict-encrypt
source-mysql

github-actions · 2022-09-01T12:06:40Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql-strict-encrypt
source-mysql

VitaliiMaltsev · 2022-09-01T12:26:32Z

/test connector=connectors/source-mysql-strict-encrypt

🕑 connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2971605041
✅ connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2971605041
No Python unittests run

Build Passed

Test summary info:

All Passed

VitaliiMaltsev · 2022-09-01T12:26:44Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2971605671
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2971605671
No Python unittests run

Build Passed

Test summary info:

All Passed

github-actions · 2022-09-01T12:27:03Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql-strict-encrypt
source-mysql

evantahler · 2022-09-02T00:02:32Z

Removing myself as a PR reviewer, as there is no longer a change in the platform/worker codebase.

# Conflicts: # docs/integrations/sources/mysql.md

…dardize-spec' into vmaltsev/12917-mysql-source-standardize-spec

VitaliiMaltsev · 2022-09-03T17:55:34Z

/publish connector=connectors/source-mysql-strict-encrypt auto-bump-version=false

🕑 Publishing the following connectors:
connectors/source-mysql-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2985278878

Connector	Did it publish?	Were definitions generated?
connectors/source-mysql-strict-encrypt	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

github-actions · 2022-09-03T17:55:42Z

NOTE ⚠️ Changes in this PR affect the following connectors. Make sure to run corresponding integration tests:

source-mysql
source-mysql-strict-encrypt

VitaliiMaltsev · 2022-09-03T17:55:52Z

/publish connector=connectors/source-mysql

🕑 Publishing the following connectors:
connectors/source-mysql
https://github.com/airbytehq/airbyte/actions/runs/2985279525

Connector	Did it publish?	Were definitions generated?
connectors/source-mysql	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

VitaliiMaltsev · 2022-09-04T11:02:24Z

/test connector=connectors/source-mysql-strict-encrypt

🕑 connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2987837983
✅ connectors/source-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2987837983
No Python unittests run

Build Passed

Test summary info:

All Passed

VitaliiMaltsev · 2022-09-04T11:02:34Z

/test connector=connectors/source-mysql

🕑 connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2987838229
✅ connectors/source-mysql https://github.com/airbytehq/airbyte/actions/runs/2987838229
No Python unittests run

Build Passed

Test summary info:

All Passed

VitaliiMaltsev · 2022-09-04T11:34:10Z

/publish connector=connectors/source-mysql-strict-encrypt auto-bump-version=false run-tests=false

🕑 Publishing the following connectors:
connectors/source-mysql-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2987932917

Connector	Did it publish?	Were definitions generated?
connectors/source-mysql-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

VitaliiMaltsev · 2022-09-04T12:18:34Z

/publish connector=connectors/source-mysql run-tests=false

🕑 Publishing the following connectors:
connectors/source-mysql
https://github.com/airbytehq/airbyte/actions/runs/2988059512

Connector	Did it publish?	Were definitions generated?
connectors/source-mysql	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

* master: (47 commits) Add email to identify users analytics call (#16327) 🎉 Source Amazon Ads: improve `config.start_date` validation (#16191) Add comments about intermediate state emission (#16262) MySQL Source : Standardize spec.json for DB connectors that support log-based CDC replication (#16216) MSSQL Source : Standardize spec.json for DB connectors that support log-based CDC replication (#16215) Hide a bunch more destination with potential unsecure API access (#16320) Skip unit tests when run-tests is false (#16267) Hide Destination connections from UI (#16310) Add scheduled task to clean up old files from workspace (#16247) Source Google Analytics v4: Re-name google analytics connector (#16306) 🐛 Source Facebook Marketing: remove "end_date" from config if empty value (re-implement #16096) (#16222) Fix github action syntax (#16277) Re-name google analytics cionnectors (#16287) Bump Airbyte version from 0.40.3 to 0.40.4 (#16275) Hide ES and Redis destination connectors from Cloud (#16276) 15700 add tests for PokeAPI (#15701) Add ProtocolVersion to StandardDefs (#16237) 🪟 🔧 🧹 Migrate attempt `bytesSynced` to `totalStats.bytesEmitted` and cleanup `AttemptDetails` component (#16126) Improve behavior of password input field (#16011) Improve airbyte-metrics support in the Helm chart (#16166) ...

…og-based CDC replication (airbytehq#16216) * Fixed bucket naming for S3 * removed redundant configs * MySQL Source : Standardize spec.json for DB connectors that support log-based CDC replication * fixed strict encrypt tests * fixed mysql tests * bump version * auto-bump connector version [ci skip] Co-authored-by: Octavia Squidington III <octavia-squidington-iii@users.noreply.github.com>

VitaliiMaltsev added 19 commits August 1, 2022 12:18

Fixed bucket naming for S3

979fe1f

Merge branch 'master' of https://github.com/airbytehq/airbyte

7d2963a

Merge branch 'master' of https://github.com/airbytehq/airbyte

e921180

Merge branch 'master' of https://github.com/airbytehq/airbyte

0e6f7df

Merge branch 'master' of https://github.com/airbytehq/airbyte

7924657

Merge branch 'master' of https://github.com/airbytehq/airbyte

b5c1106

Merge branch 'master' of https://github.com/airbytehq/airbyte

fc34cfb

Merge branch 'master' of https://github.com/airbytehq/airbyte

8ae35a5

Merge branch 'master' of https://github.com/airbytehq/airbyte

e5e1d12

Merge branch 'master' of https://github.com/airbytehq/airbyte

f003cc6

Merge branch 'master' of https://github.com/airbytehq/airbyte

a526c81

Merge branch 'master' of https://github.com/airbytehq/airbyte

b9a4c3d

Merge remote-tracking branch 'origin/master'

3917237

removed redundant configs

aeae719

Merge branch 'master' of https://github.com/airbytehq/airbyte

9ec1b7a

Merge branch 'master' of https://github.com/airbytehq/airbyte

ebbcedf

Merge branch 'master' of https://github.com/airbytehq/airbyte

c2a123b

Merge branch 'master' of https://github.com/airbytehq/airbyte

28875bd

MySQL Source : Standardize spec.json for DB connectors that support l…

6badf97

…og-based CDC replication

VitaliiMaltsev requested a review from a team as a code owner September 1, 2022 10:53

github-actions bot added area/connectors Connector related issues area/documentation Improvements or additions to documentation labels Sep 1, 2022

octavia-squidington-iii added connectors/source/mysql connectors/source/mysql-strict-encrypt labels Sep 1, 2022

fixed strict encrypt tests

abe4d34

fixed mysql tests

e7c2c7d

Merge branch 'master' into vmaltsev/12917-mysql-source-standardize-spec

1c668a7

Merge branch 'master' of https://github.com/airbytehq/airbyte

b3c7e13

VitaliiMaltsev mentioned this pull request Sep 1, 2022

Standardize spec.json for DB connectors that support log-based CDC replication #14910

Closed

37 tasks

VitaliiMaltsev requested review from grishick and evantahler September 1, 2022 13:48

evantahler removed their request for review September 2, 2022 00:01

grishick approved these changes Sep 2, 2022

View reviewed changes

VitaliiMaltsev added 4 commits September 3, 2022 20:44

Merge branch 'master' of https://github.com/airbytehq/airbyte

e9b8486

Merge branch 'master' into vmaltsev/12917-mysql-source-standardize-spec

5af7f2c

# Conflicts: # docs/integrations/sources/mysql.md

bump version

cc46bbf

Merge remote-tracking branch 'origin/vmaltsev/12917-mysql-source-stan…

51509b5

…dardize-spec' into vmaltsev/12917-mysql-source-standardize-spec

auto-bump connector version [ci skip]

537ff1f

VitaliiMaltsev merged commit e9a8a05 into master Sep 4, 2022

VitaliiMaltsev deleted the vmaltsev/12917-mysql-source-standardize-spec branch September 4, 2022 12:30

VitaliiMaltsev linked an issue Sep 5, 2022 that may be closed by this pull request

Standardize spec.json for DB connectors that support log-based CDC replication #12917

Closed

octavia-squidington-iii mentioned this pull request Sep 12, 2022

Bump Airbyte version from 0.40.4 to 0.40.5 #16610

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

MySQL Source : Standardize spec.json for DB connectors that support log-based CDC replication #16216

MySQL Source : Standardize spec.json for DB connectors that support log-based CDC replication #16216

VitaliiMaltsev commented Sep 1, 2022

github-actions bot commented Sep 1, 2022

github-actions bot commented Sep 1, 2022

github-actions bot commented Sep 1, 2022

VitaliiMaltsev commented Sep 1, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 1, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Sep 1, 2022

evantahler commented Sep 2, 2022

VitaliiMaltsev commented Sep 3, 2022 •

edited by github-actions bot

Loading

github-actions bot commented Sep 3, 2022

VitaliiMaltsev commented Sep 3, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 4, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 4, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 4, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 4, 2022 •

edited by github-actions bot

Loading

MySQL Source : Standardize spec.json for DB connectors that support log-based CDC replication #16216

MySQL Source : Standardize spec.json for DB connectors that support log-based CDC replication #16216

Conversation

VitaliiMaltsev commented Sep 1, 2022

What

How

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

github-actions bot commented Sep 1, 2022

github-actions bot commented Sep 1, 2022

github-actions bot commented Sep 1, 2022

VitaliiMaltsev commented Sep 1, 2022 • edited by github-actions bot Loading

Build Passed

VitaliiMaltsev commented Sep 1, 2022 • edited by github-actions bot Loading

Build Passed

github-actions bot commented Sep 1, 2022

evantahler commented Sep 2, 2022

VitaliiMaltsev commented Sep 3, 2022 • edited by github-actions bot Loading

github-actions bot commented Sep 3, 2022

VitaliiMaltsev commented Sep 3, 2022 • edited by github-actions bot Loading

VitaliiMaltsev commented Sep 4, 2022 • edited by github-actions bot Loading

Build Passed

VitaliiMaltsev commented Sep 4, 2022 • edited by github-actions bot Loading

Build Passed

VitaliiMaltsev commented Sep 4, 2022 • edited by github-actions bot Loading

VitaliiMaltsev commented Sep 4, 2022 • edited by github-actions bot Loading

VitaliiMaltsev commented Sep 1, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 1, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 3, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 3, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 4, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 4, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 4, 2022 •

edited by github-actions bot

Loading

VitaliiMaltsev commented Sep 4, 2022 •

edited by github-actions bot

Loading