New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Allow additional properties in db stream state #15538

Merged

tuliren merged 9 commits into master from liren/prepare-for-record-count-emission

Aug 11, 2022

Contributor

tuliren commented Aug 11, 2022 •

edited

Loading

What

I am going to introduce a new field in DbStreamState for SQL Sources should include previous sync's cursor value in SELECT statement as not to skip rows added mid-sync #14732.
This PR prepares the main JDBC source connectors for that change by allowing additional properties in the JSON schema.
- Postgres
- MySQL
- MSSQL
In this way, if anything goes wrong with the new field, there is a version that users can rollback to without breaking existing connectors.

🚨 User Impact 🚨

None expected.


          Allow additional properties in db stream state

3248ce3

github-actions bot added the area/connectors label

octavia-squidington-iii added the connectors/source/relational-db label

tuliren added 3 commits

August 10, 2022 20:08


          Bump postgres version

0f97f8e


          Bump mysql version


          Bump mssql version

638e952

github-actions bot added the area/documentation label

octavia-squidington-iii added connectors/source/mssql connectors/source/mysql connectors/source/postgres connectors/source/mssql-strict-encrypt connectors/source/mysql-strict-encrypt connectors/source/postgres-strict-encrypt labels

tuliren commented

View reviewed changes

...yte-integrations/connectors/source-relational-db/src/main/resources/db_models/db_models.yaml

@@ @@ -27,7 +27,7 @@ definitions: @@
                         "$ref": "#/definitions/DbStreamState"
                 DbStreamState:
                   type: object
-                  additionalProperties: false
+                  additionalProperties: true

Contributor Author

tuliren Aug 11, 2022

This is the main change.

tuliren marked this pull request as ready for review

August 11, 2022 03:10

tuliren requested a review from a team as a code owner

August 11, 2022 03:10

tuliren requested a review from evantahler

August 11, 2022 03:11

This was referenced Aug 11, 2022

SQL Sources should include previous sync's cursor value in SELECT statement as not to skip rows added mid-sync #14732

Closed

🎉 JDBC sources: emit cursor counts #15535

Merged

evantahler approved these changes

View reviewed changes

Contributor

evantahler left a comment

👍

There are a lot of PRs these days turning additionalProperties to true. Should we do that everywhere?

Contributor Author

tuliren commented Aug 11, 2022 •

edited

Loading

There are a lot of PRs these days turning additionalProperties to true. Should we do that everywhere?

I think so. Anything with additionalProperties = false is not forward compatible, preventing users from downgrading if there is anything wrong with the new version. I really don't see the harm for setting additionalProperties to true universally.


          Merge branch 'master' into liren/prepare-for-record-count-emission

b36a64c

Contributor Author

tuliren commented Aug 11, 2022 •

edited by github-actions bot

Loading

/publish connector=connectors/source-postgres-strict-encrypt

🕑 Publishing the following connectors:
connectors/source-postgres-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2842311963

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Contributor Author

tuliren commented Aug 11, 2022 •

edited by github-actions bot

Loading

/publish connector=connectors/source-postgres

🕑 Publishing the following connectors:
connectors/source-postgres
https://github.com/airbytehq/airbyte/actions/runs/2842312426

Connector	Did it publish?	Were definitions generated?
connectors/source-postgres	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Contributor Author

tuliren commented Aug 11, 2022 •

edited by github-actions bot

Loading

/publish connector=connectors/source-mysql-strict-encrypt

🕑 Publishing the following connectors:
connectors/source-mysql-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2842313615

Connector	Did it publish?	Were definitions generated?
connectors/source-mysql-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Contributor Author

tuliren commented Aug 11, 2022 •

edited by github-actions bot

Loading

/publish connector=connectors/source-mysql

🕑 Publishing the following connectors:
connectors/source-mysql
https://github.com/airbytehq/airbyte/actions/runs/2842314120

Connector	Did it publish?	Were definitions generated?
connectors/source-mysql	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Contributor Author

tuliren commented Aug 11, 2022 •

edited by github-actions bot

Loading

/publish connector=connectors/source-mssql-strict-encrypt

🕑 Publishing the following connectors:
connectors/source-mssql-strict-encrypt
https://github.com/airbytehq/airbyte/actions/runs/2842314611

Connector	Did it publish?	Were definitions generated?
connectors/source-mssql-strict-encrypt	✅	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

Contributor Author

tuliren commented Aug 11, 2022 •

edited by github-actions bot

Loading

/publish connector=connectors/source-mssql

🕑 Publishing the following connectors:
connectors/source-mssql
https://github.com/airbytehq/airbyte/actions/runs/2842315365

Connector	Did it publish?	Were definitions generated?
connectors/source-mssql	❌	❌

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

From @tuliren: the failure seems transient. Will retry.

🕑 Publishing the following connectors:
connectors/source-mssql
https://github.com/airbytehq/airbyte/actions/runs/2842315365

Connector	Did it publish?	Were definitions generated?
connectors/source-mssql	✅	✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️


          auto-bump connector version [ci skip]

b0256b4

tuliren mentioned this pull request

Standardize spec.json for DB connectors that support log-based CDC replication #14910

Closed

37 tasks

octavia-squidington-iii added 2 commits

August 11, 2022 20:54


          auto-bump connector version [ci skip]

7d4ce83


          Merge branch 'liren/prepare-for-record-count-emission' of https://git…

708073b

…hub.com/airbytehq/airbyte into liren/prepare-for-record-count-emission

octavia-squidington-iii temporarily deployed to more-secrets

August 11, 2022 20:56

Inactive


          auto-bump connector version [ci skip]

tuliren merged commit c2152f4 into master

tuliren deleted the liren/prepare-for-record-count-emission branch

August 11, 2022 22:09

octavia-squidington-iii mentioned this pull request

Bump Airbyte version from 0.40.0-alpha to 0.40.1-alpha #15775

Closed

evantahler mentioned this pull request

Bump Airbyte version from 0.40.0-alpha to 0.40.1 #15857

Merged

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/connectors area/documentation connectors/source/mssql connectors/source/mssql-strict-encrypt connectors/source/mysql connectors/source/mysql-strict-encrypt connectors/source/postgres connectors/source/postgres-strict-encrypt connectors/source/relational-db