-
Notifications
You must be signed in to change notification settings - Fork 4.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
🎉 Source MSSQL: implementation for CDC #4543
Conversation
This reverts commit 20c6768.
/test connector=source-mssql
|
/test connector=source-mssql
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull in the latest changes from master to fix compilation error.
Also I was playing around with the property decimal.handling.mode
and it worked without upgrading the debezium version. so perhaps we can stick to this version right? What do you think
} | ||
|
||
@Override | ||
public List<AutoCloseableIterator<AirbyteMessage>> getIncrementalIterators(JsonNode config, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This would cause compilation error, in master the signature of the method has been changed
* {@link io.debezium.config.CommonConnectorConfig#DEFAULT_MAX_BATCH_SIZE} is 2048 (so 20,480) | ||
* {@link io.debezium.config.CommonConnectorConfig#DEFAULT_MAX_QUEUE_SIZE} is 8192 (so 81,920) | ||
*/ | ||
final LinkedBlockingQueue<ChangeEvent<String, String>> queue = new LinkedBlockingQueue<>(100000); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As discussed we should probably stick with the default numbers so queue should be of 10k size
This PR is superseded by #4689 |
Closed completed by #4689 |
What
Related issue: #2848
How
MSSQL CDC follows conventions laid out by Postgres and MySql sources, therefore is built on the debezium SQL Server connector. The CDC works using MSSQL's CDC tables and the LSN value as an offset.
Some things worth pointing out:
Pre-merge Checklist
Expand the checklist which is relevant for this PR.
Connector checklist
./gradlew :airbyte-integrations:connectors:<name>:integrationTest
./test connector=connectors/<name>
command as documented here is passing.docs/integrations/<source or destination>/<name>
.docs/integrations/...
. See changelog example/publish
command described here