Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add incremental fetching using change tracking #156

Open
wants to merge 10 commits into
base: master
Choose a base branch
from

Conversation

toma10
Copy link

@toma10 toma10 commented Apr 3, 2020

Hi.

At my company, I had to add change tracking option to this extractor. Feel free to merge it, change it or delete it.

@pivnicek
Copy link
Contributor

pivnicek commented Apr 3, 2020

Wow, thanks for sharing this with us Tomas! We'll take a look and let you know.

@toma10
Copy link
Author

toma10 commented Apr 9, 2020

Hi.
I fixed some bugs and change some behavior. There was a requirement for the first run to fully load the table. In addition, the use of incrementalFetchingLimit was forbidden because problems could arise if multiple records were inserted at once. Change tracking in this case assigns the same sys_change_version to the inserted rows, which could mean that some new records might not be exported. In addition, the behavior will still be modified so that it does not have to specify an incremntalFetchingColumn, because it is always the primary key + there will be support for multiple primary keys. It is already more custom solution, so it is a question whether it belongs to this extractor. Maybe we could generalize it for this extractor.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants