Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[source-mysql] : don't emit final state if there is an underlying stream failure #34881

Closed
akashkulk opened this issue Feb 6, 2024 · 1 comment · Fixed by #34869
Closed
Assignees
Labels
team/db-dw-sources Backlog for Database and Data Warehouse Sources team

Comments

@akashkulk
Copy link
Contributor

akashkulk commented Feb 6, 2024

Topic

No response

Relevant information

Context : https://airbytehq-team.slack.com/archives/C043JHEEYKG/p1707152897893279

We're seeing failures in source-mysql where :

  1. A stream fails
  2. State iterator emits final state message (indicating that the snapshot is complete)
  3. Airbyte proceeds to perform incremental syncs.

This causes data loss, as 2 should emit an intermediate state message. We've lost all the data since the first failure.

The fix is to change the logic :

  1. If a stream snapshot has failed, emit the intermediate message OR fail
  2. Throw an exception to prevent the sync from progressing further

This would be inline with how source-mongo & source-postgres deal with these failures. Furthermore, as SourceStateIterator will be used as the base class for emitting state counts for Postgres & Mongo, this behavior should be standardized.

@akashkulk
Copy link
Contributor Author

This was a regression introduced in version 3.3.1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
team/db-dw-sources Backlog for Database and Data Warehouse Sources team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants