-
Notifications
You must be signed in to change notification settings - Fork 4.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Source-postgres]: Add logging in case of multiple records with same LSN #35842
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Ignored Deployment
|
@@ -57,6 +57,7 @@ private ContainerModifier(String methodName) { | |||
|
|||
} | |||
|
|||
@SuppressWarnings("deprecation") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
??
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not quite sure, but this underlying library seems to have a deprecation that is failing the gradle check, so it i's necessary
/publish-java-cdk
|
if (targetPosition.isEventAheadOffset(checkpointOffsetToSend, event)) { | ||
sendCheckpointMessage = true; | ||
} else { | ||
LOGGER.info("Encountered {} records with the same event offset", recordsLastSync); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This log message is causing our airbyte logs to significantly grow in size as we commonly see this message >1 million times in each sync.
Is it possible to selectively disable this log or would we have to change our log level? We don't believe multiple records with the same offset is an issue for us because most of our database operations are batch inserts.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@WillKoehrsen are you able to submit a PR which limits how often this is logged? Maybe only once per offset or stop after 100 logs?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry, I don't have the experience with the codebase to feel comfortable opening a PR. We were able to get around the issue in the short term by downgrading our postgres connector to 3.3.12.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ok, we'll track this bug via #36903
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@WillKoehrsen we have removed this log line in the new version of source-postgres
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot! Really appreciate it.
testExecutionConcurrency=-1 | ||
|
||
JunitMethodExecutionTimeout=5 m |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@akashkulk what's this for?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this PR, CDK 0.23.15 #35827 default timeout was reduced from 5m to 1m by default. It wasn't really validated against any other certified connectors other than source-mssql for which it seems like this fix exclusively applied.
This timeout just adds back that functionality.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No description provided.