-
Notifications
You must be signed in to change notification settings - Fork 2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug][mongo] Fix MongoDB's rewriteOutputBuffer to emit +I rather than +U(or -U) #2760
Conversation
@Jiabao-Sun , @Shawn-Hx , CC |
+U and +I should have the same processing in the external system. |
@Jiabao-Sun , this +U is without same key data(+I) before, which is not required the changelog semantic. What happens if the downstream need the data before when process +U record? I wonder whether a +U (without +I before) will cause other problems in streaming compute or not? @leonardBang , CC , WDYT? |
@Jiabao-Sun , moreover, if |
-U and +U contains the same key, it should be covered by +U ? |
@Jiabao-Sun , |
I agree with @loserwang1024. Logically, data before highwatermark are snapshot data, which should be seen as +I. Otherwise the downstream operators may see a +U/-U record without +I before. I'm not sure whether this behavior may cause data incorrectness or not, but using +I is correct with no doubt. @Jiabao-Sun |
OK, @loserwang1024 could you help add some tests for this? |
@Jiabao-Sun , thanks a lot. It's not easy to add test now. |
@@ -162,6 +163,7 @@ public void rewriteOutputBuffer( | |||
case INSERT: | |||
case UPDATE: | |||
case REPLACE: | |||
value.put(MongoDBEnvelope.OPERATION_TYPE_FIELD, INSERT.getValue()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
value.put(MongoDBEnvelope.OPERATION_TYPE_FIELD, INSERT.getValue()); | |
value.put(MongoDBEnvelope.OPERATION_TYPE_FIELD, OperationType.INSERT.getValue()); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks a lot , I will do it
a212811
to
6162be5
Compare
6162be5
to
2e10978
Compare
Fix #2759