-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix hasMessageAvailable return true but can't read message #10414
Conversation
There are many reasons for this problem, this is just one of the scenarios, for example: there are messages in incomeQueue, but they cannot be consumed. there will be follow-up pr to fix this problem. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Cool, LGTM +1
) ### Motivation I temporarily fixed this problem in PR apache#10190. Now we have found a better way, this way can avoid the seek, then avoid trigger another reconnection. Thank you @codelipenghui to troubleshoot this issue with me all night. We have added a lot of log and found that this issue is caused by some race condition problems. Here is the first reason: https://github.com/apache/pulsar/blob/f2d72c9fc13a33df584ec1bd96a4c147774b858d/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1808-L1818 Now we have an acknowledgmentsGroupingTracker to filter duplicate messages, and this Tracker will be cleaned up after seek. However, it is possible that the connection is ready and Broker has pushed message, but `acknowledgmentsGroupingTracker.flushAndClean(); ` has not been executed yet. Finally hasMessageAvailableAsync returns true, but the message cannot be read because it is filtered by the acknowledgmentsGroupingTracker ### Modifications clean the tracker when connection was open ### Verifying this change
### Motivation I temporarily fixed this problem in PR #10190. Now we have found a better way, this way can avoid the seek, then avoid trigger another reconnection. Thank you @codelipenghui to troubleshoot this issue with me all night. We have added a lot of log and found that this issue is caused by some race condition problems. Here is the first reason: https://github.com/apache/pulsar/blob/f2d72c9fc13a33df584ec1bd96a4c147774b858d/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1808-L1818 Now we have an acknowledgmentsGroupingTracker to filter duplicate messages, and this Tracker will be cleaned up after seek. However, it is possible that the connection is ready and Broker has pushed message, but `acknowledgmentsGroupingTracker.flushAndClean(); ` has not been executed yet. Finally hasMessageAvailableAsync returns true, but the message cannot be read because it is filtered by the acknowledgmentsGroupingTracker ### Modifications clean the tracker when connection was open ### Verifying this change (cherry picked from commit f69a03b)
) I temporarily fixed this problem in PR apache#10190. Now we have found a better way, this way can avoid the seek, then avoid trigger another reconnection. Thank you @codelipenghui to troubleshoot this issue with me all night. We have added a lot of log and found that this issue is caused by some race condition problems. Here is the first reason: https://github.com/apache/pulsar/blob/f2d72c9fc13a33df584ec1bd96a4c147774b858d/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1808-L1818 Now we have an acknowledgmentsGroupingTracker to filter duplicate messages, and this Tracker will be cleaned up after seek. However, it is possible that the connection is ready and Broker has pushed message, but `acknowledgmentsGroupingTracker.flushAndClean(); ` has not been executed yet. Finally hasMessageAvailableAsync returns true, but the message cannot be read because it is filtered by the acknowledgmentsGroupingTracker clean the tracker when connection was open (cherry picked from commit f69a03b)
Motivation
I temporarily fixed this problem in PR #10190.
Now we have found a better way, this way can avoid the seek, then avoid trigger another reconnection.
Thank you @codelipenghui to troubleshoot this issue with me all night.
We have added a lot of log and found that this issue is caused by some race condition problems. Here is the first reason:
pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
Lines 1808 to 1818 in f2d72c9
Now we have an acknowledgmentsGroupingTracker to filter duplicate messages, and this Tracker will be cleaned up after seek.
However, it is possible that the connection is ready and Broker has pushed message, but
acknowledgmentsGroupingTracker.flushAndClean();
has not been executed yet.Finally hasMessageAvailableAsync returns true, but the message cannot be read because it is filtered by the acknowledgmentsGroupingTracker
Modifications
clean the tracker when connection was open
Verifying this change