-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Client] Fix message being ignored when the non-persistent topic reader reconnect. #12348
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…ect. Signed-off-by: Zike Yang <ar@armail.top>
codelipenghui
approved these changes
Oct 13, 2021
codelipenghui
requested review from
eolivelli,
315157973,
hangc0276,
merlimat and
sijie
October 14, 2021 09:57
315157973
approved these changes
Oct 14, 2021
hangc0276
reviewed
Oct 14, 2021
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice Catch!
zeo1995
pushed a commit
to zeo1995/pulsar
that referenced
this pull request
Oct 14, 2021
* up/master: (37 commits) re-enabling integration tests for Sinks (apache#12307) [PIP 95][Issue 12040][web] Topic lookup with listener header (apache#12072) Fix the master CI broken with update dispatch rate block issue (apache#12360) Fix message being ignored when the non-persistent topic reader reconnect. (apache#12348) Fix log format. (apache#12346) [website][upgrade]feat: docs migration - version-2.7.2 Concepts and Architecture (apache#12354) [website][upgrade] feat: full docs migration for version 2.8.0 (apache#12359) [website][upgrade]feat: dynamic replace version info before build (apache#12337) Fix flaky tests: ElasticSearchClientTests (apache#12347) Use asyncCloseCursorLedger to replace cursorLedger.asyncClose method in the ManagedCursorImpl.VoidCallback#operationComplete (apache#12113) fix-npe-ZkBookieRackAffinityMapping (apache#11947) [pulsar-admin] Allow setting --forward-source-message-property to false when updating a pulsar function (apache#12128) [website][upgrade]feat: docs migration - Development (apache#12320) Update delete inactive topic configuration documentation (apache#12350) [PIP 95][Issue 12040][broker] Multiple bind addresses for Pulsar protocol (apache#12056) Added Debezium Source for MS SQL Server (apache#12256) Fix: flaky oracle tests (apache#12306) [C++] Use URL encoded content type for OAuth 2.0 authentication (apache#12341) [C++] Handle OAuth 2.0 exceptional cases gracefully (apache#12335) feat(cli): add restart command to pulsar-daemon (apache#12279) ... # Conflicts: # site2/website-next/docusaurus.config.js # site2/website-next/versioned_sidebars/version-2.7.2-sidebars.json # site2/website-next/versions.json
codelipenghui
pushed a commit
that referenced
this pull request
Oct 15, 2021
…ect. (#12348) Consumers should not perform message ignore checks when consuming messages from non-persistent topics. Otherwise, it may lead to a case where the message is incorrectly ignored when the non-persistent reader(or non-durable subscription non-persistent consumer) reconnects to the broker. Currently, when the reader of the non-persistent topic has its `currentMessageQueue` empty before reconnection, its `startMessageId` is set to `lastDequeuedMessageId` after reconnection: https://github.com/apache/pulsar/blob/4ae7f6a1b38a003c9fc26844e52771b776bf64bf/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L856 and if we specify `startMessageId` as the default value (-1:-1), it will be set to (0:0) after reconnection, thus causes all subsequent messages to be ignored: https://github.com/apache/pulsar/blob/4ae7f6a1b38a003c9fc26844e52771b776bf64bf/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1202-L1211 and for batch messages here: https://github.com/apache/pulsar/blob/4ae7f6a1b38a003c9fc26844e52771b776bf64bf/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1029-L1037 * Remove the message ignore check when the topic of the consumer is non-persistent (cherry picked from commit 867d71c)
nicoloboschi
pushed a commit
to datastax/pulsar
that referenced
this pull request
Oct 27, 2021
…ect. (apache#12348) Consumers should not perform message ignore checks when consuming messages from non-persistent topics. Otherwise, it may lead to a case where the message is incorrectly ignored when the non-persistent reader(or non-durable subscription non-persistent consumer) reconnects to the broker. Currently, when the reader of the non-persistent topic has its `currentMessageQueue` empty before reconnection, its `startMessageId` is set to `lastDequeuedMessageId` after reconnection: https://github.com/apache/pulsar/blob/4ae7f6a1b38a003c9fc26844e52771b776bf64bf/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L856 and if we specify `startMessageId` as the default value (-1:-1), it will be set to (0:0) after reconnection, thus causes all subsequent messages to be ignored: https://github.com/apache/pulsar/blob/4ae7f6a1b38a003c9fc26844e52771b776bf64bf/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1202-L1211 and for batch messages here: https://github.com/apache/pulsar/blob/4ae7f6a1b38a003c9fc26844e52771b776bf64bf/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1029-L1037 * Remove the message ignore check when the topic of the consumer is non-persistent (cherry picked from commit 867d71c) (cherry picked from commit 8e2b003)
bharanic-dev
pushed a commit
to bharanic-dev/pulsar
that referenced
this pull request
Mar 18, 2022
…ect. (apache#12348) ### Motivation Consumers should not perform message ignore checks when consuming messages from non-persistent topics. Otherwise, it may lead to a case where the message is incorrectly ignored when the non-persistent reader(or non-durable subscription non-persistent consumer) reconnects to the broker. Currently, when the reader of the non-persistent topic has its `currentMessageQueue` empty before reconnection, its `startMessageId` is set to `lastDequeuedMessageId` after reconnection: https://github.com/apache/pulsar/blob/4ae7f6a1b38a003c9fc26844e52771b776bf64bf/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L856 and if we specify `startMessageId` as the default value (-1:-1), it will be set to (0:0) after reconnection, thus causes all subsequent messages to be ignored: https://github.com/apache/pulsar/blob/4ae7f6a1b38a003c9fc26844e52771b776bf64bf/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1202-L1211 and for batch messages here: https://github.com/apache/pulsar/blob/4ae7f6a1b38a003c9fc26844e52771b776bf64bf/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L1029-L1037 ### Modifications * Remove the message ignore check when the topic of the consumer is non-persistent
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
cherry-picked/branch-2.8
Archived: 2.8 is end of life
doc-not-needed
Your PR changes do not impact docs
release/2.8.2
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Consumers should not perform message ignore checks when consuming messages from non-persistent topics.
Otherwise, it may lead to a case where the message is incorrectly ignored when the non-persistent reader(or non-durable subscription non-persistent consumer) reconnects to the broker.
Currently, when the reader of the non-persistent topic has its
currentMessageQueue
empty before reconnection, itsstartMessageId
is set tolastDequeuedMessageId
after reconnection:pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
Line 856 in 4ae7f6a
and if we specify
startMessageId
as the default value (-1:-1), it will be set to (0:0) after reconnection, thus causes all subsequent messages to be ignored:pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
Lines 1202 to 1211 in 4ae7f6a
and for batch messages here:
pulsar/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java
Lines 1029 to 1037 in 4ae7f6a
Modifications
Verifying this change
This change is already covered by existing tests, such as testReaderReconnectAndRead and testReaderReconnectAndReadBatchMessages.
Does this pull request potentially affect one of the following parts:
Documentation
This is a bug fix.