Fix semaphore and memory leak when chunks failed to enqueue #13454
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
BewareMyPower
added
type/bug
BewareMyPower
requested review from
merlimat,
lhotari,
rdhabalia,
eolivelli,
315157973 and
codelipenghui
hezhangjian
approved these changes
Dec 23, 2021
pulsar-broker/src/test/java/org/apache/pulsar/client/impl/MessageChunkingTest.java
Show resolved
Hide resolved
codelipenghui
approved these changes
Dec 23, 2021
Technoboy-
pushed a commit
to Technoboy-/pulsar
that referenced
this pull request
Dec 29, 2021
…3454) ### Motivation When a large message is sent by chunks, each chunk needs to reserve a spot of the semaphore. However, when it failed, the already reserved memory from limiter and spots from semaphore are not released. ### Modifications - Release the semaphore and memory when `canEnqueueRequest` returns false for chunks. - Add `testChunksEnqueueFailed` to cover this case. It sends a large message whose number of chunks is greater than the `maxPendingMessages` so that the first time `canEnqueueRequest` returns true while the following `canEnqueueRequest` calls will return false.
codelipenghui
pushed a commit
that referenced
this pull request
Dec 30, 2021
### Motivation When a large message is sent by chunks, each chunk needs to reserve a spot of the semaphore. However, when it failed, the already reserved memory from limiter and spots from semaphore are not released. ### Modifications - Release the semaphore and memory when `canEnqueueRequest` returns false for chunks. - Add `testChunksEnqueueFailed` to cover this case. It sends a large message whose number of chunks is greater than the `maxPendingMessages` so that the first time `canEnqueueRequest` returns true while the following `canEnqueueRequest` calls will return false. (cherry picked from commit 2e2cd57)
codelipenghui
pushed a commit
that referenced
this pull request
Dec 30, 2021
### Motivation When a large message is sent by chunks, each chunk needs to reserve a spot of the semaphore. However, when it failed, the already reserved memory from limiter and spots from semaphore are not released. ### Modifications - Release the semaphore and memory when `canEnqueueRequest` returns false for chunks. - Add `testChunksEnqueueFailed` to cover this case. It sends a large message whose number of chunks is greater than the `maxPendingMessages` so that the first time `canEnqueueRequest` returns true while the following `canEnqueueRequest` calls will return false. (cherry picked from commit 2e2cd57)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
When a large message is sent by chunks, each chunk needs to reserve a spot of the semaphore. However, when it failed, the already reserved memory from limiter and spots from semaphore are not released.
Modifications
canEnqueueRequestreturns false for chunks.testChunksEnqueueFailedto cover this case. It sends a large message whose number of chunks is greater than themaxPendingMessagesso that the first timecanEnqueueRequestreturns true while the followingcanEnqueueRequestcalls will return false.Verifying this change
This change added tests
testChunksEnqueueFailed.