[PIP-132] Include message header size when check maxMessageSize for non-batch message on the client side. #14007
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Jason918
requested review from
michaeljmarshall,
eolivelli and
merlimat
and removed request for
michaeljmarshall and
eolivelli
Jason918
requested review from
michaeljmarshall,
eolivelli,
codelipenghui,
hangc0276,
merlimat and
315157973
and removed request for
merlimat
|
/pulsarbot run-failure-checks |
|
/pulsarbot run-failure-checks |
aloyszhang
reviewed
Feb 9, 2022
pulsar-client/src/main/java/org/apache/pulsar/client/impl/ProducerImpl.java
Outdated
Show resolved
Hide resolved
|
/pulsarbot run-failure-checks |
codelipenghui
approved these changes
Feb 24, 2022
pulsar-client/src/test/java/org/apache/pulsar/client/util/MathUtilsTest.java
Outdated
Show resolved
Hide resolved
|
/pulsarbot run-failure-checks |
|
@RobertIndie PTAL |
gaozhangmin
pushed a commit
to gaozhangmin/pulsar
that referenced
this pull request
Mar 8, 2022
…on-batch message on the client side. (apache#14007) Master Issue: apache#13591 ### Motivation See apache#13591 ### Modifications 1. Add max message size check before sending request for non-batch and non-chunked messages. 2. Decrease chunk size by metadata size for chunked messages.
Nicklee007
pushed a commit
to Nicklee007/pulsar
that referenced
this pull request
Apr 20, 2022
…on-batch message on the client side. (apache#14007) Master Issue: apache#13591 ### Motivation See apache#13591 ### Modifications 1. Add max message size check before sending request for non-batch and non-chunked messages. 2. Decrease chunk size by metadata size for chunked messages.
BewareMyPower
added a commit
to BewareMyPower/pulsar
that referenced
this pull request
Jun 23, 2022
…hunks after PIP-132 Fixes apache#16195 ### Motivation [PIP-132](apache#14007) considers the message metadata size when computing the payload chunk size and the number of chunks. However, it could make some messages whose size is less than `maxMessageSize` cannot be sent. There are two reasons: 1. The `MessageMetadata` will be updated after computing the payload chunk size, i.e. the actual metadata size would be greater. 2. `OpSendMsg#getMessageHeaderAndPayloadSize` doesn't exclude all bytes other than the metadata and payload, e.g. the 4 bytes checksum field. For example, if the max message size is 100, send a string whose size is 60 with chunking enabled. 1. The initial metadata size is 25 so the chunk size is 75, the message won't be spit into chunks. 2. After `serializeAndSendMessage`, the metadata size becomes 32, so the serialized header's total size is 4 + 8 + 6 + 4 + 32 = 54, and the total size is 54 + 60 = 114, see `headerContentSize` in `serializeCommandSendWithSize`. 3. In `getMessageHeaderAndPayloadSize`, the returned value is computed by 114 - 8 - 4 = 102 > 100. The 6 bytes magic and checksum and 4 bytes metadata length field are not included. ### Modifications - Update the message metadata before computing the chunk size. - Compute the correct size in `getMessageHeaderAndPayloadSize`. ### Verifying this change Add `testChunkSize` to verify all sizes in range [1, maxMessageSize] can be sent successfully when chunking is enabled.
Merged
4 tasks
BewareMyPower
added a commit
that referenced
this pull request
Jun 28, 2022
…hunks after PIP-132 (#16196) Fixes #16195 ### Motivation [PIP-132](#14007) considers the message metadata size when computing the payload chunk size and the number of chunks. However, it could make some messages whose size is less than `maxMessageSize` cannot be sent. There are two reasons: 1. The `MessageMetadata` will be updated after computing the payload chunk size, i.e. the actual metadata size would be greater. 2. `OpSendMsg#getMessageHeaderAndPayloadSize` doesn't exclude all bytes other than the metadata and payload, e.g. the 4 bytes checksum field. For example, if the max message size is 100, send a string whose size is 60 with chunking enabled. 1. The initial metadata size is 25 so the chunk size is 75, the message won't be spit into chunks. 2. After `serializeAndSendMessage`, the metadata size becomes 32, so the serialized header's total size is 4 + 8 + 6 + 4 + 32 = 54, and the total size is 54 + 60 = 114, see `headerContentSize` in `serializeCommandSendWithSize`. 3. In `getMessageHeaderAndPayloadSize`, the returned value is computed by 114 - 8 - 4 = 102 > 100. The 6 bytes magic and checksum and 4 bytes metadata length field are not included. ### Modifications - Update the message metadata before computing the chunk size. - Compute the correct size in `getMessageHeaderAndPayloadSize`. ### Verifying this change Add `testChunkSize` to verify all sizes in range [1, maxMessageSize] can be sent successfully when chunking is enabled.
2 tasks
2 tasks
Merged
13 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Master Issue: #13591
Motivation
See #13591
Modifications
Verifying this change
This change added tests and can be verified as follows:
Does this pull request potentially affect one of the following parts:
If
yeswas chosen, please highlight the changesDocumentation
Check the box below and label this PR (if you have committer privilege).
Need to update docs?
no-need-doc