[fix][broker] Fixed error when delayed messages trackers state grows to >1.5GB #16490
Conversation
merlimat
added
type/bug
| } | ||
|
|
||
| public void increaseCapacity() { | ||
| if (capacity < MAX_SEGMENT_SIZE) { |
There was a problem hiding this comment.
The size of the last segment might also < MAX_SEGMENT_SIZE, we need to consider to update the capacity of the buffer?
There was a problem hiding this comment.
And the last segment size < MAX_SEGMENT_SIZE will also affect the writeLong and readLong method because we always uses the MAX_SEGMENT_SIZE to calculate the index.
There was a problem hiding this comment.
All the segments, after the first one, will be created directly at MAX_SEGMENT_SIZE and will not get expanded/shrinked, only added or removed.
There was a problem hiding this comment.
I think here is possible to add a buffer which is not with MAX_SEGMENT_SIZE ? https://github.com/apache/pulsar/pull/16490/files#diff-ee5391a05253205e8b9bb0f52faa7397326ec0687513204b3bf1b5244d2b8504R45-R50
There was a problem hiding this comment.
Ah, that's true! I need to fix the constructor
👍 Added |
codelipenghui
changed the title
…to >1.5GB (#16490) * Fixed error when delayed messages trackers state grows to >1.5GB * Fixed spotbugs issues * Fixed javadocs * In the constructor, ensure all segments after the first one are of max size * Use poll to figure out where the test is stuck * Added SegmentedLongArray specific unit test * Removed unused imports
…to >1.5GB (#16490) * Fixed error when delayed messages trackers state grows to >1.5GB * Fixed spotbugs issues * Fixed javadocs * In the constructor, ensure all segments after the first one are of max size * Use poll to figure out where the test is stuck * Added SegmentedLongArray specific unit test * Removed unused imports
…to >1.5GB (apache#16490) * Fixed error when delayed messages trackers state grows to >1.5GB * Fixed spotbugs issues * Fixed javadocs * In the constructor, ensure all segments after the first one are of max size * Use poll to figure out where the test is stuck * Added SegmentedLongArray specific unit test * Removed unused imports (cherry picked from commit eca6c4a)
…to >1.5GB (apache#16490) * Fixed error when delayed messages trackers state grows to >1.5GB * Fixed spotbugs issues * Fixed javadocs * In the constructor, ensure all segments after the first one are of max size * Use poll to figure out where the test is stuck * Added SegmentedLongArray specific unit test * Removed unused imports
Motivation
The delayed messages tracker is using a
ByteBufin direct memory to store the priority queue. When the number of messages tracked grows a lot, the array gets doubled up to 1.5 GB. The next size increase will fail because 3.0 GB would be bigger than the max size for ByteBuf, 2GB, since they're indexed by integers.Modification
Instead of using a single
ByteBuf, use a list of buffers with a max segment size.The first buffer will be expanded as normal, though the other buffers will always be of fixed size.
This approach has multiple benefits:
doc-not-needed