-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
PBTree: Implement dual-buffer container for MNode management #12048
Merged
MarcosZyk
merged 12 commits into
apache:master
from
linxt20:PBtreeContainerDivideToTwoBuffer
Feb 23, 2024
Merged
PBTree: Implement dual-buffer container for MNode management #12048
MarcosZyk
merged 12 commits into
apache:master
from
linxt20:PBtreeContainerDivideToTwoBuffer
Feb 23, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
MarcosZyk
changed the title
P btree container divide to two buffer
PBTree: Implement dual-buffer container for MNode management
Feb 21, 2024
MarcosZyk
reviewed
Feb 22, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
PTAL
...va/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/memory/MemoryManager.java
Outdated
Show resolved
Hide resolved
...va/org/apache/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/memory/MemoryManager.java
Outdated
Show resolved
Hide resolved
...tdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/CachedMNodeContainer.java
Outdated
Show resolved
Hide resolved
.../iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/IMNodeChildBuffer.java
Outdated
Show resolved
Hide resolved
…the BufferIterator into CachedMNodeContainerIterator
MarcosZyk
reviewed
Feb 23, 2024
...tdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/CachedMNodeContainer.java
Outdated
Show resolved
Hide resolved
...tdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/CachedMNodeContainer.java
Outdated
Show resolved
Hide resolved
MarcosZyk
reviewed
Feb 23, 2024
.../iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/IMNodeChildBuffer.java
Outdated
Show resolved
Hide resolved
...e/iotdb/db/schemaengine/schemaregion/mtree/impl/pbtree/mnode/container/MNodeChildBuffer.java
Outdated
Show resolved
Hide resolved
MarcosZyk
reviewed
Feb 23, 2024
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Bravo work! LGTM~
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This work is part of Pbtree's internal and external memory collaborative concurrency control work. It mainly replaces the original single buffer processing of ChildrenContainer with dual buffer alternating processing to achieve more efficient concurrency control.
The main ideas for implementation at this stage are as follows: planning the abstract base class MNodeChildBuffer, which inherits from IMNodeContainer. Among them, FlushingBuffer is used to store old nodes, and ReceivingBuffer is used to store newly created or modified nodes. Through the combination of the two Buffers, basic Map processing functions are provided to the outside world. Thus, both NewChildBuffer and UpdateChildBuffer in CachedMNodeContainer can be modified to use the MNodeChildBuffer type.
However, compared with the original single Buffer situation, using double buffers requires handling more complex situations:
The first point is that due to the different functions of FlushingBuffer and ReceivingBuffer, write functions such as put are written directly to ReceivingBuffer, while flushing is processed directly in FlushingBuffer. The data exchange between the two parts occurs before flushing. At this time, the nodes in the ReceivingBuffer need to be handed over to the FlushingBuffer.
Secondly, the original data is stored in the same Buffer. Since the Buffer is a Map structure, it can ensure that the contents will not be repeated. After changing to double Buffer, FlushingBuffer and Receiving can respectively ensure that their internal nodes will not be repeated, but nodes in FlushingBuffer may appear in ReceivingBuffer. Overlap needs to be considered when counting statistics
Subsequently, since NewChildBuffer is a Buffer used to receive newly created nodes, there will be no overlap between FlushingBuffer and ReceivingBuffer. This is because the names of new nodes will not be repeated. The situation with UpdateChildBuffer is different. This is a buffer used to accept modified nodes, so the same node can be processed multiple times. There is a possibility of overlap between FlushingBuffer and ReceivingBuffer. Therefore, in response to this phenomenon, two subclasses are used to manage the two Buffers respectively, inheriting from the abstract base class MNodeChildBuffer.
To pave the way for subsequent internal and external memory merging and sorting, here when obtaining the pointer of MNodeChildBuffer, it is required to be able to deduplicate and sort. Consider using the merge sorting algorithm here and using the trygetnext mechanism to gradually complete the merging steps while continuously obtaining next.
For the newly implemented double buffer alternating processing mechanism, MNodeChildBufferTest was designed to detect the rationality of its basic function implementation. At the same time, targeted modifications were made to the single buffer call in the original UT, and the corresponding test cases were improved.