HBASE-28596 Optimise BucketCache usage upon regions splits/merges. #5906
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
wchevreuil
requested review from
bbeaudreault,
ankitsinghal,
petersomogyi,
Apache9 and
comnetwork
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Change-Id: I510be0cb8041fead191f965178fd7336b6274352
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Change-Id: Ib4454ed5de44ce35271e53a88caf7fce3b2d05ad
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Apache9
reviewed
May 20, 2024
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
Show resolved
Hide resolved
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
Show resolved
Hide resolved
Change-Id: I970b6179aa4ccfe89dcd8eb7bb8eedbfc77037d5
|
🎊 +1 overall
This message was automatically generated. |
Change-Id: I6dd46a801a44d9da8a824d10d57b88fb3c8cf0da
Change-Id: Ie893a0ef25bedc252a475b0f80f13beb28861e62
Change-Id: I8c88c9dd0eeefbb276c6d5f29b7c485df5482f3c
Thanks for the initial review, I have added extra comments and javadocs to help with reviewing. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Change-Id: I94f27b3e5f6787a1bc159693805c0c8929a8fcea
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Just checked on these latest failures, I believe those are unrelated flakeys. |
Apache9
reviewed
Jun 12, 2024
| @Override | ||
| public void close(boolean evictOnClose) throws IOException { | ||
| if (closed.compareAndSet(false, true)) { | ||
| if (evictOnClose && StoreFileInfo.isReference(this.reader.getPath())) { |
There was a problem hiding this comment.
Do we really need to check isReference here? We will only use HalfStoreFileReader when the store file is a reference file?
There was a problem hiding this comment.
Yeah, I think there's no need for this extra check, as we'll always have a reference with HalfStoreFileReader. Let me remove this.
| @@ -244,6 +248,45 @@ public static int getMaxCachedBlocksByFile(Configuration conf) { | |||
| return conf == null ? DEFAULT_MAX : conf.getInt("hbase.ui.blockcache.by.file.max", DEFAULT_MAX); | |||
| } | |||
|
|
|||
| /** | |||
| * Similarly to HFileBlock.Writer.getBlockForCaching(), creates a HFileBlock instance without | |||
There was a problem hiding this comment.
Could we share some common code between these two methods? Otherwise how can we align these two methods?
There was a problem hiding this comment.
Yes, let me work on this.
There was a problem hiding this comment.
So the most I could reuse here is the new HFIleContext part. The block builder portion is hard to reuse because in HFileBlock.Writer the parameters are calculated on the go, so any sort of reuse would require an awful lot of parameters.
| @@ -45,7 +45,7 @@ public HFilePreadReader(ReaderContext context, HFileInfo fileInfo, CacheConfig c | |||
| }); | |||
|
|
|||
| // Prefetch file blocks upon open if requested | |||
| if (cacheConf.shouldPrefetchOnOpen() && cacheIfCompactionsOff() && shouldCache.booleanValue()) { | |||
| if (cacheConf.shouldPrefetchOnOpen() && shouldCache.booleanValue()) { | |||
There was a problem hiding this comment.
Why removing cacheIfCompactionsOff?
There was a problem hiding this comment.
Same reason as explained here: https://github.com/apache/hbase/pull/5906/files#r1606971440. We want the prefetch to run even for refs or links, the cache implementation should decide what to do with refs/links.
There was a problem hiding this comment.
Better add some comments to explain the history here?
| public void close(boolean evictOnClose) throws IOException { | ||
| if (closed.compareAndSet(false, true)) { | ||
| if (evictOnClose && StoreFileInfo.isReference(this.reader.getPath())) { | ||
| final HFileReaderImpl.HFileScannerImpl s = |
There was a problem hiding this comment.
Usually it is not a good idea to the implementation class directly at upper layer. Why we need to use HFileScannerImpl here?
There was a problem hiding this comment.
We need to seek to the split cell, in order to calculate offsets for the cache keys of referred file to evict.
There was a problem hiding this comment.
But there is a seekTo method in HFileScanner interface? I just mean why we need the cast here?
There was a problem hiding this comment.
I need the HFileScannerImpl.getCurBlock for figuring out the offset.
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java
Show resolved
Hide resolved
Change-Id: If712a4310a8f651b2395dd233fec7f8430362da5
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Change-Id: I480a6a4f062d1d57d8a432d93d9f9c4e62c9c826
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
Any further concerns, @Apache9 ? |
wchevreuil
added a commit
that referenced
this pull request
Jun 19, 2024
…5906) Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Reviewed-by: Duo Zhang <zhangduo@apache.org>
wchevreuil
added a commit
that referenced
this pull request
Jun 21, 2024
…5906) Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Reviewed-by: Duo Zhang <zhangduo@apache.org>
wchevreuil
added a commit
to wchevreuil/hbase
that referenced
this pull request
Dec 2, 2024
…pache#5906) Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org> Reviewed-by: Duo Zhang <zhangduo@apache.org> Change-Id: I6d6fc84c0e526cc1ed1c14072791b0bd5b9b92fa
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.