HBASE-27028 Add a shell command for flushing master local region #4457
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Apache9
requested changes
May 22, 2022
Thanks for the review @Apache9. You are right, it may conflict with the user's 'master: store' table. |
2005hithlj
force-pushed
the
HBASE-27028
branch
2 times, most recently
from
5292647 to
21f9cc4
Compare
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
The failures are caused by #4442 |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@carp84 Could you take a look? Thanks. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
Apache9
reviewed
May 28, 2022
There was a problem hiding this comment.
So the implementation is just to request a flush, not complete the flush, but the method name is called flush and we returns a CompletableFuture, which seems to tell users that joining on the future then you can wait for the flush to complete...
Better have a more clear design and naming of the method name and behavior.
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AdminOverAsyncAdmin.java
Show resolved
Hide resolved
| @@ -154,6 +154,10 @@ public RegionScanner getRegionScanner(Scan scan) throws IOException { | |||
| return region.getScanner(scan); | |||
| } | |||
|
|
|||
| public void rpcRequestFlush() { | |||
There was a problem hiding this comment.
Just call it requestFlush?
|
Thanks @Apache9 for the review. I will revise it according to your suggestion. |
Apache9
reviewed
May 31, 2022
| /** | ||
| * Force request flush master local region | ||
| */ | ||
| void forceRequestFlushMasterStore() throws IOException; |
There was a problem hiding this comment.
I mean add a force flag... Not add a force to the method name...
Checked the code, we do not have this flag in the Admin interface, and at rs side, it will always use true when calling region.flush...
So sorry, let's keep align with other methods in this interface, just call it flushMasterStore, and at master side, we call region.flush(true).
There was a problem hiding this comment.
Thanks @Apache9 for the review.
If we call region.flush(true) at the master side, the changesAfterLastFlush value of MasterRegionFlusherAndCompactor will become incorrect, which may trigger an invalid flush operation.
But we can call MasterRegionFlusherAndCompactor.requestFlush() to avoid this.
Do we need to consider this situation?
There was a problem hiding this comment.
Then we could introduce a method in MasterRegionFlusherAndCompactor? Just reset changesAfterLastFlush after calling flush?
Anyway, what I mean is we should try our best to align the behavior in our Admin interface.
There was a problem hiding this comment.
Yes, we need to introduce a new method to reset the value of changesAfterLastFlush after calling flush.
Let me modify the code here. Thanks @Apache9 .
There was a problem hiding this comment.
I checked the code, because region.flush() is a synchronous method, we should call resetChangesAfterLastFlush() before calling region.flush() method, which should be similar to MasterRegionFlusherAndCompactor.flushLoop() method code logic. @Apache9
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
I have a new commit, could you take a look ? |
Apache9
reviewed
Jun 7, 2022
| @@ -457,6 +457,9 @@ public class HMaster extends HBaseServerBase<MasterRpcServices> implements Maste | |||
| public static final String WARMUP_BEFORE_MOVE = "hbase.master.warmup.before.move"; | |||
| private static final boolean DEFAULT_WARMUP_BEFORE_MOVE = true; | |||
|
|
|||
| // Only for testing | |||
| private boolean isLocalRegionFlushed = false; | |||
There was a problem hiding this comment.
Is it possible to not adding a flag which is only used to verify test result? I think you can add a CP method for the flushMasterRegion call, and use attach a CP to verify that whether the master is called.
There was a problem hiding this comment.
Does CP here mean coprocessor? Is there a similar implementation example? sir @Apache9
| @@ -155,7 +158,16 @@ public RegionScanner getRegionScanner(Scan scan) throws IOException { | |||
| } | |||
|
|
|||
| public FlushResult flush(boolean force) throws IOException { | |||
| return region.flush(force); | |||
| flusherAndCompactor.resetChangesAfterLastFlush(); | |||
There was a problem hiding this comment.
We do not need to record the lastFlushTime here?
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
@Apache9 I have a new commit, could you take a look? Thanks. |
|
💔 -1 overall
This message was automatically generated. |
Apache9
approved these changes
Jun 9, 2022
There was a problem hiding this comment.
Overall LGTM. Just a naming problem. Will merge after you fix it.
Thanks @2005hithlj for the patient.
| changesAfterLastFlush.set(0); | ||
| } | ||
|
|
||
| void resetLastFlushTime() { |
There was a problem hiding this comment.
Should name it recordLastFlushTime?
There was a problem hiding this comment.
Thanks @Apache9 for the review.
I have modified it in the latest commit.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
Checked the output, I think the problem is that, we run protoc test in the sub module directory so the session.executionRootDirectory is pointed to the sub module then we can not find license-header file. Let me file an issue about this. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
In my local environment, the UT TestReplicationDroppedTables can be executed successfully. |
wenwj0
pushed a commit
to wenwj0/hbase
that referenced
this pull request
Jun 14, 2022
…che#4457) Signed-off-by: Duo Zhang <zhangduo@apache.org>
wenwj0
added a commit
to wenwj0/hbase
that referenced
this pull request
Jun 14, 2022
…ion (apache#4457)" This reverts commit 958ecc3.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
https://issues.apache.org/jira/browse/HBASE-27028