HBASE-28260: Add NO_WRITE_LOCAL flag to WAL file creation #5733
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
🎊 +1 overall
This message was automatically generated. |
jojochuang
reviewed
Feb 29, 2024
Comment on lines
111
to
113
| dfsBuilder.replicate(); | ||
| } | ||
| dfsBuilder.noLocalWrite(); |
There was a problem hiding this comment.
shouldn't this be like the following?
Suggested change
| dfsBuilder.replicate(); | |
| } | |
| dfsBuilder.noLocalWrite(); | |
| dfsBuilder.noLocalWrite(); | |
| } | |
| dfsBuilder.replicate(); |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
jojochuang
approved these changes
Mar 1, 2024
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
By default we use AsyncProtobufLogWriter, so we also need to change the code there. |
Apache9
reviewed
Mar 2, 2024
| DistributedFileSystem.HdfsDataOutputStreamBuilder dfsBuilder = | ||
| (DistributedFileSystem.HdfsDataOutputStreamBuilder) builder; | ||
| dfsBuilder.replicate(); | ||
| if (fs.getConf().getBoolean("hbase.regionserver.wal.avoid-local-writes", false)) { |
There was a problem hiding this comment.
Better declare a String constants for this config.
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Thanks for pointing this out, I've addressed it now |
Apache9
reviewed
Mar 6, 2024
| @@ -180,8 +182,10 @@ public AsyncFSOutput getOutput() { | |||
| protected void initOutput(FileSystem fs, Path path, boolean overwritable, int bufferSize, | |||
| short replication, long blockSize, StreamSlowMonitor monitor) | |||
| throws IOException, StreamLacksCapabilityException { | |||
| boolean noLocalWrite = | |||
There was a problem hiding this comment.
Better add this as a parameter of the initOutput method? So we do not need to write the similar code twice in AsyncProtobufLogWriter and ProtobufLogWriter.
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
bbeaudreault
approved these changes
Mar 7, 2024
Apache9
approved these changes
Mar 10, 2024
charlesconnell
added a commit
to HubSpot/hbase
that referenced
this pull request
Mar 11, 2024
bbeaudreault
pushed a commit
that referenced
this pull request
Mar 11, 2024
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
charlesconnell
added a commit
to HubSpot/hbase
that referenced
this pull request
Mar 11, 2024
Signed-off-by: Bryan Beaudreault <bbeaudreault@apache.org> Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Having all three replicas of each WAL file blocks be non-local to the RegionServer makes it less likely that failures of a co-located RegionServer and DataNode result in a corrupt WAL file. See the JIRA ticket for more details about how this can happen. This new setting is off by default, and can be enabled with
hbase.regionserver.wal.avoid-local-writes. I think off by default makes sense, because this can have performance implications.