HDFS-17642. Add target node list, exclude source node list, and exclude target node list parameters to balancer #7127
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…y moves blocks to those nodes
Jtdellaringa
changed the title
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
aQuigs
reviewed
Oct 23, 2024
...-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
Outdated
Show resolved
Hide resolved
...-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
Outdated
Show resolved
Hide resolved
…, clarified when processing excluded source/target lists
aQuigs
reviewed
Oct 23, 2024
...-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/balancer/Balancer.java
Outdated
Show resolved
Hide resolved
…ludeSource or target & excludeTarget options
aQuigs
approved these changes
Oct 23, 2024
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
@aQuigs This PR includes functionality - the change in private long init(List reports) method is where we filter out the allowed/excluded nodes from being targets or or sources |
Ah, got it, I overlooked that piece. LGTM - thanks |
ctrezzo
requested changes
Oct 30, 2024
There was a problem hiding this comment.
Thank you for the work! I left a comment about adding test coverage.
Also please fix the checkstyle issues that this patch adds: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7127/5/artifact/out/results-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
...s-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/balancer/TestBalancer.java
Show resolved
Hide resolved
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
Thanks @Jtdellaringa and @ctrezzo @aQuigs . Make sense to me. Please rebase the source code and fix the checkstyle first. |
ctrezzo
reviewed
Nov 4, 2024
| BalancerParameters.DEFAULT.getIncludedNodes()), false, | ||
| false, p); | ||
| } catch (Exception e) { | ||
| if (!e.getMessage().contains(String.valueOf(ExitStatus.NO_MOVE_BLOCK.getExitCode()))) { |
There was a problem hiding this comment.
Right now the unit tests pass in the cases where:
- no exception is thrown
- an exception with NO_MOVE_BLOCK exit code in the message
Are the above two cases intended behavior?
If 2 is the only expected pass case, then we should either annotate the test with the expected failure or use the junit assertThrows (I forget which version this assert is available in).
There was a problem hiding this comment.
I have updated the tests accordingly, thank you for catching this.
ctrezzo
approved these changes
Nov 4, 2024
There was a problem hiding this comment.
Thanks @Jtdellaringa for the work! It looks good to me. I will wait for yetus to do a full run to see of all issues are resolved and then commit if it looks good.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
|
Committed! Thanks for the work @Jtdellaringa ! |
Jtdellaringa
added a commit
to Jtdellaringa/hadoop
that referenced
this pull request
Nov 7, 2024
…de target node list parameters to balancer (apache#7127) HDFS-17642. Add target node list, exclude source node list, and exclude target node list parameters to balancer (apache#7127) --------- Co-authored-by: Joseph DellAringa <jdellari@linkedin.com>
Jtdellaringa
added a commit
to Jtdellaringa/hadoop
that referenced
this pull request
Nov 7, 2024
…de target node list parameters to balancer (apache#7127) HDFS-17642. Add target node list, exclude source node list, and exclude target node list parameters to balancer (apache#7127) --------- Co-authored-by: Joseph DellAringa <jdellari@linkedin.com>
Jtdellaringa
added a commit
to Jtdellaringa/hadoop
that referenced
this pull request
Nov 9, 2024
…de target node list parameters to balancer (apache#7127) HDFS-17642. Add target node list, exclude source node list, and exclude target node list parameters to balancer (apache#7127) --------- Co-authored-by: Joseph DellAringa <jdellari@linkedin.com>
Jtdellaringa
added a commit
to Jtdellaringa/hadoop
that referenced
this pull request
Nov 14, 2024
…de target node list parameters to balancer (apache#7127) HDFS-17642. Add target node list, exclude source node list, and exclude target node list parameters to balancer (apache#7127) --------- Co-authored-by: Joseph DellAringa <jdellari@linkedin.com>
4 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of PR
Adds target parameter to balancer which will allow only datanodes in the provided list of hosts to be picked as a target node.
Adds excludeSource and excludeTarget parameters to balancer which will exclude specified hosts from being picked as a source or target node.
How was this patch tested?
mvn install -DskipTests
add unit test for parameter check in balancer
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?