YARN-11212. [Federation] Add getNodeToLabels REST APIs for Router. #4614
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
🎊 +1 overall
This message was automatically generated. |
goiri
reviewed
Jul 23, 2022
| public class NodeLabelsInfo { | ||
|
|
||
| @XmlElement(name = "nodeLabelInfo") | ||
| private ArrayList<NodeLabelInfo> nodeLabelsInfo = |
Contributor
Author
There was a problem hiding this comment.
Thanks for your help reviewing the code, I will fix it.
| ClientMethod remoteMethod = new ClientMethod("getNodeToLabels", argsClasses, args); | ||
| Map<SubClusterInfo, NodeToLabelsInfo> nodeToLabelsInfoMap = | ||
| invokeConcurrent(subClustersActive.values(), remoteMethod, NodeToLabelsInfo.class); | ||
| nodeToLabelsInfo = RouterWebServiceUtil.mergeNodeToLabels(nodeToLabelsInfoMap.values()); |
Member
There was a problem hiding this comment.
mergeNodeToLabels() could take Map<SubClusterInfo, NodeToLabelsInfo.
| NodeToLabelsInfo nodeToLabelsInfo = new NodeToLabelsInfo(); | ||
| Map<SubClusterId, SubClusterInfo> subClustersActive; | ||
| try { | ||
| subClustersActive = getActiveSubclusters(); |
Member
There was a problem hiding this comment.
You could do this in a single try, and catch the exception there.
There is no best way to return the error than an empty NodeToLabelsInfo?
|
|
||
| nodeToLabelsInfos.stream().forEach(nodeToLabelsInfo ->{ | ||
| for (Map.Entry<String, NodeLabelsInfo> item : nodeToLabelsInfo.getNodeToLabels().entrySet()) { | ||
| if (nodeToLabels.containsKey(item.getKey())) { |
| import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.NodesInfo; | ||
| import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ResourceInfo; | ||
| import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.ResourceOptionInfo; | ||
| import org.apache.hadoop.yarn.server.resourcemanager.webapp.dao.*; |
| cpuLabels.add(cpuLabel); | ||
|
|
||
| String gpuLabel = "GPU"; | ||
| HashSet<String> gpuLabels = new HashSet<>(); |
Member
There was a problem hiding this comment.
I think we have hash constructors that already take the default values at construction instead of having to add.
Contributor
Author
There was a problem hiding this comment.
Thanks for your suggestion, I will modify the code.
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
goiri
reviewed
Jul 24, 2022
| ClientMethod remoteMethod = new ClientMethod("getNodeToLabels", argsClasses, args); | ||
| Map<SubClusterInfo, NodeToLabelsInfo> nodeToLabelsInfoMap = | ||
| invokeConcurrent(subClustersActive.values(), remoteMethod, NodeToLabelsInfo.class); | ||
| nodeToLabelsInfo = RouterWebServiceUtil.mergeNodeToLabels(nodeToLabelsInfoMap); |
Member
There was a problem hiding this comment.
What if we just do:
return RouterWebServiceUtil.mergeNodeToLabels(nodeToLabelsInfoMap);
Not sure what the default case would look like; but I think the exception should just surface.
Contributor
Author
There was a problem hiding this comment.
I refactored part of the code to print and throw exceptions, so that once an exception occurs, users can immediately find the problem.
|
🎊 +1 overall
This message was automatically generated. |
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
|
@goiri please help me review the code again, thank you very much! |
goiri
reviewed
Jul 25, 2022
| subClustersActive = getActiveSubclusters(); | ||
| } catch (Exception e) { | ||
| LOG.error("Cannot get nodes: {}", e.getMessage()); | ||
| LOG.error("Cannot get nodes: {}", e.getMessage(), e); |
Contributor
Author
There was a problem hiding this comment.
This part does not require full stack trace. I will refactor this part of the code to use invokeConcurrent.
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
|
@goiri Please help me review the code again, thank you very much! |
goiri
reviewed
Jul 26, 2022
| }); | ||
|
|
||
| result.setNodeToLabels(nodeToLabels); | ||
| return result; |
Member
There was a problem hiding this comment.
It seems cleaner to do:
return new NodeToLabelsInfo(nodeToLabels);
| nodeLabels.put("node1", cpuNode); | ||
| nodeLabels.put("node2", gpuNode); | ||
| info.setNodeToLabels(nodeLabels); | ||
| return info; |
Member
There was a problem hiding this comment.
return new NodeToLabelsInfo(nodeLabels);
| } | ||
| } | ||
|
|
||
| public NodeLabelsInfo(HashSet<NodeLabel> nodeLabels) { |
Member
There was a problem hiding this comment.
Set<NodeLabel> nodeLabels or Collection<NodeLabel> nodeLabels
Contributor
Author
There was a problem hiding this comment.
Thanks for your suggestion, I will fix it.
Contributor
Author
There was a problem hiding this comment.
Refactor this method using Collection.
|
💔 -1 overall
This message was automatically generated. |
|
🎊 +1 overall
This message was automatically generated. |
Contributor
Author
|
@goiri Please help me to review the code again, thank you very much! |
goiri
approved these changes
Jul 27, 2022
Contributor
Author
|
@goiri Thanks a lot for your help reviewing the code, I'll follow up with YARN-11220 after this pr is done. |
Contributor
Author
|
@goiri Thanks for your help reviewing the code! |
HarshitGupta11
pushed a commit
to HarshitGupta11/hadoop
that referenced
this pull request
Nov 28, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
JIRA:YARN-11212. [Federation] Add getNodeToLabels REST APIs for Router.