Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

MXNET_KVSTORE_USETREE problems #13341

Open
larroy opened this issue Nov 20, 2018 · 4 comments
Open

MXNET_KVSTORE_USETREE problems #13341

larroy opened this issue Nov 20, 2018 · 4 comments
Labels
Bug KVStore

Comments

@larroy
Copy link
Contributor

larroy commented Nov 20, 2018

Description

Related to #12994

  • Kernighan-Lin never succeeds, always goes into exponential algorithm
  • Binary Tree depth is calculated wrong (mxnet::kvstore::ComputeDepth)
  • Fails to find a tree in p3.16xlarge and DGX1

Environment info (Required)

MXNET_KVSTORE_USETREE=1 in multi-gpu nodes
larroy added a commit to larroy/mxnet that referenced this issue Nov 20, 2018
@larroy
Mitigate apache#13341
f496894 
- KL never succeeds so it always goes exponential
- Too many weight matrices were rejected because of zero weights, simplify generation to not include 0 weight edges
@larroy larroy mentioned this issue Nov 20, 2018
Merged
5 tasks
@vdantu
Copy link
Contributor

vdantu commented Nov 20, 2018 •
edited
Loading

@mxnet-label-bot add [C++]

@marcoabreu marcoabreu added the C++ Related to C++ label Nov 20, 2018
larroy added a commit to larroy/mxnet that referenced this issue Nov 20, 2018
@larroy
Mitigate apache#13341
4900f7c 
- KL never succeeds so it always goes exponential
- Too many weight matrices were rejected because of zero weights, simplify generation to not include 0 weight edges
larroy added a commit to larroy/mxnet that referenced this issue Nov 20, 2018
@larroy
Mitigate apache#13341
9f6b9cd 
- KL never succeeds so it always goes exponential
- Too many weight matrices were rejected because of zero weights, simplify generation to not include 0 weight edges
larroy added a commit to larroy/mxnet that referenced this issue Nov 26, 2018
@larroy
Mitigate apache#13341
deae581 
- KL never succeeds so it always goes exponential
- Too many weight matrices were rejected because of zero weights, simplify generation to not include 0 weight edges
larroy added a commit to larroy/mxnet that referenced this issue Nov 27, 2018
@larroy
Mitigate apache#13341
b637747 
- KL never succeeds so it always goes exponential
- Too many weight matrices were rejected because of zero weights, simplify generation to not include 0 weight edges
@marcoabreu
Copy link
Contributor

#13343

marcoabreu pushed a commit that referenced this issue Nov 29, 2018
@larroy @marcoabreu
Mitigate #13341 (#13343)
dcd8950 
- KL never succeeds so it always goes exponential
- Too many weight matrices were rejected because of zero weights, simplify generation to not include 0 weight edges
@larroy
Copy link
Contributor Author

larroy commented Nov 29, 2018

issue remains, let's reopen

@marcoabreu marcoabreu reopened this Nov 29, 2018
aaronmarkham pushed a commit to aaronmarkham/incubator-mxnet that referenced this issue Nov 30, 2018
@larroy @aaronmarkham
Mitigate apache#13341 (apache#13343)
417909a 
- KL never succeeds so it always goes exponential
- Too many weight matrices were rejected because of zero weights, simplify generation to not include 0 weight edges
@larroy larroy mentioned this issue Dec 17, 2018
Merged
7 tasks
@leleamol
Copy link
Contributor

leleamol commented Jan 4, 2019

@mxnet-label-bot update [KVStore, Bug]

@marcoabreu marcoabreu added Bug KVStore and removed C++ Related to C++ labels Jan 4, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Bug KVStore
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@larroy @marcoabreu @leleamol @vdantu