perf: improve lightgbm training performance 4x-10x by setting num_threads to be cores-1 by default for single dataset mode #1282
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
Codecov Report
@@ Coverage Diff @@
## master #1282 +/- ##
==========================================
+ Coverage 83.33% 83.37% +0.04%
==========================================
Files 300 300
Lines 13828 13830 +2
Branches 672 675 +3
==========================================
+ Hits 11523 11531 +8
+ Misses 2305 2299 -6
Continue to review full report at Codecov.
|
|
An added benefit seems to be that the lightgbm tests finish faster as well - the lightgbm1 unit tests finished in 11 min in this build and took 33 min in previous build 😅 |
…eads to be cores-1
imatiach-msft
force-pushed
the
ilmat/def-num-threads
branch
from
0ffee85 to
6fb7fc4
Compare
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
mhamilton723
approved these changes
Dec 2, 2021
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
|
/azp run |
|
Azure Pipelines successfully started running 1 pipeline(s). |
This was referenced Dec 27, 2021
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
In benchmarking, it was discovered that lightgbm training time could be reduced 4X-10X by setting the num_threads to be equal to the # of machine cores - 1 for the single dataset mode (which is now the default mode).
This actually was already suggested in the lightgbm docs:
https://lightgbm.readthedocs.io/en/latest/Parameters.html#num_threads
Poor network communication can lead to very slow training execution time.
On one scenario with a 37 GB dataset on disk with parameters:
learning_rate = 0.1, num_leaves = 768, num_trees = 1000, min_data_in_leaf = 15000, max_bin = 512the training time without setting num_threads was 5.5 hours, while setting num_threads to (number of machine cores)-1 reduced the training time to 1.2 hours. The change in performance will vary depending on the parameters used.
This PR sets the distributed lightgbm number of threads by default to be (num cores)-1 if the parameter is not specified by the user. The user can still override the parameter.