what is the distribution communication framework in lightGBM?

[ICDI0906]

Is there a distribution distribution communication framework in LightGBM like rabit in xgboost?

[jameslamb]

Unlike rabit, LightGBM's implementation of collective operations like Allreduce, Allgather, etc. was never released as a standalone library. It is an integrated part of LightGBM itself.

Linking in case you aren't aware...last year, XGBoost decided to use a similar approach, pulling rabit back into XGBoost and shipping it with that library directly: dmlc/xgboost#5995.

More Details

The core communication code can be found here:

• https://github.com/microsoft/LightGBM/tree/master/src/network
• https://github.com/microsoft/LightGBM/blob/master/include/LightGBM/network.h

Notice there that LightGBM by default uses TCP sockets directly for communication, but it can also be compiled with MPI features enabled and run as an MPI application.

Those core communication routines are used to implement Data Parallel, Feature Parallel, and Voting Parallel training.

• https://github.com/microsoft/LightGBM/blob/master/src/treelearner/data_parallel_tree_learner.cpp
• https://github.com/microsoft/LightGBM/blob/master/src/treelearner/feature_parallel_tree_learner.cpp
• https://github.com/microsoft/LightGBM/blob/master/src/treelearner/voting_parallel_tree_learner.cpp

Those features are then exposed in several interfaces (including running the LightGBM CLI, Dask, or Spark). See https://lightgbm.readthedocs.io/en/latest/Parallel-Learning-Guide.html#integrations for a more complete list.

[no-response]

This issue has been automatically closed because it has been awaiting a response for too long. When you have time to to work with the maintainers to resolve this issue, please post a new comment and it will be re-opened. If the issue has been locked for editing by the time you return to it, please open a new issue and reference this one. Thank you for taking the time to improve LightGBM!

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.