Portable, scalable and reliable distributed machine learning.

Wormhole is a place where DMLC projects works together to provide scalable and reliable machine learning toolkits that can run on various platforms

Features

Portable:
- Supported platforms: YARN, MPI and Sungrid Engine
- Planned: docker support
Rich support of Data Source
- All projects can read data from HDFS, S3 or local filesystem
Scalable and Reliable

List of Tools

Boosted Trees (GBDT): XGBoost: eXtreme Gradient Boosting
L-BFGS based linear solver
Asynchrouns SGD linear solver

Build

copy make/config.mk to root folder
modify according to your settings
type make or make name-of-tool-you-want

How to Submit Jobs

make sure dmlc-core exist in root folder
- type make dmlc-core to get it
Use the submission script in dmlc-core/tracker to submit job to the platform of your choice

Contributing

We believe that we can create machine learning tools that are portable and works with each other.
Contributing of machine learning projects, tutorials and to core dmlc projects are welcomed.
- All machine learning projects can depends on dmlc-core, rabit or parameter-server

Project Structure

learn contains simple but powerful learning tools in wormhole
repo is used to clone other DMLC repos that wormhole can depend on
Depending DMLC Libraries
- dmlc-core gives the core modules of most DMLC projects.
- rabit provides reliable BSP Allreduce communication.
- parameter-server provides asynchronize parameter server abstraction.