Multiverso document

Multiverso

Multiverso provides a parameter server architecture to scale machine learning tasks to distributed clusters. For each Multiverso job, computing nodes are logically grouped into two categories -- server nodes and worker nodes. A group of server nodes are serving for the model storage and access. The global shared parameters are distributed at server nodes and each server node maintains one model partition. The workload and training data are distributed at a group of worker nodes. Each worker node, where client programs run on, contains a partition of training data and trains on its data partition based on the local model replica. The local model replica is Get from the parameter server and the producing updates is Add to the parameter server.

Example

An Multiverse example is shown as follow.

#include <multiverso/multiverso.h>
#include <multiverso/table/array_table.h>
using namespace multiverso;

int main(int argc, char* argv) {
  MV_SetFlag("sync", true);
  MV_Init();
  
  ArrayTableOption<int> option;
  option.size = 100;
  auto table = MV_CreateTable(option);

  std::vector<int> model(100, 0);
  std::vector<int> delta(100, 1); 

  for (int iter = 0; iter < 100; ++iter) {
    table->Add(delta.data(), delta.size());
    table->Get(model.data(), model.size());
    // CHECK_EQ(model[i], (iter+1) * MV_NumWorkers());
  }

  MV_Shutdown();
}

Programming with Multiverso follows the following steps.

1. Set the config and Init the Multiverso environment.

      MV_SetFlag("sync", true); // optional
      MV_Init(argc, argv);
   

2. Create table, which is the abstraction

      auto table = MV_CreateTable(option);
   

3. Write your main logic, for example your machine learning algorithm, by access the global parameter with Table API

      // Your learning algorithm to produce delta
      // ...
      // Communicating with PS 
      table->Add(delta.data(), delta.size());
      table->Get(model.data(), model.size());
   

4. Shutdown

      MV_Shutdown();
   

For the detailed information about the API, please refer to the API document.