582 Add ability to model load values used by LB strategies

docs/md/lb-manager.md

@@ -19,13 +19,13 @@ To run a load balancer at runtime:
 
 The LB specification file allows users to specify which load balancer along with
 which LB-specific configuration parameters are passed to the load balancer
-instance for any given phase. The order of the LB phase specficiation lines in
+instance for any given phase. The order of the LB phase specification lines in
 the file disambiguates lines---higher precedence for earlier lines.
 
 The format of the LB specification file is:
 
 \code
-[%] <$phase> <$lbname> [$LB-specific-arg-1] ... [$LB-specfic-arg-N]
+[%] <$phase> <$lbname> [$LB-specific-arg-1] ... [$LB-specific-arg-N]
 \endcode
 
 If a `%` is present, the line matches phases where:
@@ -57,3 +57,68 @@ The following is an example LB specification:
 | HierarchicalLB | Hierarchical            | Build tree to move objects nodes               | `vt::vrt::collection::lb::HierarchicalLB` |
 | ZotltanLB      | Hyper-graph Partitioner | Run Zoltan in hyper-graph mode to LB           | `vt::vrt::collection::lb::ZoltanLB` |
 | StatsMapLB     | User-specified          | Read file to determine mapping                 | `vt::vrt::collection::lb::StatsMapLB` |
+
+\section load-models Object Load Models
+
+The performance-oriented load balancers described in the preceding
+section require a prediction of the loads each object will represent
+during the phases between one load balancing invocation and the
+next. These predictions are provided by load models, which are
+implementations of the `vt::vrt:collection::balance::LoadModel`
+interface. There are a number of general-purpose load model
+implementations provided by \vt.
+
+By default, \vt uses a load model that predicts each object's work
+load for all future phases will match its workload in the most recent
+past phase. The system also provides an interface for applications and
+users to arrange use of a non-default load model where that may be
+desirable for reasons such as performance experimentation,
+specialization to application details, or execution environment
+considerations. To install a custom load model, application code
+should call `vt::theLBManager()->setLoadModel(user_model)`. To
+simplify implementation of custom load models, and allow them to
+benefit from future system-level improvements, we recommend that
+custom load models be composed atop the default model, which can be
+obtained by calling `vt::theLBManager()->getBaseLoadModel()`.
+
+Most provided load models are designed as composable filters inherited
+from the `vt::vrt:collection::balance::ComposedModel` class. This
+allows them to form a 'model stack' in which each class makes a
+particular adjustment to the predictions generated, and relies on
+others above and below to refine them further. One exception is the
+`vt::vrt:collection::balance::RawData` model, which directly returns
+past values recorded in the instrumented statistics structures that
+`LBManager` provides.
+
+To illustrate the design concept, the default model is implemented as
+a stack of two other components. At the base of the stack is a
+`RawData` model that will refer to the instrumented object load
+timings recorded by the system during preceding execution. Layered on
+that is a `vt::vrt:collection::balance::NaivePersistence` model that
+queries the underlying `RawData` model for the times taken in the most
+recent phase, and returns those same times as its prediction of the
+times those objects will take in all future phases.
+
+The full set of load model classes provided with \vt is as follows
+
+| Load Model         | Description                                         | Reference |
+| -------------------|-----------------------------------------------------|---------- |
+| **Utilities**      |
+| LoadModel          | Pure virtual interface class, which the following implement | `vt::vrt:collection::balance::LoadModel` |
+| ComposedModel      | A convenience class for most implementations to inherit from, that passes unmodified calls through to an underlying model instance | `vt::vrt:collection::balance::ComposedModel` |
+| RawData            | Returns historical data only, from the measured times | `vt::vrt:collection::balance::RawData` |
+| **Transformers**   | Transforms the values computed by the composed model(s), agnostic to whether a query refers to a past or future phase |
+| Norm               | When asked for a `WHOLE_PHASE` value, computes a specified l-norm over all subphases | `vt::vrt:collection::balance::Norm` |
+| SelectSubphases    | Filters and remaps the subphases with data present in the underlying model | `vt::vrt:collection::balance::SelectSubphases` |
+| CommOverhead       | Adds a specified amount of imputed 'system overhead' time to each object's work based on the number of messages received | `vt::vrt:collection::balance::CommOverhead` |
+| PerCollection      | Maintains a set of load models associated with different collection instances, and passes queries for an object through to the model corresponding to its collection | `vt::vrt:collection::balance::PerCollection` |
+| **Predictors**     | Computes values for future phase queries, and passes through past phase queries |
+| NaivePersistence   | Passes through historical queries, and maps all future queries to the most recent past phase | `vt::vrt:collection::balance::NaivePersistence` |
+| PersistenceMedianLastN | Similar to NaivePersistence, except that it predicts based on a median over the N most recent phases | `vt::vrt:collection::balance::PersistenceMedianLastN` |
+| LinearModel        | Computes a linear regression over on object's loads from a number of recent phases | `vt::vrt:collection::balance::LinearModel` |
+| MultiplePhases     | Computes values for future phases based on sums of the underlying model's predictions for N corresponding future phases |  `vt::vrt:collection::balance::MultiplePhases` |
+
+All of the provided load balancers described in the previous section
+require that the installed load model provide responses to future
+phase queries for at least `PhaseOffset::NEXT_PHASE` (i.e. `0`), as
+the **Predictors** described above do.

src/CMakeLists.txt

@@ -83,6 +83,7 @@ set(
         vrt/collection/balance/zoltanlb
         vrt/collection/balance/randomlb
         vrt/collection/balance/lb_invoke
+        vrt/collection/balance/model
         vrt/collection/balance/proxy
     lb/instrumentation
       lb/instrumentation/centralized

src/vt/utils/stats/linear_regression.h

@@ -0,0 +1,135 @@
+/*
+//@HEADER
+// *****************************************************************************
+//
+//                             linear_regression.h
+//                           DARMA Toolkit v. 1.0.0
+//                       DARMA/vt => Virtual Transport
+//
+// Copyright 2019 National Technology & Engineering Solutions of Sandia, LLC
+// (NTESS). Under the terms of Contract DE-NA0003525 with NTESS, the U.S.
+// Government retains certain rights in this software.
+//
+// Redistribution and use in source and binary forms, with or without
+// modification, are permitted provided that the following conditions are met:
+//
+// * Redistributions of source code must retain the above copyright notice,
+//   this list of conditions and the following disclaimer.
+//
+// * Redistributions in binary form must reproduce the above copyright notice,
+//   this list of conditions and the following disclaimer in the documentation
+//   and/or other materials provided with the distribution.
+//
+// * Neither the name of the copyright holder nor the names of its
+//   contributors may be used to endorse or promote products derived from this
+//   software without specific prior written permission.
+//
+// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+// AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+// IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
+// ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE
+// LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR
+// CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF
+// SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS
+// INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
+// CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+// POSSIBILITY OF SUCH DAMAGE.
+//
+// Questions? Contact darma@sandia.gov
+//
+// *****************************************************************************
+//@HEADER
+*/
+
+#if !defined INCLUDED_VT_UTILS_STATS_LINEAR_REGRESSION_H
+#define INCLUDED_VT_UTILS_STATS_LINEAR_REGRESSION_H
+
+#include "vt/config.h"
+
+#include <vector>
+#include <numeric>
+
+namespace vt { namespace util { namespace stats {
+
+/**
+ * \struct LinearRegression
+ *
+ * \brief Perform a simple linear regression to predict values with a linear
+ * model
+ */
+struct LinearRegression {
+
+  /**
+   * \brief Construct a linear regression
+   *
+   * \param[in] in_x the x-values
+   * \param[in] in_y the y-values
+   */
+  LinearRegression(std::vector<double> const& in_x, std::vector<double> const& in_y)
+    : x_(in_x),
+      y_(in_y)
+  { }
+
+  /**
+   * \brief Perform the regression
+   */
+  void compute() {
+    vtAssert(x_.size() == y_.size(), "Sizes must be the same");
+    vtAssert(x_.size() != 0,         "Sizes must not be zero");
+    auto const n = x_.size();
+    auto const sum_x = std::accumulate(x_.begin(), x_.end(), 0.0);
+    auto const sum_y = std::accumulate(y_.begin(), y_.end(), 0.0);
+    auto const p_xx = std::inner_product(x_.begin(), x_.end(), x_.begin(), 0.0);
+    auto const p_xy = std::inner_product(x_.begin(), x_.end(), y_.begin(), 0.0);
+    // numerator:   sum over i of (x[i] - X_mean) * (y[i] - Y_mean)
+    // denominator: sum over i of (x[i] - X_mean) * (x[i] - X_mean)
+    auto const numerator   = p_xy * n - sum_x * sum_y;
+    auto const denominator = p_xx * n - sum_x * sum_x;
+
+    vtAssert(denominator != 0, "Denominator must not be zero");
+
+    slope_ = numerator / denominator;
+    intercept_ = (sum_y - slope_ * sum_x) / n;
+    computed_ = true;
+  }
+
+  /**
+   * \brief Get the slope
+   *
+   * \return the slope of the line
+   */
+  double getSlope() const { return slope_; }
+
+  /**
+   * \brief Get the y-intercept
+   *
+   * \return the y-intercept
+   */
+  double getIntercept() const { return intercept_; }
+
+  /**
+   * \brief Predict a value using the linear model
+   *
+   * \param[in] in_x the x value to predict
+   *
+   * \return the y value
+   */
+  double predict(double in_x) {
+    if (not computed_) {
+      compute();
+    }
+    return intercept_ + slope_ * in_x;
+  }
+
+private:
+  std::vector<double> const& x_;
+  std::vector<double> const& y_;
+  double slope_ = 0.;
+  double intercept_ = 0.;
+  bool computed_ = false;
+};
+
+}}} /* end namespace vt::util::stats */
+
+#endif /*INCLUDED_VT_UTILS_STATS_LINEAR_REGRESSION_H*/

src/vt/vrt/collection/balance/baselb/baselb.cc

@@ -51,6 +51,7 @@
 #include "vt/vrt/collection/balance/lb_invoke/start_lb_msg.h"
 #include "vt/vrt/collection/balance/read_lb.h"
 #include "vt/vrt/collection/balance/lb_invoke/lb_manager.h"
+#include "vt/vrt/collection/balance/proc_stats.h"
 #include "vt/timing/timing.h"
 #include "vt/collective/reduce/reduce.h"
 #include "vt/collective/collective_alg.h"
@@ -63,14 +64,15 @@ namespace vt { namespace vrt { namespace collection { namespace lb {
 void BaseLB::startLB(
   PhaseType phase,
   objgroup::proxy::Proxy<BaseLB> proxy,
-  balance::ProcStats::LoadMapType const& in_load_stats,
+  balance::LoadModel* model,
   ElementCommType const& in_comm_stats
 ) {
   start_time_ = timing::Timing::getCurrentTime();
   phase_ = phase;
   proxy_ = proxy;
+  load_model_ = model;
 
-  importProcessorData(in_load_stats, in_comm_stats);
+  importProcessorData(in_comm_stats);
 
   term::TerminationDetector::Scoped::collective(
     [this] { computeStatistics(); },
@@ -92,17 +94,16 @@ BaseLB::ObjBinType BaseLB::histogramSample(LoadType const& load) const {
 }
 
 void BaseLB::importProcessorData(
-  ElementLoadType const& load_in, ElementCommType const& comm_in
+  ElementCommType const& comm_in
 ) {
   auto const& this_node = theContext()->getNode();
   vt_debug_print(
     lb, node,
     "{}: importProcessorData: load stats size={}, load comm size={}\n",
-    this_node, load_in.size(), comm_in.size()
+    this_node, load_model_->getNumObjects(), comm_in.size()
   );
-  for (auto&& stat : load_in) {
-    auto const& obj = stat.first;
-    auto const& load = stat.second;
+  for (auto obj : *load_model_) {
+    auto load = load_model_->getWork(obj, {balance::PhaseOffset::NEXT_PHASE, balance::PhaseOffset::WHOLE_PHASE});
     auto const& load_milli = loadMilli(load);
     auto const& bin = histogramSample(load_milli);
     this_load += load_milli;
@@ -116,7 +117,6 @@ void BaseLB::importProcessorData(
     );
   }
 
-  load_data = &load_in;
   comm_data = &comm_in;
 }
 
@@ -331,8 +331,8 @@ void BaseLB::computeStatisticsOver(Statistic stat) {
       if (elm.first.onNode() or elm.first.selfEdge()) {
         continue;
       }
-      //vt_print(lb, "comm_load={}, elm={}\n", comm_load, elm.second);
-      comm_load += elm.second;
+      //vt_print(lb, "comm_load={}, elm={}\n", comm_load, elm.second.bytes);
+      comm_load += elm.second.bytes;
     }
     auto msg = makeMessage<StatsMsgType>(Statistic::P_c, comm_load);
     proxy_.template reduce<ReduceOp>(msg,cb);
@@ -344,7 +344,7 @@ void BaseLB::computeStatisticsOver(Statistic stat) {
     for (auto&& elm : *comm_data) {
       // Only count object-to-object direct edges in the O_c statistics
       if (elm.first.cat_ == balance::CommCategory::SendRecv and not elm.first.selfEdge()) {
-        lds.emplace_back(balance::LoadData(elm.second));
+        lds.emplace_back(balance::LoadData(elm.second.bytes));
       }
     }
     auto msg = makeMessage<StatsMsgType>(Statistic::O_c, reduceVec(std::move(lds)));
@@ -354,8 +354,8 @@ void BaseLB::computeStatisticsOver(Statistic stat) {
   case Statistic::O_l: {
     // Perform the reduction for O_l -> object load only
     std::vector<balance::LoadData> lds;
-    for (auto&& elm : *load_data) {
-      lds.emplace_back(balance::LoadData(elm.second));
+    for (auto elm : *load_model_) {
+      lds.emplace_back(load_model_->getWork(elm, {balance::PhaseOffset::NEXT_PHASE, balance::PhaseOffset::WHOLE_PHASE}));
     }
     auto msg = makeMessage<StatsMsgType>(Statistic::O_l, reduceVec(std::move(lds)));
     proxy_.template reduce<ReduceOp>(msg,cb);

src/vt/vrt/collection/balance/baselb/baselb.h

@@ -49,10 +49,11 @@
 #include "vt/vrt/collection/balance/lb_common.h"
 #include "vt/vrt/collection/balance/lb_invoke/start_lb_msg.h"
 #include "vt/vrt/collection/balance/baselb/baselb_msgs.h"
-#include "vt/vrt/collection/balance/proc_stats.h"
+#include "vt/vrt/collection/balance/stats_msg.h"
 #include "vt/vrt/collection/balance/lb_comm.h"
 #include "vt/vrt/collection/balance/read_lb.h"
 #include "vt/objgroup/headers.h"
+#include "vt/vrt/collection/balance/model/load_model.h"
 
 #include <set>
 #include <map>
@@ -106,9 +107,9 @@ struct BaseLB {
    * through calls to `migrateObjectTo`. Callers can then access that
    * set using `getTransfers` and apply it using `applyMigrations`.
    */
-  void startLB(PhaseType phase, objgroup::proxy::Proxy<BaseLB> proxy, balance::ProcStats::LoadMapType const& in_load_stats, ElementCommType const& in_comm_stats);
+  void startLB(PhaseType phase, objgroup::proxy::Proxy<BaseLB> proxy, balance::LoadModel *model, ElementCommType const& in_comm_stats);
   void computeStatistics();
-  void importProcessorData(ElementLoadType const& ld, ElementCommType const& cm);
+  void importProcessorData(ElementCommType const& cm);
   void statsHandler(StatsMsgType* msg);
   void finishedStats();
 
@@ -140,14 +141,15 @@ struct BaseLB {
   int32_t bin_size_                               = 10;
   ObjSampleType obj_sample                        = {};
   LoadType this_load                              = 0.0f;
-  ElementLoadType const* load_data                = nullptr;
   ElementCommType const* comm_data                = nullptr;
   StatisticMapType stats                          = {};
   objgroup::proxy::Proxy<BaseLB> proxy_           = {};
   PhaseType phase_                                = 0;
   bool comm_aware_                                = false;
   bool comm_collectives_                          = false;
   std::unique_ptr<balance::SpecEntry> spec_entry_ = nullptr;
+  // Observer only - LBManager owns the instance
+  balance::LoadModel* load_model_                 = nullptr;
 
 private:
   TransferVecType transfers_                      = {};

src/vt/vrt/collection/balance/elm_stats.cc

@@ -85,7 +85,7 @@ void ElementStats::recvObjData(
 ) {
   comm_.resize(cur_phase_ + 1);
   LBCommKey key(LBCommKey::CollectionTag{}, pfrom, tfrom, pto, tto, bcast);
-  comm_.at(cur_phase_)[key] += bytes;
+  comm_.at(cur_phase_)[key].receiveMsg(bytes);
 }
 
 void ElementStats::recvFromNode(
@@ -94,7 +94,7 @@ void ElementStats::recvFromNode(
 ) {
   comm_.resize(cur_phase_ + 1);
   LBCommKey key(LBCommKey::NodeToCollectionTag{}, from, pto, tto, bcast);
-  comm_.at(cur_phase_)[key] += bytes;
+  comm_.at(cur_phase_)[key].receiveMsg(bytes);
 }
 
 void ElementStats::recvToNode(
@@ -103,7 +103,7 @@ void ElementStats::recvToNode(
 ) {
   comm_.resize(cur_phase_ + 1);
   LBCommKey key(LBCommKey::CollectionToNodeTag{}, pfrom, tfrom, to, bcast);
-  comm_.at(cur_phase_)[key] += bytes;
+  comm_.at(cur_phase_)[key].receiveMsg(bytes);
 }
 
 void ElementStats::setModelWeight(TimeType const& time) {

src/vt/vrt/collection/balance/elm_stats.impl.h

@@ -49,7 +49,6 @@
 #include "vt/vrt/collection/balance/elm_stats.h"
 #include "vt/vrt/collection/balance/phase_msg.h"
 #include "vt/vrt/collection/balance/stats_msg.h"
-#include "vt/vrt/collection/balance/proc_stats.h"
 #include "vt/vrt/collection/balance/lb_type.h"
 #include "vt/vrt/collection/manager.h"
 #include "vt/vrt/collection/balance/lb_invoke/lb_manager.h"