Initial support for multi-target tree. #8616

trivialfis · 2022-12-20T08:06:31Z

This is a rough PR for early reviews and discussions, it contains bugs and unfinished code.

Add a new multi-target tree structure that's embedded into the existing regtree.
Add a new builder to hist.
Add a new evaluator to hist.

I try to reuse as much existing code as possible. For instance, there's no structural change to the histogram builder and the implementation just iterates over a list of builders for each target. However, this might change in the future as we want a more integrated implementation. The evaluation code has to be rewritten for performance. Lastly, there are some optimization techniques for the multi-target tree when the number of targets is huge. Some of the known methods include summarizing the gradient, selecting the gradient, projecting the gradient, optimizing for sparse gradient, etc. I haven't implemented any of those yet, the PR is for the core multi-target tree structure.

There are other cases where we have vector leaf but not multi-target tree grower. For instance, we might want the leaf to be a linear model, or it might contain extra parameters for a probability distribution. These will require new tree training algorithms, but the tree structure is largely the same. The PR is a proof of concept.

The implementation is not as efficient as the single-target one, which doesn't represent the theoretical performance of the strategy.

For small testing datasets, using vector-leaf might lead to significant overfit.

what's working

Multi-target regression and single-target-multi-class classification training using hist and gbtree with most of the tree parameters except for the mono constraint. Numeric feature only.
Inference on CPU including QDM, inplace, DMatrix for normal outputs. (no leaf index, SHAP, etc)

what's not working

everything else.

@@ -310,14 +300,8 @@ void PredictBatchByBlockOfRowsKernel(DataView batch, gbm::GBTreeModel const &mod

    FVecFill(block_size, batch_offset, num_feature, &batch, fvec_offset, p_thread_temp);
    // process block of rows through all trees to keep cache locality
-    if (model.learner_model_param->IsVectorLeaf()) {


This is to not rely on the model parameter, which is not serialized into JSON model.

trivialfis · 2023-03-17T16:13:20Z

include/xgboost/linalg.h

@@ -530,17 +530,17 @@ class TensorView {
  /**
   * \brief Number of items in the tensor.
   */
-  LINALG_HD [[nodiscard]] std::size_t Size() const { return size_; }
+  [[nodiscard]] LINALG_HD std::size_t Size() const { return size_; }


clangd is not quite happy about the place of c++ attribute when running in CUDA mode.

trivialfis · 2023-03-17T16:14:17Z

src/common/quantile.h

@@ -352,19 +352,6 @@ struct WQSummary {
      prev_rmax = data[i].rmax;
    }
  }
-  // check consistency of the summary


Unused function.

trivialfis · 2023-03-17T16:15:03Z

src/gbm/gbtree.cc

 #include "xgboost/objective.h"
 #include "xgboost/predictor.h"
-#include "xgboost/string_view.h"
+#include "xgboost/string_view.h"  // for StringView


I kept the custom string view for now. Some changes in c++20 string_view might be useful, we can back-port it to xgboost when needed.

trivialfis · 2023-03-17T16:16:35Z

src/tree/updater_quantile_hist.cc

+    monitor_->Stop(__func__);
+  }
+
+  void LeafPartition(RegTree const &tree, linalg::MatrixView<GradientPair const> gpair,


This is not used yet. We need some work on L1 and quantile regression for estimating vector leaf.

trivialfis · 2023-03-17T16:17:18Z

tests/ci_build/lint_python.py

@@ -230,6 +236,11 @@ def main(args: argparse.Namespace) -> None:
    parser.add_argument("--format", type=int, choices=[0, 1], default=1)
    parser.add_argument("--type-check", type=int, choices=[0, 1], default=1)
    parser.add_argument("--pylint", type=int, choices=[0, 1], default=1)
+    parser.add_argument(
+        "--fix",


A new argument for convenience.

trivialfis · 2023-03-17T16:19:20Z

tests/python-gpu/test_gpu_updaters.py

@@ -32,6 +32,19 @@ def train_result(param, dmat: xgb.DMatrix, num_rounds: int) -> dict:
    return result


+class TestGPUUpdatersMulti:


I have extracted all the multi-target/class datasets into an independent hypothesis search strategy. Other than the test for CPU hist, no testing logic is changed.

trivialfis · 2023-03-17T16:20:46Z

python-package/xgboost/testing/__init__.py

@@ -352,137 +352,6 @@ def __repr__(self) -> str:
        return self.name


-@memory.cache


pylint complains about the file being too huge (>1000 loc). I moved some of the data fetchers into testing/data.py.

trivialfis · 2023-03-17T19:25:21Z

Not sure if this is useful, but you can do it just for fun:

def alternate(plot_result: bool) -> None:
    """Draw a circle with 2-dim coordinate as target variables."""
    from xgboost.callback import TrainingCallback

    class ResetStrategy(TrainingCallback):
        def before_iteration(self, model, epoch: int, evals_log) -> bool:
            strategy = "multi_output_tree" if epoch % 2 == 0 else "one_output_per_tree"
            model.set_param({"multi_strategy": strategy})
            return False

    X, y = gen_circle()
    # Train a regressor on it
    reg = xgb.XGBRegressor(
        tree_method="hist",
        n_estimators=4,
        n_jobs=1,
        max_depth=8,
        subsample=0.6,
        callbacks=[ResetStrategy()]
    )
    reg.fit(X, y, eval_set=[(X, y)])

doc/parameter.rst

- Add new hist tree builder. - Move data fetchers for tests. - Dispatch function calls in gbm base on the tree type.

s-banach · 2023-03-27T03:43:35Z

(Just seeing this for the first time, haven't put much brain power into it yet.)
Will it be easy to use this where the multiple targets are different parameters of a random variable?
E.g. shape and scale of a gamma variable.

StatMixedML · 2023-05-07T08:05:36Z

@s-banach

Essentially, multi target trees can be used if there are more than one parameter to predict. This can be useful if you want to model all parameters of a univariate and multivariate parametric distribution, see

Multi-Target XGBoostLSS Regression
Distributional Gradient Boosting Machines
XGBoostLSS - An extension of XGBoost to probabilistic forecasting

trivialfis mentioned this pull request Dec 20, 2022

Multiple output regression #2087

Closed

trivialfis force-pushed the multi-target-hist branch from 2dcc11a to ba0a5b4 Compare December 24, 2022 22:39

trivialfis force-pushed the multi-target-hist branch 2 times, most recently from c1cb30e to 0995252 Compare January 4, 2023 20:14

trivialfis mentioned this pull request Jan 18, 2023

Extract CPU sampling routines. #8697

Merged

trivialfis force-pushed the multi-target-hist branch from 0940fe7 to 6ad1e57 Compare January 19, 2023 18:16

trivialfis mentioned this pull request Jan 19, 2023

Small refactor for hist builder. #8698

Merged

trivialfis force-pushed the multi-target-hist branch from 6ad1e57 to d87affb Compare January 19, 2023 18:33

trivialfis force-pushed the multi-target-hist branch 2 times, most recently from 4a63670 to ec56a0f Compare February 9, 2023 09:30

trivialfis force-pushed the multi-target-hist branch 2 times, most recently from 0961133 to 99404c7 Compare March 6, 2023 13:06

trivialfis mentioned this pull request Mar 6, 2023

Support F order for the tensor type. #8872

Merged

trivialfis force-pushed the multi-target-hist branch from 3a2270e to d9ea74d Compare March 7, 2023 20:27

trivialfis mentioned this pull request Mar 8, 2023

Define core multi-target regression tree structure. #8884

Merged

trivialfis force-pushed the multi-target-hist branch from 7306f98 to b06792c Compare March 9, 2023 14:57

This was referenced Mar 9, 2023

Pass obj info by reference instead of by value. #8889

Merged

Define multi-strategy parameter. #8890

Merged

Define multi expand entry. #8895

Merged

trivialfis force-pushed the multi-target-hist branch from baacee0 to 543002c Compare March 10, 2023 19:13

trivialfis mentioned this pull request Mar 10, 2023

Predictor for vector leaf. #8898

Merged

trivialfis force-pushed the multi-target-hist branch from 0dff6d9 to b3a2141 Compare March 13, 2023 17:37

trivialfis mentioned this pull request Mar 13, 2023

Implement hist evaluator for multi-target tree. #8908

Merged

trivialfis force-pushed the multi-target-hist branch from b3a2141 to fd670a8 Compare March 14, 2023 20:27

trivialfis mentioned this pull request Mar 15, 2023

Partitioner for multi-target tree. #8922

Merged

trivialfis force-pushed the multi-target-hist branch from fd670a8 to 64b5187 Compare March 16, 2023 11:06

trivialfis mentioned this pull request Mar 16, 2023

Refactor hist to prepare for multi-target builder. #8928

Merged

trivialfis force-pushed the multi-target-hist branch from 7a2b940 to f4c2a02 Compare March 17, 2023 09:56

trivialfis changed the title ~~[WIP] Initial support for multi-target tree.~~ Initial support for multi-target tree. Mar 17, 2023

trivialfis marked this pull request as ready for review March 17, 2023 14:00

trivialfis commented Mar 17, 2023

View reviewed changes

RAMitchell approved these changes Mar 22, 2023

View reviewed changes

doc/parameter.rst Outdated Show resolved Hide resolved

trivialfis force-pushed the multi-target-hist branch from 82930dc to c4948a3 Compare March 22, 2023 10:32

trivialfis added 3 commits March 22, 2023 18:34

Implement multi-target for hist.

c597d40

- Add new hist tree builder. - Move data fetchers for tests. - Dispatch function calls in gbm base on the tree type.

naming is hard.

8dbd690

lint.

abc1f4b

trivialfis force-pushed the multi-target-hist branch from c4948a3 to abc1f4b Compare March 22, 2023 10:40

trivialfis added 3 commits March 22, 2023 19:24

Tidy.

dbd08df

Fix rebase.

458e865

fix mq path.

2056fa8

trivialfis merged commit 151882d into dmlc:master Mar 22, 2023

trivialfis deleted the multi-target-hist branch March 22, 2023 15:50

ShellLM mentioned this pull request Aug 11, 2024

Xgboost 2.0.0 · dmlc/xgboost irthomasthomas/undecidability#878

Open

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Initial support for multi-target tree. #8616

Initial support for multi-target tree. #8616

trivialfis commented Dec 20, 2022 •

edited

Loading

trivialfis Mar 17, 2023

trivialfis Mar 17, 2023

trivialfis Mar 17, 2023

trivialfis Mar 17, 2023

trivialfis Mar 17, 2023

trivialfis Mar 17, 2023

trivialfis Mar 17, 2023 •

edited

Loading

trivialfis Mar 17, 2023

trivialfis commented Mar 17, 2023 •

edited

Loading

s-banach commented Mar 27, 2023

StatMixedML commented May 7, 2023

		@@ -32,6 +32,19 @@ def train_result(param, dmat: xgb.DMatrix, num_rounds: int) -> dict:
		return result


		class TestGPUUpdatersMulti:

		@@ -352,137 +352,6 @@ def __repr__(self) -> str:
		return self.name


		@memory.cache

Initial support for multi-target tree. #8616

Initial support for multi-target tree. #8616

Conversation

trivialfis commented Dec 20, 2022 • edited Loading

what's working

what's not working

Related

trivialfis Mar 17, 2023

Choose a reason for hiding this comment

trivialfis Mar 17, 2023

Choose a reason for hiding this comment

trivialfis Mar 17, 2023

Choose a reason for hiding this comment

trivialfis Mar 17, 2023

Choose a reason for hiding this comment

trivialfis Mar 17, 2023

Choose a reason for hiding this comment

trivialfis Mar 17, 2023

Choose a reason for hiding this comment

trivialfis Mar 17, 2023 • edited Loading

Choose a reason for hiding this comment

trivialfis Mar 17, 2023

Choose a reason for hiding this comment

trivialfis commented Mar 17, 2023 • edited Loading

s-banach commented Mar 27, 2023

StatMixedML commented May 7, 2023

trivialfis commented Dec 20, 2022 •

edited

Loading

trivialfis Mar 17, 2023 •

edited

Loading

trivialfis commented Mar 17, 2023 •

edited

Loading