WIP: FEA Quantile regression for decision trees

[cakedev0]

WIP

Follow up to PR scikit-learn#32100 and esp. discussion here: scikit-learn#32100 (comment)

TODO (bottom-up order):

• do the core changes in the algorithm
• validate correctness with test_absolute_errors_precomputation_function (now renamed test_pinball_loss_precomputation_function)
• change the MAE criterion class to a QuantileRegression class
• modify the public API of the class and pass down the "q" parameter to the QuantileRegression class:
  • validate parameter naming
• write some high-level tests (even though the current modified test_absolute_errors_precomputation_function makes me already fairly confident):
  • Would be convenient to have the test from this PR merged: MNT: Decision trees: add test for split optimality scikit-learn/scikit-learn#32193
  • Pinball criterion trees should have a smaller pinball loss (using the same alpha for both) than other trees.
  • Test that training to trees with q=0.1 and q=0.9 allow to predict a good confidence interval that fits ~80% of the data.
• update de public API doc: I need
• update the user-guide: I should
• doc: add examples? Maybe one with a

Reference Issues/PRs

scikit-learn#32100

What does this implement/fix? Explain your changes.

Any other comments?

Maths:

We consider a weighted dataset ${(y_i, w_i)}_{i}$ with non-negative weights $w_i$.

For a scalar prediction $q$, the weighted pinball loss is

$$ L_\alpha(q) = \sum_{i} w_i \big( \alpha \max(y_i - q, 0) + (1 - \alpha)\max(q - y_i, 0) \big) $$

Equivalently, splitting by whether $y_i \ge q$ or $y_i < q$:

$$ L_\alpha(q) = \alpha \sum_{i: y_i \ge q} w_i (y_i - q) + (1 - \alpha) \sum_{i: y_i < q} w_i (q - y_i) $$

To evaluate this efficiently, introduce the aggregates

$$ W^+(q) = \sum_{i: y_i \ge q} w_i, \qquad Y^+(q) = \sum_{i: y_i \ge q} w_i y_i, $$

$$ W^-(q) = \sum_{i: y_i < q} w_i, \qquad Y^-(q) = \sum_{i: y_i < q} w_i y_i. $$

Using these, the loss admits the "O(1)" form

$$ L_\alpha(q) = \alpha \big( Y^+(q) - q W^+(q) \big) + (1 - \alpha) \big( q W^-(q) - Y^-(q) \big). $$

Or in the code:

q * (above.weighted_sum - quantile * above.total_weight)
+ (1 - q) * (quantile * below.total_weight - below.weighted_sum)

[github-actions]

✔️ Linting Passed

All linting checks passed. Your pull request is in excellent shape! ☀️

_{Generated for commit: 0cdeaaf. Link to the linter CI: here}

[cakedev0]

@adam2392 and @ogrisel: I moved forward here and before polishing it for review, I'd like your input:

(polishing it for review will take some time)

• do we validate we want to include quantile decision tree regressors in sklearn? Or should I open an issue to discuss this inclusion?
• naming:
  • name of the new criterion? For now: "pinball" because "poisson" stands for "pinball loss"
  • public name of the parameter that controls the quantile/pinball loss $\alpha$? For now, pinball_alpha but doesn't make a clear link with quantile... (though we can make the link in the docstring & user guide)

Rest of naming is more internal, and can be discussed during the review process.

Thanks 🙏

[adam2392]

I want to make sure we separate the notion of quantile loss criterion in training and quantile during inference since those can be independent. To your points:

• I am in favor but perhaps there should be a broader discussion? I'll let @ogrisel decide. I believe Quantiles or robustness is seen as a key focus area of scikit learn so considering we can get this feature without significant runtime penalty, then I don't see reasons to reject.
• perhaps quantile_loss? And the optional quantile therefore must be specified with default being 0.5. We can even consider having a discussion of deprecating mean_absolute_error since I doubt ppl used it given its runtime complexity.