Custom multiclass objectives in R

[Someone894]

Added workaround described in #2776

with this change the https://github.com/pmontman/fforma package works fine.

[codecov-io]

Codecov Report

Merging #4996 into master will increase coverage by 2.10%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4996      +/-   ##
==========================================
+ Coverage   81.66%   83.76%   +2.10%     
==========================================
  Files          11       11              
  Lines        2389     2409      +20     
==========================================
+ Hits         1951     2018      +67     
+ Misses        438      391      -47     

Impacted Files	Coverage Δ
python-package/xgboost/libpath.py	55.55% <0.00%> (-4.45%)	⬇️
python-package/xgboost/sklearn.py	90.88% <0.00%> (+0.96%)	⬆️
python-package/xgboost/dask.py	90.30% <0.00%> (+2.67%)	⬆️
python-package/xgboost/tracker.py	93.97% <0.00%> (+15.66%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fa26313...8eedbe1. Read the comment docs.

[trivialfis]

Could you please help taking a look @pommedeterresautee ?

[pommedeterresautee]

Tks, I check today or tomorrow.

[pommedeterresautee]

@Someone894 Hi, I have made few checks and it seems to not break simple cases (I was worried about the reshaping).
However, I am wondering how outputmargin set to TRUE will never impact in some way the training? (in some other cases that your own custom objective).
I am asking because the function xgb.iter.update is obviously central and I am under the impression you have already performed a deep work for fforma, so may be you can help me in better understand that point.

[Someone894]

The FFORMA-System needs the custom objective in order to work. So @pmontman created a custom-version (https://github.com/pmontman/customxgboost), which is not maintained anymore. This got me worried about the old code causing some trouble in the future. So I talked to him in order to understand what his changes to the basic XGBoost-package where. Then I implemented these changes and tested the FFORMA-System with them and it worked fine. but to be honest I did not understand the changes in the XGBoost-package since it feels really complicated to me.

[pommedeterresautee]

Thank you a lot for these information.
Basically the function xgb.iter.update is quite important.
It is called at each iteration of the boosting process.

xgboost/R-package/R/xgb.train.R

Line 331 in 5e97de6

xgb.iter.update(bst$handle, dtrain, iteration - 1, obj)

Its purpose is to apply the objective to the predictions of the model being built and compute the loss.

xgboost/R-package/R/utils.R

Line 149 in 9536a7e

gpair <- obj(pred, dtrain)

This PR modifies the predictions by setting outputmargin to True, basically the predictions are now the margin directly computed by XGBoost (the original untransformed predictions).
My feeling is that in several cases this will have unwanted effect on the objective, and the quality of the model.

Would it be possible for you to contact @pmontman to see if there are reasons why such change would be Ok?

Otherwise, I am thinking that another approach may be more appropriate.
May be another hook function or some information provided through parameters of xgb.iter.update making possible for third party objectives to retrieve the margin.

Regarding the reshape thing, it should be easy to perform in the objective itself.

[jeffzi]

I encountered the same issue several months ago while looking at FFORMA (see #4288).

I had to make the same modification in xgb.iter.eval to fix xgb.cv as well.

xgboost/R-package/R/utils.R

Line 175 in e089e16

preds <- predict(booster_handle, w) # predict using all trees

[pommedeterresautee]

Thanks @jeffzi . I understand the purpose of the change, my question is how this change may not have any impact on other use cases? because here we are wondering if it makes sense to apply the change to the main library.

I may miss something.

And may be there is something else to do to avoid to impact all use cases?

[Someone894]

I tried to reach @pmontman to also have a look at this topic, but nothing happened, as you can see.

[trivialfis]

@Someone894 @pommedeterresautee It makes sense to me to use raw prediction here, as objective function should handle this transformation itself. WDYT?

[jeffzi]

It makes sense to me to use raw prediction, since it reflects what happens when no custom objective is given.

xgboost/R-package/R/utils.R

Lines 145 to 151 in ad4a1c7

	if (is.null(obj)) {
	.Call(XGBoosterUpdateOneIter_R, booster_handle, as.integer(iter), dtrain)
	} else {
	pred <- predict(booster_handle, dtrain)
	gpair <- obj(pred, dtrain)
	.Call(XGBoosterBoostOneIter_R, booster_handle, dtrain, gpair$grad, gpair$hess)
	}

UpdateOneIter uses raw prediction:

xgboost/src/learner.cc

Lines 561 to 567 in ad4a1c7

	monitor_.Start("PredictRaw");
	this->PredictRaw(train, &preds_[train]);
	monitor_.Stop("PredictRaw");
	monitor_.Start("GetGradient");
	obj_->GetGradient(preds_[train], train->Info(), iter, &gpair_);
	monitor_.Stop("GetGradient");
	gbm_->DoBoost(train, &gpair_, obj_.get());

[seangupta]

Hi, I've encountered this problem as well. Will there be an update soon? Thanks.

[trivialfis]

I incline to merge this PR(or new one with tests). @hcho3 WDYT?