Custom objective function doesn't produce expected results

[JivanRoquet]

Thank you for making LightGBM available.

Following this example made for XGBoost, I've tried implementing a custom objective function which is, for the time being, aimed at reproducing the built-in multiclass objective function. The goal for now is not to be efficient, just to mimic the existing multiclass objective function's return values.

def custom_training_loss(true_, pred_):
    # `model` is the model currently being used
    n_classes = len(model.classes_)
    n_samples = len(true_)
    
    # construct a matrix of true labels
    labels_ = np.zeros((n_samples, n_classes))
    idx = true_.astype(int)
    labels_[range(len(labels_)), idx] = 1.0
    
    # reshape pred_ to get a matrix of predictions
    preds_ = pred_.reshape(n_classes, n_samples).T
    
    # compute grad/hess by arithmetic operations between the two matrices
    grad = (preds_ - labels_).T.flatten()
    hess = (2.0 * preds_ * (1.0 - preds_)).T.flatten()

    return (grad, hess)

If I initialise the model with objective=custom_training_loss, the eval multiclass loss makes no progress during training.

[1]	eval's multi_logloss: 34.5388
[2]	eval's multi_logloss: 34.5388
[3]	eval's multi_logloss: 34.5388
[4]	eval's multi_logloss: 34.5388
[5]	eval's multi_logloss: 34.5388
[6]	eval's multi_logloss: 34.5388
[7]	eval's multi_logloss: 34.5388
[8]	eval's multi_logloss: 34.5388

But if I use objective='multiclass' the eval multiclass loss behaves as expected:

[1]	eval's multi_logloss: 6.54263
[2]	eval's multi_logloss: 6.25912
[3]	eval's multi_logloss: 5.99214
[4]	eval's multi_logloss: 5.74683
[5]	eval's multi_logloss: 5.52493
[6]	eval's multi_logloss: 5.32477
[7]	eval's multi_logloss: 5.14333
[8]	eval's multi_logloss: 4.98209

I'm not sure if there's a bug in my NumPy code, or if there is something I didn't get right from what the function is supposed to return, or how it should behave.

[jameslamb]

Thanks very much for your interest in LightGBM. Are you able to provide a minimal, reproducible example that maintainers could run, to see if we observe the same behavior?

That would include

• version of lightgbm you're using
• how you installed lightgbm
• fully self-contained code that I could copy, paste, and run which produces those log messages you've shown above

[JivanRoquet]

Hello, thanks for your quick answer.

I'm using LightGBM 3.2.1, installed using pip install lightgbm (Python3.8).

The actual code I have is not trivially separable but I'll try to isolate a self-contained code that can reproduce the issue. In the meantime, please feel free to indicate if by any chance you'd spot any obvious mistake in the custom function, or in the general logic of it.

In particular, I've tried to read the multiclass objective function from the C++ native code but I haven't been able to understand how grad and hess were derived from labels and preds.

[jameslamb]

I'll try to isolate a self-contained code that can reproduce the issue.

Thank you very much for the additional information. Without a minimal, reproducible example you're unlikely to get a high-quality answer very quickly, since the lack of such an example greatly increases the effort required for maintainers to answer your question.

For example, the behavior of custom objective functions in the Python package is slightly different depending on whether you are using the functional interface (train()) or the scikit-learn interface (LGBMClassifer().fit()): #4544 (review).

spot any obvious mistake in the custom function

I do see at least one thing that confuses me, and which would be clarified with a reproducible example.

Your code is referencing objects called model and y_true which are not in scope for that function, which makes me think that code might be matching something in a higher. It's possible that that could be causing some issues.

---

I noticed that it looks like you double-posted this question to Stack Overflow around the same time that you posted here.

https://stackoverflow.com/questions/69502379/lightgbm-custom-objective-function-doesnt-work-as-expected

Please don't do this. It can lead to duplicate effort. For example, I monitor the lightgbm tag on Stack Overflow, and it's possible that I could have seen your Stack Overflow post first and started working on an answer at the same time that @StrikerRUS saw your post here and started working on an answer. That would be a bad outcome for the LightGBM project, not just us personally, since we have limited time to spend on maintenance work here.

I'm also requesting that you not do this because, in my experience, we regularly see double-posting where the poster moves on after getting their answer in one place and does not go back to update the other place. That can lead to dead-ends for people facing the same issue in the future who find the unresolved post from search engines.

[JivanRoquet]

I understand your comment about the double-posting on here and Stack Overflow. I just deleted my question on Stack Overflow for this topic to be kept here.

Will post a reproducible example asap.

[sasha-kap]

If it helps @JivanRoquet or anyone else, I created a quick notebook that shows how to reproduce the results of LightGBM's built-in Poisson objective and evaluation functions via custom functions.

The three-step process (calculating first- and second-order derivatives of the objective function, implementing objective and evaluation functions, and then using them with LightGBM's .fit() or .train() API) is described in more detail here.

[no-response]

This issue has been automatically closed because it has been awaiting a response for too long. When you have time to to work with the maintainers to resolve this issue, please post a new comment and it will be re-opened. If the issue has been locked for editing by the time you return to it, please open a new issue and reference this one. Thank you for taking the time to improve LightGBM!

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues
including a reference to this.