Setting parameters differently per round

[gugatr0n1c]

Hi,

there is way in Python to set learning_rate differently for each boosting round as follows:

• decay learning rates
• learning_rates accepts:
• 1. list/tuple with length = num_boost_round
• 2. function(curr_iter)
• 3. function(curr_iter, total_iter)
     gbm = lgb.train(params,
     lgb_train,
     num_boost_round=10,
     init_model=gbm,
     learning_rates=lambda iter: 0.05 * (0.99 ** iter),
     valid_sets=lgb_eval)

Is it possible to allow this for other parameters as well?
num_leaves
min_data_in_leaf
feature_fraction
bagging_fraction

I tested this in xgboost un-directly, with building not one model with 10k tree, but with 1k models, each with 10 tree. Each model was little bit different and there was boost in accuracy, similar what brings DART. There is no research paper on this. But idea is to build slightly different models.

The flow is than as follows:
1] find optimum parameters,ie num_leaves = 750
2] define values around this point [600, 680, 750, 830, 900]
3] generate list of random choices from this list: length = num_boosting_rounds
4] call train with this list and take for each boosting round parameetr from list

Then this can be stored with one model, not with 1k models.

Randomize tree is one argument, second is as this:
Similary as decreasing learning_rate, we can want to slowly increase min_data_in_leaf to figth overfitting.

thx for consideration

[guolinke]

Yes, it can change most of parameters during training, just implement your own callbacks.

You can follow the implementation of learning rate callback. It achieves this by calling Booster.reset_parameter.

And I just have a check, num_leaves, min_data_in_leaf, feature_fraction and bagging_fraction can be changed during training.

[wxchan]

@guolinke I think we can turn reset_learning_rate to reset_parameter(key, value) in callbacks. I can do it tomorrow and also add docs for callback functions.

Btw, I tried to change metrics during training before, seem not work. But I think this is an expected behavior, right?

[guolinke]

@wxchan , I think it should work now. check the code:
https://github.com/Microsoft/LightGBM/blob/master/src/c_api.cpp#L68-L87

BTW, currently reset logic is not efficient except learning rate. Maybe we can also optimize for some common parameters.

And we need to define which parameters can be reset during training. And throw exception or warning if passed wrong parameters.

[wxchan]

@guolinke I know it's implemented, but it takes very long time when I used it, did you try it?

[guolinke]

@gugatr0n1c I think it can work perfect now.
@wxchan can you also test the performance now? I have pushed some commits. It should be much better now.

[wxchan]

@guolinke tested. Much better now.

[gugatr0n1c]

great thx, I will wait for docs from wxchan about callback function, I am newbie in python, so not sure how to use it

[wxchan]

@gugatr0n1c I already added to docs.
There are only two options now: list/tuple with length = num_boost_round or function(curr_iter).
A simple example how to reset bagging_fraction (0.7 for first 5 iter and 0.6 for 5-10):

gbm = lgb.train(params,
                lgb_train,
                num_boost_round=10,
                init_model=gbm,
                valid_sets=lgb_eval,
                callbacks=[lgb.reset_parameter(bagging_fraction=[0.7]*5+[0.6]*5)])

[gugatr0n1c]

thx for the example,

I just compile latest version and get this error by calling this (was working before update):

num_leaves_list = np.random.randint(500, 1000, iterace).tolist()
model = lg.train(
params,
(matrix_learn, target_learn),
num_boost_round = iterace,
valid_datas = (matrix_test, target_test),
early_stopping_rounds = 50,
callbacks = [lg.reset_parameter(num_leaves = num_leaves_list)]
)

2016-12-18 17:45:41
Traceback (most recent call last):
File "search_37_pole.py", line 167, in
callbacks = [lg.reset_parameter(num_leaves = num_leaves_list)]
TypeError: train() got an unexpected keyword argument 'valid_datas'

I find out that I need to change it to "valid_sets", but then I got this:

Traceback (most recent call last):
File "search_37_pole.py", line 167, in
callbacks = [lg.reset_parameter(num_leaves = num_leaves_list)]
File "/usr/local/lib/python3.5/dist-packages/lightgbm-0.1-py3.5.egg/lightgbm/engine.py", line 93, in train
raise TypeError("Traninig only accepts Dataset object")
TypeError: Traninig only accepts Dataset object

Not sure, what to do now.. my data are numpy ndarrays, thx
But seems to be related to some datasets refactoring in some other issue, right?

[gugatr0n1c]

model = lg.train(
params,
lg.Dataset(matrix_learn, target_learn),
num_boost_round = iterace,
valid_sets = lg.Dataset(matrix_test, target_test),
early_stopping_rounds = 50,
callbacks = [lg.reset_parameter(num_leaves = num_leaves_list)]
)

this is working, so feel free to close this issue

[gugatr0n1c]

ok, one more thing.. I tried just simple test to compare if I am using it right and just called

num_leaves_list = [750] * iterace
model = lg.train(
params,
lg.Dataset(matrix_learn, target_learn),
num_boost_round = iterace,
valid_sets = lg.Dataset(matrix_test, target_test),
early_stopping_rounds = 50,
callbacks = [lg.reset_parameter(num_leaves = num_leaves_list)]
)

and comparing with just model where I set num_leaves = 750, without callback

What I understand this should give same result (not exactly because of stochasticity during training), but callback version give me much worse performance.
Speed of training is good with callback version.

Any idea about this?

[guolinke]

@gugatr0n1c did you use "feature_fraction" or "bagging" ?
if yes, you can try the latest code again.
Thanks!

[gugatr0n1c]

@guolinke yes, I used both in params... now it seems to be working nicely, accuracy is much better, thx!

[gugatr0n1c]

I can confirm now, that this brings boost in accuracy.
For example, if optimal num_leaves = 750, then better is using in callback:

num_leaves_list = np.random.randint(500, 1000, iterace).tolist()

Thx. Closing this issue.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.