[python] reset storages in early stopping callback after finishing training

[StrikerRUS]

Right now early_stopping() callback cannot be used with scikit-learn's GridSearchCV tool due to that storages for metric results are not cleaned after early stopping occurs. This results in comparing current results with the best iteration for the first grid iteration for the first fold.

This was spotted by our test_grid_search test

LightGBM/tests/python_package_test/test_sklearn.py

Line 276 in fe535a0

def test_grid_search():

which fails with early stopping setup passed via callback with the following error

>       assert grid.best_estimator_.best_score_['valid_0']['multi_logloss'] < 0.25
E       assert 0.5915701709051111 < 0.25

../tests/python_package_test/test_sklearn.py:318: AssertionError

This PR proposes resetting all global variables in the _init() helper function and determining whether initialization is required by the special variable inited instead of checking cmp_op global variable for emptiness, which results right now in that _init() is called only once during the whole grid search routine.

Extracted from #4846.

[shiyu1994]

Thank you for working on this. Just leave a comment for now since I'm not sure the inited is reset correctly.

[shiyu1994]

It seems that inited is reset to False only when early stopping is not triggered? Because

1. inited is reset to False only when env.iteration == env.end_iteration - 1
2. _final_iteration_check is not called when early stopping is triggered according to
   

   LightGBM/python-package/lightgbm/callback.py

   Lines 306 to 327 in 00f87c5

   	for i in range(len(env.evaluation_result_list)):
   	score = env.evaluation_result_list[i][2]
   	if best_score_list[i] is None or cmp_op[i](score, best_score[i]):
   	best_score[i] = score
   	best_iter[i] = env.iteration
   	best_score_list[i] = env.evaluation_result_list
   	# split is needed for "<dataset type> <metric>" case (e.g. "train l1")
   	eval_name_splitted = env.evaluation_result_list[i][1].split(" ")
   	if first_metric_only and first_metric != eval_name_splitted[-1]:
   	continue # use only the first metric for early stopping
   	if ((env.evaluation_result_list[i][0] == "cv_agg" and eval_name_splitted[0] == "train"
   	or env.evaluation_result_list[i][0] == env.model._train_data_name)):
   	_final_iteration_check(env, eval_name_splitted, i)
   	continue # train data for lgb.cv or sklearn wrapper (underlying lgb.train)
   	elif env.iteration - best_iter[i] >= stopping_rounds:
   	if verbose:
   	eval_result_str = '\t'.join([_format_eval_result(x) for x in best_score_list[i]])
   	_log_info(f"Early stopping, best iteration is:\n[{best_iter[i] + 1}]\t{eval_result_str}")
   	if first_metric_only:
   	_log_info(f"Evaluated only: {eval_name_splitted[-1]}")
   	raise EarlyStopException(best_iter[i], best_score_list[i])
   	_final_iteration_check(env, eval_name_splitted, i)

[StrikerRUS]

@shiyu1994 Thanks for your review!

It seems that inited is reset to False only when early stopping is not triggered?

I believe inited is reset to False when early stopping is triggered here

LightGBM/python-package/lightgbm/callback.py

Lines 339 to 340 in dff622c

	inited = False
	raise EarlyStopException(best_iter[i], best_score_list[i])

Triggering early stopping means to raise EarlyStopException and inited is reset to False right before that line.

Also, if it is not reset, the test test_grid_search() will fail because we use constant custom evaluation metric there which force early stopping to happen at stopping_rounds iteration because there is no improvement after the first iteration.

[shiyu1994]

@StrikerRUS Thanks for the explanation. Sorry that I did not notice line 339.

[shiyu1994]

The changes LGTM. Thank you.

[jameslamb]

thanks very much for the explanation! I think this fix makes sense.

[PhillipMaire]

is this going to be merged to master? having a working grid search and other related packaged like OPTUNA work with light GBM is a must for many people and without this update they still don't work properly with light GBM!! from OPTUNA #3145 "LightGBM 3.3.2, which has been released yesterday, does not include the change. We need to wait for the fix..." and recently OPTUNA #3625

[jameslamb]

is this going to be merged to master

This PR was merged on December 9, 2021. I think you mean "when will this be released to package managed like PyPI".

Due to a lack of maintainer activity, it will probably still be several months until the next release of LightGBM. You can subscribe to #5153 for updates on the next release, and even comment on the linked issues if there are any you'd like to contribute, to move the project closer to that release.

[PhillipMaire]

ahh yes thank you for clarifying this! I installed it on my test notebook and it seems to run fine but I get an error with OPTUNA which I mentioned to them. I installed using the following in case anyone else needs this

!git clone --recursive https://github.com/microsoft/LightGBM.git
!cd LightGBM/python-package && python setup.py install

[github-actions]

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.