Add 'nrounds' as an alias for 'num_iterations' (fixes #4743) #4746

mikemahoney218 · 2021-10-29T15:15:29Z

This PR addresses #4743, adding nrounds as an alias for num_iterations so that (within the R package) the top-level nrounds parameter may also be passed to the params argument of lightgbm and lgb.train.

I followed the template of #4637 and the new tests work on my machine (:tm:). Please let me know if I missed anything!

ghost · 2021-10-29T15:15:41Z

All CLA requirements met.

StrikerRUS · 2021-10-29T18:35:24Z

@mikemahoney218 Thanks a lot for your contribution! Sorry for the inconvenience, but could you please sync this PR with the latest master branch to fix CI issues?

mikemahoney218 · 2021-10-29T18:41:53Z

@StrikerRUS No problem, should be up-to-date now.

StrikerRUS

Thank you for adding new alias!
Generally LGTM, except one comment about asserting results in the test. But I'll defer the final decision to @jameslamb.

StrikerRUS · 2021-10-29T19:46:49Z

R-package/tests/testthat/test_basic.R

+  expect_equal(param_bst$best_score
+               , top_level_bst$best_score
+               , tolerance = TOLERANCE)
+
+  expect_equal(param_bst$current_iter(), both_customized$current_iter())
+  expect_equal(param_bst$best_score
+               , both_customized$best_score
+               , tolerance = TOLERANCE)


I believe we should check here for the exact equality without any tolerance. Given the fixed environment and the same set of parameters, consecutive calls of lightgbm should produce bit-by-bit identical results. Just like here:

LightGBM/tests/python_package_test/test_basic.py

Lines 66 to 67 in d62378b

# we need to check the consistency of model file here, so test for exact equal

np.testing.assert_array_equal(pred_from_matr, pred_from_model_file)

I agree with @StrikerRUS . And just to be clear, that would mean using testthat::expect_identical(), not just removing tolerance.

Since expect_equal() already allows for small differences by default: https://rdrr.io/cran/testthat/man/equality-expectations.html

I swapped this to expect_identical. When running interactively, this test was non-deterministic; it would sometimes fail with message top_level_l2 not identical to params_l2. Objects equal but not identical.` and sometimes succeed. My attempts to capture this using reprex succeeded every time, so I'm not sure if something about the interactive environment, R Studio, or (most likely) the way I was running tests was causing the issue, but I figured I'd flag all the same.

jameslamb

Thanks so much for picking this up! Changes look good to me, but I left a few requests on the additional unit tests.

jameslamb · 2021-10-29T20:28:43Z

R-package/tests/testthat/test_basic.R

+  expect_equal(param_bst$best_score
+               , top_level_bst$best_score
+               , tolerance = TOLERANCE)


I don't think best_score is the right thing to use here. For any LightGBM training where validation data isn't provided, this will always be NA.

library(lightgbm) data(agaricus.train, package = "lightgbm") train <- agaricus.train nrounds <- 15L top_level_bst <- lightgbm( data = train$data , label = train$label , nrounds = nrounds , params = list( objective = "regression" , metric = "l2" , num_leaves = 5L ) , save_name = tempfile(fileext = ".model") ) top_level_bst$best_score # [1] NA

To test "the model produced is identical", to be more confident that the value of num_iterations was passed through, you can use $eval_train(). This creates predictions on the training data, using the full model, and then provides evaluation metrics based on those predictions.

top_level_l2 <- top_level_bst$eval_train()[[1L]][["value"]] params_l2 <- params_bst$eval_train()[[1L]][["value"]] # check type just to be sure the subsetting didn't return a NULL expect_true(is.numeric(top_level_2)) expect_true(is.numeric(params_l2)) # check that model produces identical performance expect_equal(top_level_l2, params_l2)

Could you make this change?

jameslamb · 2021-10-29T20:30:29Z

R-package/tests/testthat/test_basic.R

+  expect_equal(param_bst$best_score
+               , top_level_bst$best_score
+               , tolerance = TOLERANCE)
+
+  expect_equal(param_bst$current_iter(), both_customized$current_iter())
+  expect_equal(param_bst$best_score
+               , both_customized$best_score
+               , tolerance = TOLERANCE)


I agree with @StrikerRUS . And just to be clear, that would mean using testthat::expect_identical(), not just removing tolerance.

Since expect_equal() already allows for small differences by default: https://rdrr.io/cran/testthat/man/equality-expectations.html

R-package/tests/testthat/test_basic.R

mikemahoney218 · 2021-10-29T21:18:02Z

I believe I've added commits that addressed all your comments, namely:

Use eval_train() rather than best_score()
Testing lgb.train() as well as lightgbm()
Explicit tests comparing current_iter() to nrounds
expect_identical() rather than expect_equal()

I want to mention that expect_identical() often failed when running these tests interactively; I don't know if it was something in how I ran the tests or in how my R Studio is configured that caused this issue. My attempts to capture this via reprex::reprex() always succeeded, so I'm expecting they'll work in a CI environment, but I wanted to flag this anyway.

I'm logging off for the weekend around now, so if you all have any other changes I'll look into them come Monday. Have a good weekend 😄

jameslamb · 2021-10-29T21:52:44Z

Thanks very much for the help! We may leave some additional comments here but no need to respond to them over the weekend. This is not urgent, and we really appreciate you taking the time so far to help out!

I'll try running the tests you've added on my machine and see if I also see nondeterministic behavior, thanks very much for flagging that.

StrikerRUS · 2021-11-10T13:14:38Z

Kindly ping @mikemahoney218

mikemahoney218 · 2021-11-10T13:17:59Z

@StrikerRUS I believe that I'm waiting for you all to review the new changes 😄 At any rate, I'm not aware of any changes that need to be made on my end right now.

StrikerRUS · 2021-11-10T13:22:44Z

@mikemahoney218 Ah, sorry, missed your the most recent commits! Then please fix the following linting errors:

[1] Total linting issues found: 10
/home/runner/work/LightGBM/LightGBM/R-package/tests/testthat/test_basic.R:233:1: style: Trailing whitespace is superfluous.
/home/runner/work/LightGBM/LightGBM/R-package/tests/testthat/test_basic.R:242:1: style: Trailing whitespace is superfluous.
/home/runner/work/LightGBM/LightGBM/R-package/tests/testthat/test_basic.R:535:1: style: Trailing whitespace is superfluous.
/home/runner/work/LightGBM/LightGBM/R-package/tests/testthat/test_basic.R:550:1: style: Trailing whitespace is superfluous.
/home/runner/work/LightGBM/LightGBM/R-package/tests/testthat/test_basic.R:565:1: style: Trailing whitespace is superfluous.
/home/runner/work/LightGBM/LightGBM/R-package/tests/testthat/test_basic.R:581:1: style: Trailing whitespace is superfluous.
/home/runner/work/LightGBM/LightGBM/R-package/tests/testthat/test_basic.R:585:1: style: Trailing whitespace is superfluous.
/home/runner/work/LightGBM/LightGBM/R-package/tests/testthat/test_basic.R:590:1: style: Trailing whitespace is superfluous.
/home/runner/work/LightGBM/LightGBM/R-package/tests/testthat/test_basic.R:594:1: style: Trailing whitespace is superfluous.
/home/runner/work/LightGBM/LightGBM/R-package/tests/testthat/test_basic.R:598:1: style: Trailing whitespace is superfluous.

https://github.com/microsoft/LightGBM/runs/4051576042?check_suite_focus=true#step:3:790

mikemahoney218 · 2021-11-10T13:27:13Z

@StrikerRUS We'll see what CI thinks but should be fixed in 0070402

jameslamb

Changes all look good to me, thanks very much!!

shiyu1994 · 2021-11-11T04:12:37Z

@mikemahoney218 @StrikerRUS @jameslamb Thank you all for the excellent work.

github-actions · 2023-08-23T14:40:48Z

This pull request has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.

Add 'nrounds' as an alias for 'num_iterations'

c84b330

mikemahoney218 requested review from btrotta, chivee, guolinke, henry0312, hzy46, jameslamb, Laurae2, shiyu1994, StrikerRUS and tongwu-sh as code owners October 29, 2021 15:15

mikemahoney218 mentioned this pull request Oct 29, 2021

[R-package] Add 'nrounds' as an alias for 'num_iterations' #4743

Closed

Merge branch 'microsoft:master' into master

2758222

jameslamb added the feature label Oct 29, 2021

StrikerRUS approved these changes Oct 29, 2021

View reviewed changes

jameslamb requested changes Oct 29, 2021

View reviewed changes

Improve tests

d47296c

jameslamb requested changes Oct 29, 2021

View reviewed changes

R-package/tests/testthat/test_basic.R Show resolved Hide resolved

Compare against nrounds directly

3028716

Fix whitespace lints

0070402

jameslamb approved these changes Nov 11, 2021

View reviewed changes

shiyu1994 merged commit 3b6ebd7 into microsoft:master Nov 11, 2021

StrikerRUS mentioned this pull request Jan 6, 2022

[DO NOT MERGE] Release 3.3.2 #4930

Closed

13 tasks

jameslamb mentioned this pull request Oct 7, 2022

[DO NOT MERGE] Release v3.3.3 #5525

Closed

40 tasks

github-actions bot locked as resolved and limited conversation to collaborators Aug 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add 'nrounds' as an alias for 'num_iterations' (fixes #4743) #4746

Add 'nrounds' as an alias for 'num_iterations' (fixes #4743) #4746

mikemahoney218 commented Oct 29, 2021

ghost commented Oct 29, 2021 •

edited by ghost

Loading

StrikerRUS commented Oct 29, 2021

mikemahoney218 commented Oct 29, 2021

StrikerRUS left a comment

StrikerRUS Oct 29, 2021

jameslamb Oct 29, 2021

mikemahoney218 Oct 29, 2021 •

edited

Loading

jameslamb left a comment

jameslamb Oct 29, 2021

jameslamb Oct 29, 2021

mikemahoney218 commented Oct 29, 2021

jameslamb commented Oct 29, 2021

StrikerRUS commented Nov 10, 2021

mikemahoney218 commented Nov 10, 2021

StrikerRUS commented Nov 10, 2021

mikemahoney218 commented Nov 10, 2021

jameslamb left a comment

shiyu1994 commented Nov 11, 2021

github-actions bot commented Aug 23, 2023

	# we need to check the consistency of model file here, so test for exact equal
	np.testing.assert_array_equal(pred_from_matr, pred_from_model_file)

Add 'nrounds' as an alias for 'num_iterations' (fixes #4743) #4746

Add 'nrounds' as an alias for 'num_iterations' (fixes #4743) #4746

Conversation

mikemahoney218 commented Oct 29, 2021

ghost commented Oct 29, 2021 • edited by ghost Loading

StrikerRUS commented Oct 29, 2021

mikemahoney218 commented Oct 29, 2021

StrikerRUS left a comment

Choose a reason for hiding this comment

StrikerRUS Oct 29, 2021

Choose a reason for hiding this comment

jameslamb Oct 29, 2021

Choose a reason for hiding this comment

mikemahoney218 Oct 29, 2021 • edited Loading

Choose a reason for hiding this comment

jameslamb left a comment

Choose a reason for hiding this comment

jameslamb Oct 29, 2021

Choose a reason for hiding this comment

jameslamb Oct 29, 2021

Choose a reason for hiding this comment

mikemahoney218 commented Oct 29, 2021

jameslamb commented Oct 29, 2021

StrikerRUS commented Nov 10, 2021

mikemahoney218 commented Nov 10, 2021

StrikerRUS commented Nov 10, 2021

mikemahoney218 commented Nov 10, 2021

jameslamb left a comment

Choose a reason for hiding this comment

shiyu1994 commented Nov 11, 2021

github-actions bot commented Aug 23, 2023

ghost commented Oct 29, 2021 •

edited by ghost

Loading

mikemahoney218 Oct 29, 2021 •

edited

Loading