{lightgbm} v4.0.0 is coming

[jameslamb]

👋 Hello! I'm James, one of the maintainers of LightGBM.

We were excited to see the release of {bonsai}. Thanks so much for making it easier for some R users to work with {lightgbm}! I want to do what I can to support you all.

First, I wanted to let you know that the next release of LightGBM (the entire project, including the R package) will be a major version release with significant breaking changes. See this discussion of v4.0.0: microsoft/LightGBM#5153. We don't have a planned date for that release yet, as we have been struggling from a lack of maintainer attention / activity. But I expect it will be months not weeks from now.

Please open issues at https://github.com/microsoft/LightGBM/issues if there's anything we could do to make {lightgbm} easier to use with {bonsai}.

I'm opening this issue mainly to ask a question. In this period prior to v4.0.0, I'm willing to contribute and maintain the following here:

• patches that make {bonsai} compatible with the latest release on CRAN (v3.3.2) and the upcoming release (v4.0.0) so that the next release of {lightgbm} isn't disruptive to you
• CI job(s) testing against the latest development version of LightGBM

Are you open to such contributions?

Thanks for your time and consideration.

[simonpcouch]

Thanks for the holler, @jameslamb! The clear communication is very much appreciated. :)

We're very much open to those sorts of contributions—I'll keep an eye out for PRs/issues and will gladly follow your lead on staying up to speed with yall's changes.

[jameslamb]

Adding for the benefit of others wanting to work on this: how to test {bonsai} against the development version of {lightgbm}.

REPO_DIR="${HOME}/repos"

# install {bonsai} and its Suggests dependencies
# NOTE: this is important because otherwise tests missing things like {modeldata} are skipped
cd "${REPO_DIR}/bonsai"
Rscript \
    --vanilla \
    -e "devtools::install_dev_deps()"
R CMD INSTALL .

# build {lightgbm} R package from source and install it
git clone \
    --recursive \
    git@github.com:microsoft/LightGBM.git \
    "${REPO_DIR}/lgb-dev"

cd "${REPO_DIR}/lgb-dev"
sh build-cran-package.sh \
    --no-build-vignettes

R CMD INSTALL \
    --with-keep.source \
    ./lightgbm_*.tar.gz

# run the unit tests
Rscript --vanilla -e "devtools::test()"

# (optionally) run the tests with {covr} and get a clickable code coverage report
Rscript \
    --vanilla \
    -e "covr::report(covr::package_coverage(), file = file.path(getwd(), 'coverage.html'))"

If you already have a local installation of LightGBM, you can re-install from the latest development version by running the following.

REPO_DIR="${HOME}/repos"
cd "${REPO_DIR}/lgb-dev"
git checkout master
git pull origin master
git submodule update --recursive

sh build-cran-package.sh \
    --no-build-vignettes

R CMD INSTALL \
    --with-keep.source \
    ./lightgbm_*.tar.gz

[simonpcouch]

Thanks for these notes!

Does the dev version of lightgbm live in https://github.com/microsoft/LightGBM/tree/master/R-package?

Trying to work up a quick example of installing with our usual tooling, though we run into issues with the format of the dev version.

devtools::install_github("microsoft/LightGBM", subdir = "R-package")
#> [EDIT: trauncated]
#> 
#> * checking for file ‘/private/var/folders/6c/w21prsj167b_x82q4_s45t340000gn/T/Rtmpd3B4tt/remotes17aaa6e70a899/microsoft-LightGBM-44fe591/R-package/DESCRIPTION’ ... OK
#> * preparing ‘lightgbm’:
#> * checking DESCRIPTION meta-information ... ERROR
#> Malformed package version.
#> 
#> See section 'The DESCRIPTION file' in the 'Writing R Extensions'
#> manual.
#> Error: Failed to install 'lightgbm' from GitHub:
#>   ! System command 'R' failed

^{Created on 2022-07-18 by the reprex package (v2.0.1)}

[jameslamb]

Yes that's where it lives, but you cannot install it directly from source with something like devtools::install_remotes().

This choice is intentional. We support two different build systems for the R package, described in https://github.com/microsoft/LightGBM/tree/master/R-package#installation.

• CRAN-style package:
  • builds lib_lightgbm.so using Autoconf (the preferred approach for CRAN packages)
• CMake-based package:
  • builds lib_lightgbm.so using an install.libs.R script which runs CMake

The CRAN-style package is optimized for portability. The CMake-based package is optimized for performance. It, for example, can be used to build a GPU-enabled version of the package.

As described in the README, the CRAN-style package can be prepared by running the following.

sh build-cran-package.sh --no-build-vignettes

And the CMake-based package can be prepared by running the following.

Rscript build_r.R --skip-install

[jameslamb]

Also I just modified my "how to test" steps to use devtools::test(), based on #41 (comment). Happy to make any other changes to make those instructions better comply with the way you usually do development here.

[simonpcouch]

Gotcha, much appreciated!

[simonpcouch]

I've really appreciated the attention and support from your end, @jameslamb! We're starting to look towards sending out a CRAN release of this package in the coming weeks. Are there any more changes you'd like to see make it into the upcoming CRAN version of bonsai in anticipation of lightgbm 4.0.0?

[jameslamb]

Exciting!!

Short Answer

There are not any other changes I think {bonsai} requires to be compatible with {lightgbm} 4.0.0.

Longer Answer

I just built {lightgbm} from the latest development version (microsoft/LightGBM@81d4d4d), and then ran {bonsai}'s tests with {covr}.

Happy to say all of the tests passed! 🎉

And looking at the areas not currently covered by tests (e.g. raw multi_predict() predictions and regression multi_predict() predictions ), I don't see anything that looks like it'll be an issue with v3.3.2 or v4.0.0 of {lightgbm}.

I'm not sure when the release of {lightgbm} 4.0.0 will be, and there might still be additional breaking changes made there. But if there are, I'll come report them here and try to help add patches.

Please @ me any time if you need some LightGBM guidance, and please don't hesitate to come open feature requests and bug reports at https://github.com/microsoft/LightGBM/issues if you want to propose changes to {lightgbm}'s behavior.

[simonpcouch]

Sure thing, thanks so much @jameslamb!

[Polligator]

any updates on the release date of v4.0.0?

[jameslamb]

You can subscribe to microsoft/LightGBM#5153 to be notified when it comes out. We don't have an estimated release date at this time.

[simonpcouch]

Many congratulations on the major release, @jameslamb!

[jameslamb]

Thanks so much!! @ me if anything breaks (hopefully not 😅)

[github-actions]

This issue has been automatically locked. If you believe you have found a related problem, please file a new issue (with a reprex: https://reprex.tidyverse.org) and link to this issue.