[R-package] Installation with R4.0.0 on Windows is broken

[mayer79]

Is it possible to install LightGBM with R 4.0.0? So far, I failed with different configs. This is my latest try. On my other Windows machine, I had no troubles today to build under R 3.6.3.

With a hack, it is possible, see my response

R 4.0.0 has been released with a new RTools version with different install paths etc.

Config

• Operating System: Windows 10 - version 10.0.18362 Build 18362
• CPU: Intel(R) Core(TM) i7-7700 CPU @ 3.60GHz
• C++ compiler version: Visual Studio 16 2019, MSVC 19.25.28614.0
• CMake version: 3.17.2
• R version: 4.0.0
• RTools40

PATH

Error

PS C:\Users\Michael\LightGBM> rscript build_r.R

• checking for file 'lightgbm_r/DESCRIPTION' ... OK

• preparing 'lightgbm':

• checking DESCRIPTION meta-information ... OK

• cleaning src

• checking for LF line-endings in source and make files and shell scripts

• checking for empty or unneeded directories
  WARNING: directory 'lightgbm/src/compute/test' is empty

• looking to see if a 'data/datalist' file should be added

• building 'lightgbm_2.3.2.tar.gz'

• installing to library 'C:/Users/Michael/Documents/R/win-library/4.0'

• installing source package 'lightgbm' ...
  ** using staged installation
  ** libs
  installing via 'install.libs.R' to C:/Users/Michael/Documents/R/win-library/4.0/00LOCK-lightgbm/00new/lightgbm
  [1] "Trying to build with: 'Visual Studio 16 2019'"
  -- Selecting Windows SDK version 10.0.15063.0 to target Windows 10.0.18362.
  -- The C compiler identification is MSVC 19.25.28614.0
  -- The CXX compiler identification is MSVC 19.25.28614.0
  -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.25.28610/bin/Hostx64/x64/cl.exe
  -- Check for working C compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.25.28610/bin/Hostx64/x64/cl.exe - works
  -- Detecting C compiler ABI info
  -- Detecting C compiler ABI info - done
  -- Detecting C compile features
  -- Detecting C compile features - done
  -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.25.28610/bin/Hostx64/x64/cl.exe
  -- Check for working CXX compiler: C:/Program Files (x86)/Microsoft Visual Studio/2019/Community/VC/Tools/MSVC/14.25.28610/bin/Hostx64/x64/cl.exe - works
  -- Detecting CXX compiler ABI info
  -- Detecting CXX compiler ABI info - done
  -- Detecting CXX compile features
  -- Detecting CXX compile features - done
  R version passed into FindLibR.cmake: 4.0.0
  Creating R.lib and R.def
  CMake Error at cmake/modules/FindLibR.cmake:48 (message):
  Either gendef.exe or dlltool.exe not found!

  Do you have Rtools installed with its MinGW's bin/ in PATH?
  Call Stack (most recent call first):
  cmake/modules/FindLibR.cmake:204 (create_rlib_for_msvc)
  CMakeLists.txt:76 (find_package)

Comments

dlltool.exe is here:

In that folder, gendef.exe is not available though. On my other machine with old R and old RTools, "gendef.exe" is present. Hmm.

[mayer79]

I manually copied gendef.exe from the old RTools bundle and *magic, magic, installation worked! Do we need to contact Rtools maintainers? Or is it just me?

[gmobaz]

Hi mayer79, exactly the same problem. I have tried different ways of solving it, but with no luck.

[jameslamb]

Summary

For people arriving from a search engine, if you have R 4.0.x, are trying to build the R package on Windows, and get this error:

R version passed into FindLibR.cmake: 4.0.0
Creating R.lib and R.def
CMake Error at cmake/modules/FindLibR.cmake:48 (message):
Either gendef.exe or dlltool.exe not found!

Do you have Rtools installed with its MinGW's bin/ in PATH?
Call Stack (most recent call first):
cmake/modules/FindLibR.cmake:204 (create_rlib_for_msvc)
CMakeLists.txt:76 (find_package)

First, please put a 👍 on this comment so we can get a sense of how many people are facingt the problem.

You have two options for now

1. Open R-package/src/install.libs.R and set use_mingw <- TRUE. This will result in a package where training might be slower, but it should work because MinGW compilers are bundled with all versions of RTools.
2. Try @mayer79 's suggestion from https://github.com/microsoft/LightGBM/issues/3064#issuecomment-626173049...install RTools 3.5.0, then put gendef.exe from there into the bin/ directory of your installation of RTools 4.0
   • You can get old versions of RTools from https://cran.r-project.org/. Click "Download R for Windows", then "Rtools", then "this page" to get old versions

UPDATE: try the fix in #3064 (comment) until #3065 is merged

The rest of the information below is for those who want to discuss why this is broken and how it will get permanently fixed. If you just want to get LightGBM's R package working, you can ignore it and trust a better solution will be coming in the next few days and documented in our official installation instructions for R.

[jameslamb]

Details

Thank you for using LightGBM and for the reports @mayer79, @gmobaz, and @RaminZi ! We don't yet test against R 4.0 (#3024 ) so I was worried this might happen.

I manually copied gendef.exe from the old RTools bundle and *magic, magic, installation worked! Do we need to contact Rtools maintainers? Or is it just me?

Thanks for trying this! It definitely makes sense that if gendef.exe is not bundled with RTools anymore, installs of the R package using Visual Studio compilers will not work. Similar to xgboost's R package, we use gendef.exe to build a .lib file that can be linked into the LighGBM library. The relevant code is here, and the documentation on why this is necessary for Visual Studio projects is here

I will look into this in the next day or two and contact CRAN about it if necessary.

If you have time, I would appreciate if you could try my first suggestion in #3064 (comment) (mingw <- TRUE). I know it's not ideal and I'm committed to getting Visual Studio builds fixed soon, but if you're able to tell me how that goes it will help me understand if there are other R4.0-specific issues with the LightGBM R package.

[jameslamb]

noting here that @RaminZi has reported the same issue with R4.0 on #2714 (comment)

[mayer79]

James, I tested with use_mingw <- TRUE but there is no mingw32-make.exe in C:\rtools40\mingw64\bin (the old rtools had it here). rtools40 is not frozen yet, so I am actually not sure if this is a version undergoing changes or not. Thanks a lot for caring about such installation issues. Very highly appreciated!

[jameslamb]

James, I tested with use_mingw <- TRUE but there is no mingw32-make.exe in C:\rtools40\mingw64\bin (the old rtools had it here). rtools40 is not frozen yet, so I am actually not sure if this is a version undergoing changes or not. Thanks so much for looking into this.

😱 ok thanks! I am going to be looking into this later today. I thought from https://github.com/r-windows/docs/blob/master/faq.md that all the mingw stuff that was in RTools 3.5 would be in 4.0, but maybe I made too many assumptions.

I've been on R3.6.x in my personal dev environment (waiting for R 4.0 and RTools 4.0 to be out a bit longer), so I haven't hit this. Really appreciate your report and the information you gave to help us reproduce it.

Worst-case, we can host gendef.exe ourselves and have the install script download it. That's not ideal but it's also not too terrible, I think.

By the way we are working hard in parallel on getting the package to CRAN to alleviate some of this pain.

[mayer79]

No worries, rtools40 is probably just slightly reorganized. Compiling the R packages itself work, so a compiler is still part of rtools ;-).

lgb on CRAN would be the ultimate thing!

[jameslamb]

No worries, rtools40 is probably just slightly reorganized. Compiling the R packages itself work, so a compiler is still part of rtools ;-).

lgb on CRAN would be the ultimate thing!

I have a pull request coming in the next few days that I think will get us close. If you want to follow the progress you can subscribe to #629 .

The CRAN package would be mingw-only for Windows, since Visual Studio isn't part of the CRAN toolchain for Windows, but for people who are ok with that I think it would make life easier. We'll get there!

[gmobaz]

Thank you @jameslamb , I am sure many R users will appreciate this!

[jameslamb]

I just created #3065 , which introduces a possible fix! It will take some time to get it ready to be merged (especially since I want to wait for #2965 to be merged so we can test with Visual Studio in CI), but if you want to try it you can run the following:

git clone https://github.com/jameslamb/LightGBM.git
cd LightGBM
git fetch origin fix/r-4.0
git checkout fix/r-4.0

# install LightGBM
Rscript build_r.R

# test that it worked
cd R-package/tests
Rscript testthat.R

^ if anyone arriving at this issue tries that, please let me know how it goes! I updated to R 4.0 and Rtools 4.0 on my machine tonight and this is working for me.

[mayer79]

It worked (with VS compiler) - awesome!

[RaminZi]

Worked for me as well.
Thanks James

[gmobaz]

@jameslamb . Worked for me also; great. Thank you! Just a few remarks: I am attaching some screenshots that seem to be minor bugs, but better to pay attention. While performing, Test MM_PREFETCH and Test MM_MALLOC - Failed. Also, there is a warning (in Spanish) that says "The intermediate or the output directory cannot be in the temporary directory as it could lead to problems with incremental compilation_"

Here is a screenshot of the PATH for Rtools, etc..

Also, with CreatePredictionEarlyStopInstance, "not all control paths return a value"

Thanks a lot and feel free to ask for tests if they are needed.

[jameslamb]

Thanks so much for trying it out, and especially to @gmobaz for the thorough report! Those warnings you noted are all current things that I've seen building our project even with R 3.6.x so I don't think we need to worry about them for now.

[jameslamb]

I just created #3067 (and #3068 to fix it) for the warning about CreatePredictionEarlyStopInstance

[jeroen]

To install gendef and mingw32-make.exe, open the rtools40 shell and type:

 pacman -S mingw-w64-{i686,x86_64}-{tools,make}

[jameslamb]

Thanks @jeroen for your help here and your amazing work on the R toolchain for Windows!

I think we will move ahead with my proposal in #3065 , using objdump to generate a .def file and make.exe for building the package, since those tools are bundled in previous Rtools and Rtools40.

I really want to preserve the user experience of our installation "just working" if you have CMake and Rtools.

[srinadhan]

This worked for me also, you are great life saver, I am struggling this for last 2 days

[jameslamb]

Thanks for the report @srinadhan , and for using LightGBM. Glad it is working for you!

The changes in #3065 will be merged in the next few days.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.