[R-package] [gpu] Cmake version reported to be low when compiling the R package through build_r.R in Windows

[Vinnish-A]

On my Windows PC, my cmake version is 3.30

PS F:\0Local\project\LightGBM> cmake --version
cmake version 3.30.2

CMake suite maintained and supported by Kitware (kitware.com/cmake).

However, when running Rscript build_r.R --use-gpu --opencl-library="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/lib/x64/OpenCL.lib" --boost-librarydir=G:/boost_1_85_0/lib64-msvc-14.3 command, it prompts that my cmake version is too low.

PS F:\0Local\project\LightGBM> Rscript build_r.R  --use-gpu  --opencl-library="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/lib/x64/OpenCL.lib"  --boost-librarydir=G:/boost_1_85_0/lib64-msvc-14.3
Warning message:
In readLines(DESCRIPTION_FILE) :
  incomplete final line found on 'F:/0Local/project/LightGBM/lightgbm_r/DESCRIPTION'
* checking for file 'F:/0Local/project/LightGBM/lightgbm_r/DESCRIPTION' ... OK
* preparing 'lightgbm':
* checking DESCRIPTION meta-information ... OK
* cleaning src
* installing the package to build vignettes
      -----------------------------------
* installing *source* package 'lightgbm' ...
** using staged installation
** libs
installing via 'install.libs.R' to C:/Users/vinnish/AppData/Local/Temp/RtmpGQya0t/Rinst56cc3ebf1eaa/00LOCK-lightgbm/00new/lightgbm
Trying 'Visual Studio 17 2022'
-- Configuring incomplete, errors occurred!
CMake Error at CMakeLists.txt:26 (cmake_minimum_required):
  CMake 3.28 or higher is required.  You are running version 3.24.3


Trying 'Visual Studio 16 2019'
-- Configuring incomplete, errors occurred!
CMake Error at CMakeLists.txt:26 (cmake_minimum_required):
  CMake 3.28 or higher is required.  You are running version 3.24.3


Trying 'Visual Studio 15 2017'
-- Configuring incomplete, errors occurred!
CMake Error at CMakeLists.txt:26 (cmake_minimum_required):
  CMake 3.28 or higher is required.  You are running version 3.24.3


Trying 'Visual Studio 14 2015'
-- Configuring incomplete, errors occurred!
CMake Error at CMakeLists.txt:26 (cmake_minimum_required):
  CMake 3.28 or higher is required.  You are running version 3.24.3


Warning in eval(ei, envir) :
  Building with Visual Studio failed. Attempting with MSYS2
CMake Error: Error: generator : MSYS Makefiles
Does not match the generator used previously: Visual Studio 14 2015
Either remove the CMakeCache.txt file and CMakeFiles directory or choose a different binary directory.
CMake Error: Error: generator : MSYS Makefiles
Does not match the generator used previously: Visual Studio 14 2015
Either remove the CMakeCache.txt file and CMakeFiles directory or choose a different binary directory.
Error in .run_shell_command("cmake", c(cmake_args, "..")) :
  Command failed with exit code: 1
* removing 'C:/Users/vinnish/AppData/Local/Temp/RtmpGQya0t/Rinst56cc3ebf1eaa/lightgbm'
      -----------------------------------
ERROR: package installation failed
Error in .run_shell_command("R", r_build_args) :
  Command failed with exit code: 1
Execution halted

[jameslamb]

Thanks for using {lightgbm}, sorry about that!

To work around it, change this line to 3.18 in your local copy of LightGBM (should work for you, as the only newer CMake features we've added do not affect the R build):

LightGBM/CMakeLists.txt

Line 26 in 9a76aae

cmake_minimum_required(VERSION 3.28)

• What version of R and Rtools do you have?
• what does this print?

  •  Rscript -e "print(Sys.which('cmake'))"

I hadn't considered this in #6550 (comment) ... R builds might end up finding the CMake that's bundled with Rtools, and maybe people who have an older Rtools than the ones used in LightGBM's CI will have an older CMake that's harder to update.

If you can answer those questions above, it'll help us debug this.

[Arup-Chauhan]

@jameslamb I guess the confusion might occur due to different CMake (bundled or installed external).

What we can do is,: Add a CMake version check in the build_r.R script to verify the path and version of CMake being used. This function will ensure that the correct version of CMake (3.28 or higher) is detected before the build process begins.

If an outdated or bundled version is found, users will be prompted to update CMake or specify the correct path, preventing build failures related to CMake compatibility.

I have not tried this out but maybe this could work?

[jameslamb]

Thanks for the suggestion, but I don't want to add more complexity to build_r.R to get behavior that CMake already provides.

Let's wait to hear the answers to the questions I asked before jumping to an implementation.

[Arup-Chauhan]

Thanks for the suggestion, but I don't want to add more complexity to build_r.R to get behavior that CMake already provides.

Let's wait to hear the answers to the questions I asked before jumping to an implementation.

Hi @jameslamb understandable, would keep a lookout on this issue.

[Vinnish-A]

Thank you for your generous reply!

1. I' m using R4.3.1 and RTools43, here' s my session infomation.

PS F:\0Local\project\LightGBM> Rscript -e "sessionInfo()"
R version 4.3.1 (2023-06-16 ucrt)
Platform: x86_64-w64-mingw32/x64 (64-bit)
Running under: Windows 11 x64 (build 22631)

Matrix products: default


locale:
[1] LC_COLLATE=Chinese (Simplified)_China.utf8
[2] LC_CTYPE=Chinese (Simplified)_China.utf8
[3] LC_MONETARY=Chinese (Simplified)_China.utf8
[4] LC_NUMERIC=C
[5] LC_TIME=Chinese (Simplified)_China.utf8

time zone: Asia/Shanghai
tzcode source: internal

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base

other attached packages:
[1] paperTools_0.1.0

loaded via a namespace (and not attached):
 [1] compiler_4.3.1  magrittr_2.0.3  cli_3.6.1       tools_4.3.1
 [5] pillar_1.9.0    glue_1.6.2      tibble_3.2.1    utf8_1.2.4
 [9] fansi_1.0.5     vctrs_0.6.4     stringi_1.8.2   stringr_1.5.1
[13] lifecycle_1.0.4 pkgconfig_2.0.3 rlang_1.1.3     purrr_1.0.2

2. The running result in the command line is as follows:

PS F:\0Local\project\LightGBM> Rscript -e "print(Sys.which('cmake'))"
                                       cmake
"C:\\rtools43\\X86_64~1.POS\\bin\\cmake.exe"

3. After modifying the relevant content in the CMakeLists.txt file to cmake_minimum_required(VERSION 3.18), run the same commands to build, errors are:

log.txt

[Vinnish-A]

By using pip install lightgbm --config-settings=cmake.define.USE_GPU=ON --config-settings=cmake.define.OpenCL_INCLUDE_DIR="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/include" --config-settings=cmake.define.Boost_ROOT="G:\boost_1_85_0" --config-settings=cmake.define.BOOST_LIBRARYDIR="G:\boost_1_85_0\lib64-msvc-14.3", I successfully built lightgbm in python.

But Rscript build_r.R --use-gpu --opencl-include-dir="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/include" --opencl-library="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/lib/x64/OpenCL.lib" --boost-librarydir="G:/boost_1_85_0/lib64-msvc-14.3" --boost-root="G:\boost_1_85_0" remained error.

[jameslamb]

Thanks for that!

Can you please try installing the latest rtools43 from https://cran.r-project.org/bin/windows/Rtools/rtools43/rtools.html and try again?

According to https://cran.r-project.org/bin/windows/Rtools/rtools43/news.html, the latest release of rtools43 contains CMake 3.28.1:

These packages have been updated:
...
cmake-host 3.24.3 to 3.28.1
...

That'd be new enough to compile the latest development version of LightGBM.

[Vinnish-A]

Thank you for your reply. I have reinstalled the latest version of rtools43 and passed the check with cmake minimum version 3.28. Now when I rerun Rscript build_r.R --no-build-vignettes --use-gpu --opencl-include-dir="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/include" --opencl-library="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/lib/x64/OpenCL.lib" --boost-librarydir="G:/boost_1_85_0/lib64-msvc-14.3" --boost-root="G:\boost_1_85_0", the error message is as follows:

log2.txt

These two issues are troubling me. Do I not need to install Boost externally? The specified path is not working.

-- Found Boost: C:/rtools43/x86_64-w64-mingw32.static.posix/lib/cmake/Boost-1.84.0/BoostConfig.cmake (found suitable version "1.84.0", minimum required is "1.56.0") found components: filesystem system
... ...
CMake Warning:
  Manually-specified variables were not used by the project:

    BOOST_LIBRARYDIR

[jameslamb]

Do I not need to install Boost externally?

You do not need to install it externally. It's provided as a git submodule here.

Could you try this?

git submodule update --init --recursive
Rscript build_r.R --use-gpu

If that succeeds, then we'll know that the issue you're facing has something to do with passing a custom Boost to the R build. And if that's true, we'll probably close this report and ask you to subscribe to #5960.

the error message is as follows:

Thanks for that, but the contents of that file are a bit difficult for me to read (in a text editor on my Mac).

Are the error messages you see the same ones reported at #6537?

[Vinnish-A]

Thanks for your reply!

git submodule update --init --recursive
Rscript build_r.R --use-gpu

I tried this, and it didn' t work.

And I have checked issue #6537. It seems we got the same error!

[jameslamb]

Thanks for checking!

Ok, if you don't mind could you please try 2 more ways, and see if either of them helps?

Rscript build_r.R --use-gpu --use-msys2
Rscript build_r.R --use-gpu --use-mingw

[Vinnish-A]

Wow, Rscript build_r.R --use-gpu --use-msys2 works!

message:

[100%] Linking CXX shared library /C/Users/vinnish/AppData/Local/Temp/RtmpKEfiXW/R.INSTALL30c4db078dc/lightgbm/src/lightgbm.dll
[100%] Built target _lightgbm
Found library file: C:\Users\vinnish\AppData\Local\Temp\RtmpKEfiXW\R.INSTALL30c4db078dc/lightgbm/src/lightgbm.dll to move to G:/R-4.3.1/library/00LOCK-lightgbm/00new/lightgbm/libs/x64
Removing 'build/' directory
** R
** data
** demo
** inst
** byte-compile and prepare package for lazy loading
** help
*** installing help indices
*** copying figures
** building package indices
** installing vignettes
** testing if installed package can be loaded from temporary location
** testing if installed package can be loaded from final location
** testing if installed package keeps a record of temporary installation path
* DONE (lightgbm)

test:

> model = lgb.train(params, dtrain)
[LightGBM] [Info] This is the GPU trainer!!
[LightGBM] [Info] Total Bins 214
[LightGBM] [Info] Number of data points in the train set: 6513, number of used features: 107
[LightGBM] [Info] Using GPU Device: NVIDIA GeForce RTX 4060 Ti, Vendor: NVIDIA Corporation
[LightGBM] [Info] Compiling OpenCL Kernel with 16 bins...
[LightGBM] [Info] GPU programs have been built
[LightGBM] [Info] Size of histogram bin entry: 8
[LightGBM] [Info] 17 dense feature groups (0.07 MB) transferred to GPU in 0.001507 secs. 1 sparse feature groups
[LightGBM] [Info] Start training from score 0.482113

Thank you for your help!

[jameslamb]

Oh great, glad to hear it!

Rtools switched to the msys2 toolchain starting in R 4.0 (if I recall correctly), so it makes sense to me that that'd work better for you than MinGW. Sorry I didn't think of it before!

In theory, LightGBM's R package is supposed to automatically fall back to that when compilation with Visual Studio fails

LightGBM/R-package/README.md

Lines 133 to 141 in 5e4ab24

	**MSYS2 (R 4.x)**
	If you are using R 4.x and installation fails with Visual Studio, `LightGBM` will fall back to using [MSYS2](https://www.msys2.org/). This should work with the tools already bundled in `Rtools` 4.0.
	If you want to force `LightGBM` to use MSYS2 (for any R version), pass `--use-msys2` to the installation script.
	```shell
	Rscript build_r.R --use-msys2
	```

LightGBM/R-package/src/install.libs.R

Lines 162 to 169 in 5e4ab24

	# Rtools 4.0 moved from MinGW to MSYS toolchain. If user tries
	# Visual Studio install but that fails, fall back to the toolchain
	# supported in Rtools
	if (R_ver >= 4.0) {
	windows_toolchain <- "MSYS2"
	} else {
	windows_toolchain <- "MinGW"
	}

But looking at it again... it'd be more accurate to say that LightGBM will fall back to MSYS2 if CMake cannot find Visual Studio and generate a build system based on it

LightGBM/R-package/src/install.libs.R

Lines 192 to 194 in 5e4ab24

	visual_studio_succeeded <- .generate_vs_makefiles(cmake_args)
	if (!isTRUE(visual_studio_succeeded)) {
	warning(sprintf("Building with Visual Studio failed. Attempting with %s", windows_toolchain))

In your case, Visual Studio 17 2022 was found. I see this in the logs you provided:

Trying 'Visual Studio 17 2022'
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.
-- The C compiler identification is MSVC 19.37.32824.0
-- The CXX compiler identification is MSVC 19.37.32824.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: G:/Visual Studio/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: G:/Visual Studio/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- R version passed into FindLibR.cmake: 4.3.1
...
Successfully created build files for 'Visual Studio 17 2022'
Building lightgbm.dll

But then later on, compilation failed and also seems to be using MinGW headers (not the MSVC ones)

C:/rtools43/x86_64-w64-mingw32.static.posix/include\malloc.h(80,10): fatal  error C1083: mm_malloc.h: No such file or directory

I don't understand why any Rtools headers would have been used if CMake correctly found MSVC 🤔

@StrikerRUS do you have any ideas? You're a lot better with Windows than I am. The logs I'm referring to are the file attached here: #6599 (comment)

[StrikerRUS]

Sorry, I'm out of ideas right now.

[Arup-Chauhan]

@jameslamb do we prefer using the MSVC headers over MinGW?

[jameslamb]

Sorry, I'm out of ideas right now.

No problem, thanks for looking!

do we prefer using the MSVC headers over MinGW?

If a new-enough MSVC exists on the system, then Rscript build_r.R should use it's toolchain (e.g. compiler, linker, and C/C++ runtimes).

Wow, Rscript build_r.R --use-gpu --use-msys2 works!

Glad to hear this. If you'd be willing to help us a bit more @Vinnish-A , could you share a few more things?

• what paths do you have in environment variable PATH and LD_LIBRARY_PATH?
• does simply Rscript build_r.R (no --use-gpu or --use-msys2) succeed?

It's a little difficult for me to get an interactive Windows development environment (I just have an old Windows laptop from 2017 that's very slow), so if it's easy for you to provide these details it'd help a lot.

[Vinnish-A]

Of course, I will do my best to cooperate with your work.

1. I did not find the environment variable related to LD_LIBRARY_PATH, and I searched, if LD_LIBRARY_PATH is something in Linux, then I will just confidently show my PATH. Here is my PATH environment variable.

> Sys.getenv("PATH") |> str_replace_all(';', '\n') |> cat()
C:\rtools43\x86_64-w64-mingw32.static.posix\bin
C:\rtools43\usr\bin
G:\R-4.3.1\bin\x64
G:\ImageMagick-7.1.1-Q16-HDRI
G:\Rtools35\bin
C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\libnvvp
C:\Windows\system32
C:\Windows
C:\Windows\System32\Wbem
C:\Windows\System32\WindowsPowerShell\v1.0\
C:\Windows\System32\OpenSSH\
C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR
C:\WINDOWS\system32
C:\WINDOWS
C:\WINDOWS\System32\Wbem
C:\WINDOWS\System32\WindowsPowerShell\v1.0\
C:\WINDOWS\System32\OpenSSH\
C:\Program Files\NVIDIA Corporation\Nsight Compute 2022.3.0\
G:\XShell\
G:\Git\cmd
C:\Program Files\Calibre2\
C:\Windows\System32
C:\Program Files\Microsoft SQL Server\Client SDK\ODBC\170\Tools\Binn\
C:\Program Files\dotnet\
G:\cmake\bin
C:\Program Files (x86)\Windows Kits\10\Windows Performance Toolkit\
C:\rtools43\mingw64\bin
C:\rtools43\usr\bin
G:\boost_1_83_0
G:\boost_1_83_0\stage\lib
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\include
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.8\extras
G:\R-4.3.1\bin
C:\Program Files\Bandizip\
G:\CTEX\UserData\miktex\bin
G:\CTEX\MiKTeX\miktex\bin\x64
G:\CTEX\CTeX\ctex\bin
G:\CTEX\CTeX\cct\bin
G:\CTEX\CTeX\ty\bin
G:\CTEX\Ghostscript\gs10.01.1\bin
G:\CTEX\WinEdt
G:\Positron\bin
G:\nodejs\
C:\Program Files\PostgreSQL\16\bin
C:\Users\vinnish\AppData\Local\Programs\Quarto\bin
C:\Users\vinnish\AppData\Local\Microsoft\WindowsApps
G:\PyCharm 2023.2.1\bin
C:\Users\徐\AppData\Roaming\TinyTeX\bin\windows
C:\Users\vinnish\AppData\Local\Programs\EmEditor
C:\Users\vinnish\.dotnet\tools
C:\Users\vinnish\AppData\Roaming\TinyTeX\bin\windows
C:\Users\vinnish\AppData\Roaming\npm
C:\Users\vinnish\AppData\Local\Programs\Quarto\bin
G:\RStudio\resources\app\bin\postback

2. Rscript build_r.R failed.

PS F:\0Local\project\LightGBM> Rscript build_r.R
Warning message:
In readLines(DESCRIPTION_FILE) :
  incomplete final line found on 'F:/0Local/project/LightGBM/lightgbm_r/DESCRIPTION'
* checking for file 'F:/0Local/project/LightGBM/lightgbm_r/DESCRIPTION' ... OK
* preparing 'lightgbm':
* checking DESCRIPTION meta-information ... OK
* cleaning src
* installing the package to build vignettes
* creating vignettes ... OK
* cleaning src
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
WARNING: directory 'lightgbm/src/build' is empty
* looking to see if a 'data/datalist' file should be added
* building 'lightgbm_4.5.0.99.tar.gz'

* installing to library 'G:/R-4.3.1/library'
* installing *source* package 'lightgbm' ...
** using staged installation
** libs
installing via 'install.libs.R' to G:/R-4.3.1/library/00LOCK-lightgbm/00new/lightgbm
Trying 'Visual Studio 17 2022'
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.
-- The C compiler identification is MSVC 19.37.32824.0
-- The CXX compiler identification is MSVC 19.37.32824.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: G:/Visual Studio/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: G:/Visual Studio/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- R version passed into FindLibR.cmake: 4.3.1
-- Creating R.lib and R.def
Creating 'C:/Users/vinnish/AppData/Local/Temp/RtmpGmHXAd/R.INSTALL5f443c1a48f1/lightgbm/src/build/R.def' from 'G:/R-4.3.1/bin/x64/R.dll'
Successfully created 'C:/Users/vinnish/AppData/Local/Temp/RtmpGmHXAd/R.INSTALL5f443c1a48f1/lightgbm/src/build/R.def'
-- Found LibR: G:/R-4.3.1
-- LIBR_EXECUTABLE: G:/R-4.3.1/bin/x64/R.exe
-- LIBR_INCLUDE_DIRS: G:/R-4.3.1/include
-- LIBR_CORE_LIBRARY: G:/R-4.3.1/bin/x64/R.dll
-- Found OpenMP_C: -openmp (found version "2.0")
-- Found OpenMP_CXX: -openmp (found version "2.0")
-- Found OpenMP: TRUE (found version "2.0")
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Failed
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Failed
-- Looking for inet_pton
-- Looking for inet_pton - found
-- Configuring done (31.7s)
-- Generating done (0.1s)
-- Build files have been written to: C:/Users/vinnish/AppData/Local/Temp/RtmpGmHXAd/R.INSTALL5f443c1a48f1/lightgbm/src/build
Successfully created build files for 'Visual Studio 17 2022'
Building lightgbm.dll
 .NET Framework MSBuild 汾 17.7.2+d6990bcfa

G:\Visual Studio\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(541,5): warning MSB8029: мĿ¼Ŀ¼޷פʱĿ¼£Ϊܻᵼɳ⡣ [C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\ZERO_CHECK.vcxproj]
  1>Checking Build System
G:\Visual Studio\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(541,5): warning MSB8029: мĿ¼Ŀ¼޷פʱĿ¼£Ϊܻᵼɳ⡣ [C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\lightgbm_capi_objs.vcxproj]
  Building Custom Rule C:/Users/vinnish/AppData/Local/Temp/RtmpGmHXAd/R.INSTALL5f443c1a48f1/lightgbm/src/CMakeLists.txt
  c_api.cpp
  lightgbm_R.cpp
C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\external_libs\fast_double_parser\include\fast_double_parser.h(1,1): warning C4819: ļڵǰҳ(936)бʾַ뽫ļΪ Unicode ʽԷֹݶʧ (Դļ C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\src\c_api.cpp) [C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\lightgbm_capi_objs.vcxproj]
C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\external_libs\fast_double_parser\include\fast_double_parser.h(1,1): warning C4819: ļڵǰҳ(936)бʾַ뽫ļΪ Unicode ʽԷֹݶʧ (Դļ C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\src\lightgbm_R.cpp) [C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\lightgbm_capi_objs.vcxproj]
C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\src\lightgbm_R.cpp(97,54): warning C4100: writeable: δõβ [C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\lightgbm_capi_objs.vcxproj]
C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\src\lightgbm_R.cpp(175,54): warning C4100: writeable: δõβ [C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\lightgbm_capi_objs.vcxproj]
C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\src\lightgbm_R.cpp(1227,3): warning C4244: : ӡR_xlen_tתintܶʧ [C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\lightgbm_capi_objs.vcxproj]
C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\src\c_api.cpp(1694,64): warning C4244: : ӡint64_tתLightGBM::data_size_tܶʧ [C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\lightgbm_capi_objs.vcxproj]
C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\src\c_api.cpp(1691,67): warning C4244: : ӡint64_tתintܶʧ [C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\lightgbm_capi_objs.vcxproj]
G:\Visual Studio\VC\Tools\MSVC\14.37.32822\include\xutility(4612,18): warning C4244: =: ӡdoubleת_Tyܶʧ [C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\lightgbm_capi_objs.vcxproj]
          with
          [
              _Ty=float
          ] (Դļ C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\src\lightgbm_R.cpp)
G:\Visual Studio\VC\Tools\MSVC\14.37.32822\include\xutility(4624,31): message : 鿴ڱĺ ģ ʵ_OutIt std::_Copy_unchecked<_InIt,_InIt,_OutIt>(_InIt,_Sent,_OutIt) [C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\lightgbm_capi_objs.vcxproj]
          with
          [
              _OutIt=float *,
              _InIt=double *,
              _Sent=double *
          ] (Դļ C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\src\lightgbm_R.cpp)
C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\src\lightgbm_R.cpp(500,10): message : 鿴ڱĺ ģ ʵ_OutIt *std::copy<double*,float*>(_InIt,_InIt,_OutIt) [C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\lightgbm_capi_objs.vcxproj]
          with
          [
              _OutIt=float *,
              _InIt=double *
          ]
  lightgbm_capi_objs.vcxproj -> C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\lightgbm_capi_objs.dir\Release\lightgbm_capi_objs.lib
G:\Visual Studio\MSBuild\Microsoft\VC\v170\Microsoft.CppBuild.targets(541,5): warning MSB8029: мĿ¼Ŀ¼޷פʱĿ¼£Ϊܻᵼɳ⡣ [C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\lightgbm_objs.vcxproj]
  Building Custom Rule C:/Users/vinnish/AppData/Local/Temp/RtmpGmHXAd/R.INSTALL5f443c1a48f1/lightgbm/src/CMakeLists.txt
  boosting.cpp
  gbdt_model_text.cpp
  gbdt_prediction.cpp
  gbdt.cpp
  prediction_early_stop.cpp
  sample_strategy.cpp
  bin.cpp
  config_auto.cpp
  config.cpp
  dataset_loader.cpp
  dataset.cpp
  file_io.cpp
C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\external_libs\fast_double_parser\include\fast_double_parser.h(1,1): warning C4819: ļڵǰҳ(936)бʾַ뽫ļΪ Unicode ʽԷֹݶʧ (Դļ C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\src\boosting\gbdt.cpp) [C:\Users\vinnish\AppData\Local\Temp\RtmpGmHXAd\R.INSTALL5f443c1a48f1\lightgbm\src\build\lightgbm_objs.vcxproj]
...

[jameslamb]

Thanks so much for the quick response!

LD_LIBRARY_PATH is something in Linux

It is generally only used on Unix-like operating systems, but I thought maybe it'd matter for MinGW. Anyway, if you don't have it set that's good... one less thing to worry about 😁

Here is my PATH environment variable.

Thanks! These logs look pretty similar to successful compilation with MSVC 2022 and R 4.3 in LightGBM's CI: https://github.com/microsoft/LightGBM/actions/runs/10415299953/job/28845578407#step:10:114

Some things to try based on what I see in PATH:

• can you remove G:\Rtools35\bin?
  • having multiple competing Rtools toolchains could be causing some issues
• could you put C:\rtools43\usr\bin BEFORE C:\rtools43\x86_64-w64-mingw32.static.posix\bin in PATH?
  • that's what we do in CI here, and I vaguely remember that being important:

    LightGBM/.ci/test-r-package-windows.ps1

    Line 97 in d67ecf9

    $env:PATH = "$env:RTOOLS_BIN;" + "$env:RTOOLS_MINGW_BIN;" + "$env:R_LIB_PATH/R/bin/x64;" + "$env:CMAKE_PATH/cmake-$env:CMAKE_VERSION-windows-x86_64/bin;" + $env:PATH

And please try completely deleting the lightgbm_r/ directory at the root of the repo before re-running Rscript build_r.R.

[Vinnish-A]

I did as you said, but the environment variable C:\rtools43\x86_64-w64-mingw32.static.posix\bin is not originally in my path. However, it was printed out in R. I think R dynamically modified it when starting up, so I added it to the Path and placed it under C:\rtools43\usr\bin.

In addition, I deleted G:\Rtools35\bin and lightgbm_r/, then reran Rscript build_r.R, successful!

Rscript build_r.R --use-gpu failed, errors are:

PS F:\0Local\project\LightGBM> Rscript build_r.R  --no-build-vignettes --use-gpu
Warning message:
In readLines(DESCRIPTION_FILE) :
  incomplete final line found on 'F:/0Local/project/LightGBM/lightgbm_r/DESCRIPTION'
* checking for file 'F:/0Local/project/LightGBM/lightgbm_r/DESCRIPTION' ... OK
* preparing 'lightgbm':
* checking DESCRIPTION meta-information ... OK
* cleaning src
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
WARNING: directory 'lightgbm/src/external_libs/compute/test' is empty
* looking to see if a 'data/datalist' file should be added
* building 'lightgbm_4.5.0.99.tar.gz'
Warning in utils::tar(filepath, pkgname, compression = compression, compression_level = 9L,  :
  storing paths of more than 100 bytes is not portable:
  'lightgbm/src/external_libs/compute/include/boost/compute/algorithm/detail/find_extrema_with_atomics.hpp'
Warning in utils::tar(filepath, pkgname, compression = compression, compression_level = 9L,  :
  storing paths of more than 100 bytes is not portable:
  'lightgbm/src/external_libs/compute/include/boost/compute/algorithm/detail/find_extrema_with_reduce.hpp'
Warning in utils::tar(filepath, pkgname, compression = compression, compression_level = 9L,  :
  storing paths of more than 100 bytes is not portable:
  'lightgbm/src/external_libs/compute/include/boost/compute/algorithm/detail/reduce_by_key_with_scan.hpp'
Warning in utils::tar(filepath, pkgname, compression = compression, compression_level = 9L,  :
  storing paths of more than 100 bytes is not portable:
  'lightgbm/src/external_libs/compute/include/boost/compute/iterator/detail/get_base_iterator_buffer.hpp'

* installing to library 'G:/R-4.3.1/library'
* installing *source* package 'lightgbm' ...
** using staged installation
** libs
installing via 'install.libs.R' to G:/R-4.3.1/library/00LOCK-lightgbm/00new/lightgbm
Trying 'Visual Studio 17 2022'
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.
-- The C compiler identification is MSVC 19.37.32824.0
-- The CXX compiler identification is MSVC 19.37.32824.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: G:/Visual Studio/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: G:/Visual Studio/VC/Tools/MSVC/14.37.32822/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- R version passed into FindLibR.cmake: 4.3.1
-- Creating R.lib and R.def
Creating 'C:/Users/vinnish/AppData/Local/Temp/Rtmp46zuAt/R.INSTALL3788629f3100/lightgbm/src/build/R.def' from 'G:/R-4.3.1/bin/x64/R.dll'
Successfully created 'C:/Users/vinnish/AppData/Local/Temp/Rtmp46zuAt/R.INSTALL3788629f3100/lightgbm/src/build/R.def'
-- Found LibR: G:/R-4.3.1
-- LIBR_EXECUTABLE: G:/R-4.3.1/bin/x64/R.exe
-- LIBR_INCLUDE_DIRS: G:/R-4.3.1/include
-- LIBR_CORE_LIBRARY: G:/R-4.3.1/bin/x64/R.dll
-- Found OpenMP_C: -openmp (found version "2.0")
-- Found OpenMP_CXX: -openmp (found version "2.0")
-- Found OpenMP: TRUE (found version "2.0")
-- Looking for CL_VERSION_3_0
-- Looking for CL_VERSION_3_0 - found
-- Found OpenCL: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/lib/x64/OpenCL.lib (found version "3.0")
-- OpenCL include directory: C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v11.8/include
-- Found Boost: C:/rtools43/x86_64-w64-mingw32.static.posix/lib/cmake/Boost-1.84.0/BoostConfig.cmake (found suitable version "1.84.0", minimum required is "1.56.0") found components: filesystem system
-- Performing Test MM_PREFETCH
-- Performing Test MM_PREFETCH - Failed
-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Failed
-- Looking for inet_pton
-- Looking for inet_pton - found
-- Configuring done (29.8s)
-- Generating done (0.1s)
-- Build files have been written to: C:/Users/vinnish/AppData/Local/Temp/Rtmp46zuAt/R.INSTALL3788629f3100/lightgbm/src/build
Successfully created build files for 'Visual Studio 17 2022'
Building lightgbm.dll
 .NET Framework MSBuild 汾 17.7.2+d6990bcfa
...
C:/rtools43/x86_64-w64-mingw32.static.posix/include\stdlib.h(684,56): error C2144: �:intǰӦС; (Դļ C:\Users\vinnish\AppData\Local\Temp\Rtmp0QBXGh\R.INSTALL26e47b6f441c\lightgbm\src\src\treelearner\voting_parallel_tree_learner.cpp) [C:\Users\vinnish\AppData\Local\Temp\Rtmp0QBXGh\R.INSTALL26e47b6f441c\lightgbm\src\build\lightgbm_objs.vcxproj]
C:/rtools43/x86_64-w64-mingw32.static.posix/include\stdlib.h(730,52): error C2169: llabs: ڲܶ (Դļ C:\Users\vinnish\AppData\Local\Temp\Rtmp0QBXGh\R.INSTALL26e47b6f441c\lightgbm\src\src\treelearner\voting_parallel_tree_learner.cpp) [C:\Users\vinnish\AppData\Local\Temp\Rtmp0QBXGh\R.INSTALL26e47b6f441c\lightgbm\src\build\lightgbm_objs.vcxproj]
C:/rtools43/x86_64-w64-mingw32.static.posix/include\malloc.h(80,10): fatal  error C1083: ޷򿪰ļ: mm_malloc.h: No such file or directory (Դļ C:\Users\vinnish\AppData\Local\Temp\Rtmp0QBXGh\R.INSTALL26e47b6f441c\lightgbm\src\src\treelearner\serial_tree_learner.cpp) [C:\Users\vinnish\AppData\Local\Temp\Rtmp0QBXGh\R.INSTALL26e47b6f441c\lightgbm\src\build\lightgbm_objs.vcxproj]
C:/rtools43/x86_64-w64-mingw32.static.posix/include\malloc.h(80,10): fatal  error C1083: ޷򿪰ļ: mm_malloc.h: No such file or directory (Դļ C:\Users\vinnish\AppData\Local\Temp\Rtmp0QBXGh\R.INSTALL26e47b6f441c\lightgbm\src\src\treelearner\tree_learner.cpp) [C:\Users\vinnish\AppData\Local\Temp\Rtmp0QBXGh\R.INSTALL26e47b6f441c\lightgbm\src\build\lightgbm_objs.vcxproj]
C:/rtools43/x86_64-w64-mingw32.static.posix/include\malloc.h(80,10): fatal  error C1083: ޷򿪰ļ: mm_malloc.h: No such file or directory (Դļ C:\Users\vinnish\AppData\Local\Temp\Rtmp0QBXGh\R.INSTALL26e47b6f441c\lightgbm\src\src\treelearner\voting_parallel_tree_learner.cpp) [C:\Users\vinnish\AppData\Local\Temp\Rtmp0QBXGh\R.INSTALL26e47b6f441c\lightgbm\src\build\lightgbm_objs.vcxproj]
Error in .run_shell_command(install_cmd, install_args) :
  Command failed with exit code: 1
Execution halted

[jameslamb]

Thank you SO MUCH for all of the help, I really really appreciate it!

In addition, I deleted G:\Rtools35\bin and lightgbm_r/, then reran Rscript build_r.R, successful!

Excellent!

That narrows it down even further... now we know this is only happening with the GPU variant of the package. That's very useful.

Rscript build_r.R --use-gpu failed, errors are:

I think these are the relevant lines:

-- Performing Test MM_MALLOC
-- Performing Test MM_MALLOC - Failed
...
C:/rtools43/x86_64-w64-mingw32.static.posix/include\malloc.h(80,10): fatal  error C1083: ޷򿪰ļ: mm_malloc.h: No such file or directory (Դļ C:\Users\vinnish\AppData\Local\Temp\Rtmp0QBXGh\R.INSTALL26e47b6f441c\lightgbm\src\src\treelearner\serial_tree_learner.cpp) [C:\Users\vinnish\AppData\Local\Temp\Rtmp0QBXGh\R.INSTALL26e47b6f441c\lightgbm\src\build\lightgbm_objs.vcxproj]
C:/rtools43/x86_64-w64-mingw32.static.posix/include\malloc.h(80,10): fatal  error C1083: ޷򿪰ļ: mm_malloc.h: No such file or directory (Դļ ...

I think we can narrow this down to where the issue is happening, but I'm still confused about how it could happen.

Here's the relevant block of code:

LightGBM/include/LightGBM/utils/common.h

Line 41 in d67ecf9

#if defined(_MSC_VER)

Re-written, with comments:

// if the compiler is MSVC, just us <malloc.h> and assume it already has _mm_malloc() and _mm_free() and include the header
#if defined(_MSC_VER)
#include <malloc.h>

// otherwise, if the compile test run by cmake / autotools detected mm_malloc.h and the expected functions, include the header
#elif MM_MALLOC
#include <mm_malloc.h>

// otherwise, if using a GNU compiler (e.g. gcc) and it says it has memalign(), use that
#elif defined(__GNUC__) && defined(HAVE_MALLOC_H)
  #include <malloc.h>
  #define _mm_malloc(a, b) memalign(b, a)
  #define _mm_free(a) free(a)

// otherwise, fall back to unaligned malloc()
#else
#include <stdlib.h>
#define _mm_malloc(a, b) malloc(a)
#define _mm_free(a) free(a)
#endif

So this seems like just a more specific way to say the same thing I said above... the chosen compiler is MSVC but for some reason it's including the MinGW for standard library stuff instead.

I will investigate this when I can. I'll try to replicate this on my Windows laptop and maybe do a build with cmake --verbose or cmake --trace to see if we can get more information.

If anyone else has ideas please do post them. I'd really like to figure out why this is happening and fix it if possible.

[Arup-Chauhan]

Hi @jameslamb , thanks for clearing the MSVC vs MinGW case,
can we create a check that if MSVC is selected, we force it include MSVC components only?

I understand forcing something is largely undesireable, but multiple compiler components can lead to a break right.

[jameslamb]

I don't understand what you're suggesting, sorry. If you have a Windows developmeny environment, it'd be helpful if you could try to reproduce this error and then try out your ideas to see if they fix it.

[Arup-Chauhan]

I don't understand what you're suggesting, sorry.

Since you mentioned that even if the MSVC is selected, it is taking some MinGW stuff, so I suggested enforcing a compiler check and ensuring that components from there only are loaded.

I do have a Windows development environment, I can look into it, how should i approach it?

If you have a Windows developmeny environment, it'd be helpful if you could try to reproduce this error and then try out your ideas to see if they fix it.

[jameslamb]

suggested enforcing a compiler check and ensuring that components from there only are loaded.

Sure but what is "a compiler check"? Is that R code, CMake, preprocessor definitions in header files, etc.? This is what I don't understand about your proposal.

how should i approach it?

Install similar R, Rtools, and Visual Studio versions as what have been reported above.

Clone the latest version of LightGBM.

Run the build command

Rscript build_r.R --use-gpu

Try making changes to the code and repeat building until you've fixed the error or learn some new information that you think might help us narrow down the root cause further.

[Arup-Chauhan]

Sure but what is "a compiler check"? Is that R code, CMake, preprocessor definitions in header files, etc.? This is what I don't understand about your proposal.

this was a suggestion, I have not followed up with the implementation yet.
Are we using some way to detect what compiler is being used?

[jameslamb]

Are we using some way to detect what compiler is being used?

At this point, I encourage you to clone the repo and look around in the code yourself. Start with build_r.R and CMakeLists.txt.

[Arup-Chauhan]

Are we using some way to detect what compiler is being used?

At this point, I encourage you to clone the repo and look around in the code yourself. Start with build_r.R and CMakeLists.txt.

@jameslamb works, will keep you in loop, thanks!