MSYS2 Mingw64 Windows Build

[lemonsqueeze]

Hi,

Looks like current Windows builds are based around Visual Studio. Has MSYS2 / Mingw64 been tried to get native Windows binaries ? I'd like to provide a Katago CPU build with Pachi releases (Pachi uses it for joseki purposes) but not too keen on having to bundle all the DLLs along with it, so looking for a more self-contained build.

I'm experimenting with this KataGo MSYS2 build at the moment: (v1.15.3)

https://github.com/lemonsqueeze/KataGo/tree/msys2

Haven't tested extensively but looks like it's doing good now:

• After installing Mingw64 toolchain, first commit gives a dynamic build (links against mingw64 dynamic libs).

• Second commit gives a native binary (depends on windows DLLs only).

$ cmake -DUSE_BACKEND=EIGEN .
-- Building for: Ninja
-- The C compiler identification is GNU 14.2.0
-- The CXX compiler identification is GNU 14.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/msys64/mingw64/bin/cc.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/msys64/mingw64/bin/c++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Building 'katago' executable for GTP engine and other tools.
-- -DUSE_BACKEND=EIGEN, using Eigen CPU backend.
-- You can also specify USE_AVX2 (-DUSE_AVX2=1 on command line) if you have a modern CPU for better performance.
-- Including Git revision in the compiled executable, specify -DNO_GIT_REVISION=1 to disable
-- Found Git: C:/msys64/usr/bin/git.exe (found version "2.47.1")
-- Found Eigen3 at C:/msys64/mingw64/include/eigen3
-- Found ZLIB: C:/msys64/mingw64/lib/libz.a (found version "1.3.1")
CMake Warning at CMakeLists.txt:399 (message):
  WARNING: libzip library was NOT found.  KataGo should still work for
  GTP/matches/analysis if everything else is good, but selfplay for writing
  training data will not be possible.


-- Setting up build for GNU or Clang.
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Enabling GNU-specific build options.
-- Configuring done (1.7s)
-- Generating done (0.1s)
-- Build files have been written to: C:/msys64/home/PC_FIXE/src/katago/cpp

$ ninja
[115/115] Linking CXX executable katago.exe

$ ldd katago.exe
            ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll
            KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL
            KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll
            msvcrt.dll => /c/WINDOWS/System32/msvcrt.dll
            SHELL32.dll => /c/WINDOWS/System32/SHELL32.dll
            msvcp_win.dll => /c/WINDOWS/System32/msvcp_win.dll
            ucrtbase.dll => /c/WINDOWS/System32/ucrtbase.dll
            USER32.dll => /c/WINDOWS/System32/USER32.dll
            win32u.dll => /c/WINDOWS/System32/win32u.dll
            GDI32.dll => /c/WINDOWS/System32/GDI32.dll
            gdi32full.dll => /c/WINDOWS/System32/gdi32full.dll
            SHLWAPI.dll => /c/WINDOWS/System32/SHLWAPI.dll
            WS2_32.dll => /c/WINDOWS/System32/WS2_32.dll
            RPCRT4.dll => /c/WINDOWS/System32/RPCRT4.dll

Tried loading models and generating moves with gtp command, looking good so far.
Also tried runtests and runnnlayertests commands. Unicode support looks good too.

• Third commit is a minimal build that should run pretty much everywhere (no unicode support though)
  It pulls in fewer system DLLs:

 $ ldd katago.exe
            ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll
            KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL
            KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll
            msvcrt.dll => /c/WINDOWS/System32/msvcrt.dll
            SHLWAPI.dll => /c/WINDOWS/System32/SHLWAPI.dll
            WS2_32.dll => /c/WINDOWS/System32/WS2_32.dll
            RPCRT4.dll => /c/WINDOWS/System32/RPCRT4.dll

Changes:

In writetrainingdata.cpp i had to change maxFilesToLoadArg type to int. Not sure what's going on, looks like having size_t here leads to some unsigned long long int in the templates which gcc chokes on. Otherwise the changes are pretty small. It's possible to keep CMAKE_CXX_STANDARD at 14 in cmake config but fileutils.cpp needs fixing then.

[lightvector]

Thanks for the report. Does this approach work with all backends, including compiling and linking with OpenCL/CUDA/TensorRT? The reason I've used Visual Studio was because I knew that Visual Studio supported all of these, and Nvidia's documentation explicitly gives instructions for Visual Studio, so it was easiest to just use Visual Studio for everything.

[lemonsqueeze]

I see, yeah I guess Msys2 is more of a niche thing among Windows developers.

I just tried an OpenCL build from my first commit (dynamic build) :
It worked after installing opencl-icd and opencl-headers packages.
(don't really know what i'm doing here, looks like there are multiple options for opencl)

$ pacman -S mingw-w64-x86_64-opencl-icd
$ pacman -S mingw-w64-x86_64-opencl-headers

$ cmake  -DUSE_BACKEND=OPENCL .
-- Building for: Ninja
-- The C compiler identification is GNU 14.2.0
-- The CXX compiler identification is GNU 14.2.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: C:/msys64/mingw64/bin/cc.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: C:/msys64/mingw64/bin/c++.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Building 'katago' executable for GTP engine and other tools.
-- -DUSE_BACKEND=OPENCL, using OpenCL backend.
-- Including Git revision in the compiled executable, specify -DNO_GIT_REVISION=1 to disable
-- Found Git: C:/msys64/usr/bin/git.exe (found version "2.47.1")
-- Looking for CL_VERSION_3_0
-- Looking for CL_VERSION_3_0 - found
-- Found OpenCL: C:/msys64/mingw64/lib/libOpenCL.dll.a (found version "3.0")
-- Found ZLIB: C:/msys64/mingw64/lib/libz.dll.a (found version "1.3.1")
CMake Warning at CMakeLists.txt:394 (message):
  WARNING: libzip library was NOT found.  KataGo should still work for
  GTP/matches/analysis if everything else is good, but selfplay for writing
  training data will not be possible.


-- Setting up build for GNU or Clang.
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Success
-- Found Threads: TRUE
-- Enabling GNU-specific build options.
-- Configuring done (2.4s)
-- Generating done (0.1s)
-- Build files have been written to: C:/msys64/home/PC_FIXE/src/katago/cpp

$ ninja
[118/118] Linking CXX executable katago.exe

It's quite a bit slower than the OpenCL release build :
Getting 500 playouts/s with a 10b model on an old GeForce GTX 1050 where the release build gets between 800 and 1000.
So almost twice as slow. I guess I screwed up something, or this OpenCL implementation isn't very good.

[lemonsqueeze]

One thing that looks peculiar with these builds is the timestamps.
Looks like Valentine mode got activated =)

Romance Standard Time: GTP Engine Starting...
Romance Standard Time: KataGo v1.15.3
Romance Standard Time: Using Japanese rules initially, unless GTP/GUI overrides this.
...

[lemonsqueeze]

Looks like CUDA under msys2 is not possible (CUDA is tightly coupled to the host compiler and Nvidia chose to only support Visual Studio on Windows). Probably same for TensorRT if it depends on CUDA ?