Support building with cmake with unity build (build speedup)

[bsergean]

Hi there,

With cmake > 3.16 there is a nice (optional) feature called unity build which help speedup building large or complex C++ projects. Here is a write up on how this works. Speed ups can be pretty large.

However tweaks are sometimes required to get it to work. Typically if you have 2 constants with the same value in 2 .cpp file, a problem happens. Variables with similar name needs to only be declared once.

Here is how I tried to build xgboost but I got those errors which I have a hard time decipher.

$ mkdir build ; cd build
build$ cmake -DCMAKE_UNITY_BUILD=ON -DUSE_OPENMP=OFF -DBUILD_STATIC_LIB=ON .. 
build$ 
build$ make
Scanning dependencies of target dmlc
[  7%] Building CXX object dmlc-core/CMakeFiles/dmlc.dir/Unity/unity_1_cxx.cxx.o
[ 15%] Building CXX object dmlc-core/CMakeFiles/dmlc.dir/Unity/unity_0_cxx.cxx.o
[ 23%] Linking CXX static library libdmlc.a
[ 23%] Built target dmlc
Scanning dependencies of target objxgboost
[ 30%] Building CXX object src/CMakeFiles/objxgboost.dir/Unity/unity_6_cxx.cxx.o
[ 38%] Building CXX object src/CMakeFiles/objxgboost.dir/Unity/unity_5_cxx.cxx.o
In file included from /Users/bsergeant/src/foss/github/xgboost/build/src/CMakeFiles/objxgboost.dir/Unity/unity_5_cxx.cxx:4:
/Users/bsergeant/src/foss/github/xgboost/src/predictor/predictor.cc:12:1: error: explicit specialization of 'Get' after instantiation
DMLC_REGISTRY_ENABLE(::xgboost::PredictorReg);
^
/Users/bsergeant/src/foss/github/xgboost/dmlc-core/include/dmlc/registry.h:236:47: note: expanded from macro 'DMLC_REGISTRY_ENABLE'
  Registry<EntryType > *Registry<EntryType >::Get() {                   \
                                              ^
/Users/bsergeant/src/foss/github/xgboost/src/predictor/cpu_predictor.cc:507:1: note: implicit instantiation first required here
XGBOOST_REGISTER_PREDICTOR(CPUPredictor, "cpu_predictor")
^
/Users/bsergeant/src/foss/github/xgboost/include/xgboost/predictor.h:240:54: note: expanded from macro 'XGBOOST_REGISTER_PREDICTOR'
          ::dmlc::Registry<::xgboost::PredictorReg>::Get()->__REGISTER__(Name)
                                                     ^
1 error generated.
make[2]: *** [src/CMakeFiles/objxgboost.dir/Unity/unity_5_cxx.cxx.o] Error 1
make[1]: *** [src/CMakeFiles/objxgboost.dir/all] Error 2
make: *** [all] Error 2

If anyone is interested or can give us clues on how to fix that it would be appreciated. Thanks !

[trivialfis]

To fix it the factory registry in dmlc core needs to be changed.

[trivialfis]

Or setup SKIP_UNITY_BUILD_INCLUSION for needed source files.

[trivialfis]

Managed to get it working and tried it out. There's some speedup but in general quite similar as XGBoost is not difficult to compile, and CMake unity build doesn't support CUDA.

[bsergean]

Thanks for trying ! On my Linux box xgboost takes quite a bit of time to compile which is why I asked. For one codebase we saw improvements from 6 to 10x. I know that cmake cuda support seems to improve with time, the latest release 3.19 mention improved cuda support so hopefully one day they will support unity builds.

…

On Oct 27, 2020, at 1:35 AM, Jiaming Yuan ***@***.***> wrote: Managed to get it working and tried it out. There's some speedup but in general quite similar as XGBoost is not difficult to compile, and CMake unity build doesn't support CUDA. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6293 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC2O6UPMJA2CNL4XNAFH6XLSM2A37ANCNFSM4TADZQMA>.

[trivialfis]

@bsergean Thank you for bringing it up, it actually helps us on Windows CI build time. :-)

[bsergean]

Well I'm very glad it's helping. If you end up making a new release which has it I'll be more than happy to test it, or I can just use a commit to test it actually.

…

On Oct 27, 2020, at 11:06 PM, Jiaming Yuan ***@***.***> wrote: @bsergean <https://github.com/bsergean> Thank you for bringing it up, it actually helps up on Windows CI build time. :-) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6293 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC2O6UMN2CE4ZWSMNTPORKDSM6YHFANCNFSM4TADZQMA>.

[trivialfis]

We are preparing 1.3 release. But will also keep you posted on merging the PR.

[trivialfis]

Merged. Feel free to try it out on master.