Windows system supports Ninja compilation #31161
Conversation
… Ninja, test=develop
Avin0323
changed the title
… compile-with-ninja
Avin0323
changed the title
| # file is that the compilation parameters are changed to 'FIwinapifamily.h'. This | ||
| # patch command can be removed when upgrading to a higher version. | ||
| if("${CMAKE_GENERATOR}" STREQUAL "Ninja") | ||
| set(CRYPTOPP_PATCH_COMMAND ${CMAKE_COMMAND} -E copy_if_different "${PADDLE_SOURCE_DIR}/patches/cryptopp/CMakeLists.txt" "<SOURCE_DIR>/") |
There was a problem hiding this comment.
cryptopp写个TODO吧，升级到哪个版本之后，删除patch中内容
| @@ -374,11 +374,16 @@ if (LINUX) | |||
| endif() | |||
|
|
|||
| if (WIN32) | |||
| if("${CMAKE_GENERATOR}" STREQUAL "Ninja") | |||
| set(paddle_framework_lib_path ${CMAKE_CURRENT_BINARY_DIR}) | |||
There was a problem hiding this comment.
这里也能改成 $<TARGET_FILE_DIR: paddle_framework_shared> 这种形式吗
There was a problem hiding this comment.
这个就不行了，后面configure_file不能正确解析cmake的生成表达式。
| @@ -417,11 +422,16 @@ if (LINUX) | |||
| endif() | |||
|
|
|||
| if (WIN32) | |||
| if("${CMAKE_GENERATOR}" STREQUAL "Ninja") | |||
There was a problem hiding this comment.
同configure_file不能正确解析cmake的生成表达式。
| @@ -26,4 +26,7 @@ cc_library(paddle_inference_c_shared SHARED SRCS ${C_API_SRCS} DEPS paddle_infer | |||
| set_target_properties(paddle_inference_c_shared PROPERTIES OUTPUT_NAME paddle_inference_c) | |||
| if(WIN32) | |||
| target_link_libraries(paddle_inference_c_shared shlwapi.lib) | |||
| if("${CMAKE_GENERATOR}" STREQUAL "Ninja") | |||
There was a problem hiding this comment.
analysis_predictor 能否直接作为 paddle_inference_c_shared 的DEP
| @@ -101,11 +101,16 @@ if(WITH_PYTHON) | |||
| set(tmp_impl_file ${impl_file}.tmp) | |||
|
|
|||
| if(WIN32) | |||
| if("${CMAKE_GENERATOR}" STREQUAL "Ninja") | |||
There was a problem hiding this comment.
同上，能否改成 $<TARGET_FILE_DIR: xxx>的形式
There was a problem hiding this comment.
同configure_file不能正确解析cmake的生成表达式。
paddle/scripts/paddle_build.bat
Outdated
| @@ -31,6 +31,7 @@ wmic process where name="op_function_generator.exe" call terminate | |||
| taskkill /f /im python.exe 2>NUL | |||
|
|
|||
| rem ------initialize common variable------ | |||
| set GENERATOR=Ninja | |||
There was a problem hiding this comment.
最后还是 默认设 Visual Studio 14 2015 Win64吧，可以在CI配置里面加这句设 Ninja
paddle/scripts/paddle_build.bat
Outdated
| @@ -146,6 +147,15 @@ rem set CLCACHE_OBJECT_CACHE_TIMEOUT_MS=1000000 | |||
| :: set maximum cache size to 20G | |||
| rem clcache.exe -M 21474836480 | |||
|
|
|||
| :: install ninja if GENERATOR is Ninja | |||
| if "%GENERATOR%" == "Ninja" ( | |||
There was a problem hiding this comment.
Ninja的话，改一下build目录，和第三方库的目录吧，加个后缀，build_ninja，防止和原有的冲突。
There was a problem hiding this comment.
暂时先不用区分build目录吧，这个PR只是支持Ninja构建，暂时不会在CI中使用，等后续使用的时候再换吧？
Avin0323
changed the title
PR types
Others
PR changes
Others
Describe
Windows系统支持使用Ninja编译Paddle代码，仅对Windows系统中生效，对Linux/MacOS系统无影响。
本PR支持Windows系统下Ninja构建系统后，Paddle在Windows下同时可以使用MSVC、Ninja进行构建，为后续依赖Ninja构建系统提升Windows/Windows CI效率体验做准备铺垫，Windows CI及用户文档均保持现状不直接切换Ninja使用，待后续逐步推进。
背景
Ninja是注重速度的小型构件系统，它存在优势：
Windows下支持Ninja的主要原因：
a. 可以在windows CI中使用sccache，与Linux CI中的ccache类似，通过编译器缓存来加速CI编译；
b. Windows CI编译错误时打印的信息过多，不利于快速定位问题，Ninja的日志展示与Linux/MacOS MakeFile类似，可以快速发现问题，提升CI体验；
存在问题
目前Paddle代码在windows上使用Ninja编译存在以下4个问题：
paddle/fluid/pybind/CMakeLists.txt中op_function_generator.exe、mklml.dll、iomp.dll、openblas.dll、mkldnn.dll共五处路径在Ninja和MSVC中有差异；cryptopp使用的CMakeLists.txt中存在一处编译参数FI\"winapifamily.h\"无法在Ninja构建系统中正确解析；extern_protobuf缺失；third_party时提示缺失protoc.exe，编译paddle时提示缺失cryptopp依赖库；解决方案
针对于上述存在的问题，分别做了如下优化修改：
测试结果
测试日志：https://xly.bce.baidu.com/paddlepaddle/paddle/newipipe/detail/2476450/job/3468727
测试日志：https://xly.bce.baidu.com/paddlepaddle/paddle/newipipe/detail/2476451/job/3468728