Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

【第三方库离线编译】add submodule pocketfft and xbyak #54344

Merged
merged 3 commits into from
Jun 7, 2023
Merged

【第三方库离线编译】add submodule pocketfft and xbyak #54344

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
53478c5
add submodule pocketfft
RedContritio Jun 5, 2023
642e3a0
add submodule xbyak
RedContritio Jun 5, 2023
bf0d829
remove shallow_copy arg
RedContritio Jun 7, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions .gitmodules
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,10 @@
path = third_party/protobuf
url = https://github.com/protocolbuffers/protobuf.git
ignore = dirty
[submodule "third_party/pocketfft"]
path = third_party/pocketfft
url = https://gitlab.mpcdf.mpg.de/mtr/pocketfft.git
ignore = dirty
[submodule "third_party/gflags"]
path = third_party/gflags
url = https://github.com/gflags/gflags.git
Expand Down Expand Up @@ -62,6 +66,10 @@
path = third_party/cutlass
url = https://github.com/NVIDIA/cutlass.git
ignore = dirty
[submodule "third_party/xbyak"]
path = third_party/xbyak
url = https://github.com/herumi/xbyak.git
ignore = dirty
[submodule "third_party/mkldnn"]
path = third_party/mkldnn
url = https://github.com/oneapi-src/oneDNN.git
Expand Down
20 changes: 12 additions & 8 deletions cmake/external/pocketfft.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,32 +19,36 @@ set(POCKETFFT_PATH
CACHE STRING "A path setting for external_pocketfft path.")
set(POCKETFFT_PREFIX_DIR ${POCKETFFT_PATH})

set(POCKETFFT_REPOSITORY https://gitlab.mpcdf.mpg.de/mtr/pocketfft.git)
set(POCKETFFT_TAG release_for_eigen)

set(POCKETFFT_INCLUDE_DIR ${POCKETFFT_PREFIX_DIR}/src)
set(POCKETFFT_SOURCE_DIR ${POCKETFFT_PREFIX_DIR}/src/extern_pocketfft)
message("POCKETFFT_INCLUDE_DIR is ${POCKETFFT_INCLUDE_DIR}")
include_directories(${POCKETFFT_INCLUDE_DIR})

set(POCKETFFT_TAG release_for_eigen)
set(SOURCE_DIR ${PADDLE_SOURCE_DIR}/third_party/pocketfft)

if(APPLE)
file(TO_NATIVE_PATH
${PADDLE_SOURCE_DIR}/patches/pocketfft/pocketfft_hdronly.h.patch
native_dst)
set(POCKETFFT_PATCH_COMMAND
git checkout -- . && git checkout ${GLOO_TAG} && patch -Nd
${POCKETFFT_INCLUDE_DIR}/extern_pocketfft < ${native_dst})
git checkout -- . && git checkout ${POCKETFFT_TAG} && patch -Nd
${SOURCE_DIR} < ${native_dst})
endif()

ExternalProject_Add(
extern_pocketfft
${EXTERNAL_PROJECT_LOG_ARGS} ${SHALLOW_CLONE}
GIT_REPOSITORY ${POCKETFFT_REPOSITORY}
GIT_TAG ${POCKETFFT_TAG}
SOURCE_DIR ${SOURCE_DIR}
RedContritio marked this conversation as resolved.
Show resolved Hide resolved
PREFIX ${POCKETFFT_PREFIX_DIR}
PATCH_COMMAND ${POCKETFFT_PATCH_COMMAND}
UPDATE_COMMAND ""
CONFIGURE_COMMAND ""
BUILD_COMMAND ""
BUILD_COMMAND
COMMAND ${CMAKE_COMMAND} -E remove_directory ${POCKETFFT_SOURCE_DIR}
COMMAND ${CMAKE_COMMAND} -E make_directory ${POCKETFFT_SOURCE_DIR}
COMMAND ${CMAKE_COMMAND} -E copy_directory ${SOURCE_DIR}
${POCKETFFT_SOURCE_DIR}
INSTALL_COMMAND ""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why?为什么要copy目录？？

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

因为现存代码里有对 extern_pocketfft 目录的引用：

Paddle/paddle/phi/kernels/funcs/fft.cc

Lines 21 to 27 in c15e53d

#if defined(PADDLE_WITH_ONEMKL)
#include "paddle/phi/kernels/funcs/mkl_fft_utils.h"
#elif defined(PADDLE_WITH_POCKETFFT)
#define POCKETFFT_CACHE_SIZE 16
#include "extern_pocketfft/pocketfft_hdronly.h"
#endif

为了保留外部代码的一致性，所以需要复制一份到 ${THIRD_PARTY}/pocketfft/src/extern_pocketfft，并且将 ${THIRD_PARTY}/pocketfft/src 作为 include_directories

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果不复制的话，就需要将 ${PADDLE_SOURCE_DIR}/third_party 作为 include_directories 了，我觉得暴露这个目录不太优雅。

一个可选的思路的将保留 extern_pocketfft 这个目录，但是仅复制需要的 "extern_pocketfft/pocketfft_hdronly.h"

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

因为现存代码里有对 extern_pocketfft 目录的引用：

Paddle/paddle/phi/kernels/funcs/fft.cc

Lines 21 to 27 in c15e53d

#if defined(PADDLE_WITH_ONEMKL)
#include "paddle/phi/kernels/funcs/mkl_fft_utils.h"
#elif defined(PADDLE_WITH_POCKETFFT)
#define POCKETFFT_CACHE_SIZE 16
#include "extern_pocketfft/pocketfft_hdronly.h"
#endif

为了保留外部代码的一致性，所以需要复制一份到 ${THIRD_PARTY}/pocketfft/src/extern_pocketfft，并且将 ${THIRD_PARTY}/pocketfft/src 作为 include_directories

#include "extern_pocketfft/pocketfft_hdronly.h" 改为 #include "pocketfft_hdronly.h" 应该也可以吧，然后include_directories(SOURCE_DIR)?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果不复制的话，就需要将 ${PADDLE_SOURCE_DIR}/third_party 作为 include_directories 了，我觉得暴露这个目录不太优雅。

一个可选的思路的将保留 extern_pocketfft 这个目录，但是仅复制需要的 "extern_pocketfft/pocketfft_hdronly.h"

复制也是一种方法，但是系统本身有一份再进行复制是不是冗余操作？这样暴露不太优雅？指的是？

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

如果不复制的话，就需要将 ${PADDLE_SOURCE_DIR}/third_party 作为 include_directories 了，我觉得暴露这个目录不太优雅。
一个可选的思路的将保留 extern_pocketfft 这个目录，但是仅复制需要的 "extern_pocketfft/pocketfft_hdronly.h"

复制也是一种方法，但是系统本身有一份再进行复制是不是冗余操作？这样暴露不太优雅？指的是？

是这样的，参考 pybind 的引用，我是倾向于 #include "<third_part_name>/<header_filename>" 这种形式。

之所以之前没有改成 #include "pocketfft_hdronly.h" ，是因为感觉不能直观看出是第三方库。

pybind 那边的目录结构是 pybind/include/pybind11/xxx.h，所以能够直接将 ${SOURCE_DIR}/include 作为 include 目录；

不优雅指的是，不同于 pybind， pocketfft 目录是展开的，如果想保留 <库名>/<头文件名> 的方式，就可能导致 ${PADDLE_SOURCE_DIR}/third_party 目录暴露，担心后续可能会出现 include 冲突的情况，"${PADDLE_SOURCE_DIR}/third_party/xxx" 和其他位置的 xxx 冲突；

如果修改 INCLUDE_DIR 到 SOURCE_DIR，会导致在 #include 的时候看不到库名。

出于这个考虑，所以直接复制了整个目录到原来的 extern_pocketfft，来保证外部引用的一致。

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

可以，那就复制吧，的确，肉眼第一时间看出来某个第三方库的头文件是比较好的。good job

TEST_COMMAND "")

Expand Down
4 changes: 2 additions & 2 deletions cmake/external/xbyak.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ set(XBYAK_INSTALL_ROOT ${THIRD_PARTY_PATH}/install/xbyak)
set(XBYAK_INC_DIR ${XBYAK_INSTALL_ROOT}/include)
set(XBYAK_REPOSITORY ${GIT_URL}/herumi/xbyak.git)
set(XBYAK_TAG v5.81) # Dec 19, 2019
set(SOURCE_DIR ${PADDLE_SOURCE_DIR}/third_party/xbyak)

include_directories(${XBYAK_INC_DIR})
include_directories(${XBYAK_INC_DIR}/xbyak)
Expand All @@ -33,8 +34,7 @@ add_definitions(-DXBYAK_NO_OP_NAMES)
ExternalProject_Add(
${XBYAK_PROJECT}
${EXTERNAL_PROJECT_LOG_ARGS} ${SHALLOW_CLONE}
GIT_REPOSITORY ${XBYAK_REPOSITORY}
GIT_TAG ${XBYAK_TAG}
SOURCE_DIR ${SOURCE_DIR}
DEPENDS ""
PREFIX ${XBYAK_PREFIX_DIR}
UPDATE_COMMAND ""
Expand Down
2 changes: 1 addition & 1 deletion cmake/third_party.cmake
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -441,7 +441,7 @@ if(WITH_DISTRIBUTE
endif()

if(WITH_XBYAK)
include(external/xbyak) # download, build, install xbyak
include(external/xbyak) # prepare submodule xbyak
list(APPEND third_party_deps extern_xbyak)
endif()

Expand Down
1 change: 1 addition & 0 deletions third_party/pocketfft
Open in desktop
Submodule pocketfft added at ea778e
1 change: 1 addition & 0 deletions third_party/xbyak
Open in desktop
Submodule xbyak added at 4ca043