Add powerpc build support #4132

danlark1 · 2019-01-23T14:25:23Z

I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en

For changelog. Remove if this is non-significant change.

Category (leave one):

Build/Testing/Packaging Improvement

Short description (up to few sentences):

Added minimal support for powerpc build

cmake/arch.cmake

alexey-milovidov · 2019-01-23T17:17:15Z

Stress test: in AlterCommands: known issue.

alexey-milovidov · 2019-01-23T17:18:27Z

clear_column_in_partition_zookeeper - unrelated, probably flappy test.

dmkoltsov · 2020-03-04T15:36:44Z

@danlark1, @alexey-milovidov - Hello, colleagues. With these commits Out-of-box Clickhouse (from github) have no successful build on ppc64le with gcc-9 or advance toolchain 13 on Ubuntu 18.04. Are these commits enough to build on ppc64le ? Were there any successful trials of clickhouse build on ppc64le platform ? Could you please inform me - are there any howto or tutorial descriptions for this build ?

alexey-milovidov · 2020-03-04T16:07:39Z

We have successfully built and run performance tests of ClickHouse on ppc64le more than one year ago. As we did not implement the support for this type of build in our CI, the build is most likely broken.

dmkoltsov · 2020-03-04T16:26:31Z

Thanks, Alexei, for quick answer!

As one can see, there is required dependency in libhdfs3 module on HWCrc32c code, which is represented in Clickhouse github project as implementation only for x86 platforms.

Is there a branch of Clickhouse on github with ppc64le version of this code segment ?

(e.g. according to this way - https://developer.ibm.com/linuxonpower/2017/08/11/adding-power-optimized-crc-32-existing-code/)

Could you please inform me - Is libhdfs3 module mandatory for successful build of Clickhouse or there is a way to successfully build one without libhdfs3 ?

alexey-milovidov · 2020-03-04T16:32:33Z

Is libhdfs3 module mandatory

No. It's only used for interaction with HFDS (rare use case).
And you may notice that we already have one non-x86 (AArch64) build in our CI that is run for every commit.
ppc64le build can be added in a similar fashion.

dmkoltsov · 2020-03-04T16:36:37Z

I see. Thanks!

dmkoltsov · 2020-03-04T17:55:13Z

There is another problem - with libcpuid on Ubuntu 18.04 ppc64le. There is no such lib for ppc64le. Did you use some stub instead of original lib or there is a way to disable libcpuid component in Clickhouse build process ?

alexey-milovidov · 2020-03-04T18:27:33Z

Yes, cpuid is also optional.
It is automatically disabled on ARM: cmake/find/cpuid.cmake
It can be also disabled for PPC in the same way.

Or, just change the condition from NOT ARM to AMD64.

alexey-milovidov · 2020-03-04T18:28:54Z

If we want to add PPC build to CI, we need to get "toolchain" for cross compilation.
Please look at the build logs for AArch64 for example:

https://clickhouse-builds.s3.yandex.net/0/06c8e0499826c525c0846dccf18ca9b35ce12304/build_log_623232378_1583340152.txt

Also look here: https://clickhouse.tech/docs/en/development/build_cross_arm/

Then we can ensure that the build is not broken.

alexey-milovidov · 2020-03-04T18:32:17Z

https://developer.ibm.com/linuxonpower/advance-toolchain/

alexey-milovidov · 2020-03-04T18:35:31Z

https://developer.ibm.com/linuxonpower/advance-toolchain/advtool-compiler/

dmkoltsov · 2020-03-04T18:56:47Z

Alexey (@alexey-milovidov ), thanks.

With two "disable"-operations for libhdfs3 and libcpuid modules I have finally build Clickhouse on ppc64le just now. Clickhouse shows help message:

# ./dbms/programs/clickhouse local --help

usage: clickhouse-local [initial table definition] [--query <query>]
clickhouse-local allows to execute SQL queries on your data files via single command line call. To do so, initially you need to define your data source and its format. After you can execute your SQL queries in usual manner.
There are two ways to define initial table keeping your data. Either just in first query like this:
    CREATE TABLE <table> (<structure>) ENGINE = File(<input-format>, <file>);
Either through corresponding command line parameters --table --structure --input-format and --file.
Main options:
  --help                                                           produce help message
  -c [ --config-file ] arg                                         config-file path
  -q [ --query ] arg                                               query
  -d [ --database ] arg                                            database
...
...

but I've not checked other functionalities of Clickhouse yet.

(btw: I'm already using at13.0 advance toolchain from ibm.com, as I wrote in my first message in this conversation).

alexey-milovidov · 2020-03-04T20:29:03Z

Most likely everything will work file.
And it gives very high chance that we will add ppc64 build in our CI process.

Another examples of minimal testing after build:

clickhouse local --query "SELECT 1"

Check that the exception handling works:

clickhouse local --query "SELECT throwIf(1)"

alexey-milovidov · 2020-03-05T15:14:31Z

@dmkoltsov

Please share all the build options that were used for ppc64le build.

dmkoltsov · 2020-03-05T16:59:17Z

2 @alexey-milovidov

Here is the final string for build options, standing in ./Clickhouse_src_dir/build/ directory:

# CC=cc CXX=c++ CXXFLAGS="-I/usr/local/include/" cmake -DUSE_INTERNAL_ZLIB_LIBRARY=FALSE -DUSE_INTERNAL_LIBGSASL_LIBRARY=OFF -DENABLE_HDFS=0 ..

But it shoud be said that many preliminary additional changes were applied to source tree before executing these command line string. And only 19.7 and 19.8 branches of Clickhouse project have simply passed cmake configuration step (I've used 19.8 branch).

alexey-milovidov · 2020-03-05T17:38:26Z

@dmkoltsov Thank you! Could you please also share these changes?
You can make a pull request (even if it cannot be merged to master as is, we can use it).

dmkoltsov · 2020-03-10T15:51:27Z

@alexey-milovidov , hello.

Those efforts/changes, which helped me to successfully build Clickhouse on ppc64le platform, are hard to name patches - they mostly look like workaround to achieve the goal.

Nevertheless, here is the successfull sequence of steps to build ClickHouse on Ubuntu 18.04.4 LTS ppc64le:

1) # git clone https://github.com/ClickHouse/ClickHouse;

2) # git checkout -f remotes/origin/19.8
   # git branch
   # git submodule sync
//(branches on github which look like "alive" (= successfully pass the cmake
configuration phase) are 19.7 and 19.8 branches, I chose the newer one;

3) # apt install zlib1g-dev libgsasl7-dev libzstd-dev;

4) install IBM Advance Toolchain 13.0 for Ubuntu 18.04.4 LTS ppc64le (see here
ftp://public.dhe.ibm.com/software/server/POWER/Linux/toolchain/at/ubuntu/dists/bionic/at13.0/binary-ppc64el/):
   # apt install ./advance-toolchain-at13.0-runtime_13.0-1_ppc64el.deb
   # apt install ./advance-toolchain-at13.0-devel_13.0-1_ppc64el.deb
   # export PATH=/opt/at13.0/bin/:$PATH

5) manually install llvm-6.0.1 (look for tar.gz here http://releases.llvm.org/download.html):
   # curl -O <...tar.gz>; tar xvf <...tar.gz>
   # cd <llvm-6.0.1.src.dir>
   # mkdir build; cd build
   # CC=cc CXX=c++ CXXFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0" cmake -DBUILD_SHARED_LIBS=ON -DLLVM_ENABLE_RTTI=ON ..
   # cmake --build . -- -j <ncores>
   # cmake --build . --target install

6) manually install clang-6.0.1 (cfe-6.0.1) (look for tar.gz here http://releases.llvm.org/download.html):
   # curl -O <...tar.gz>; tar xvf <...tar.gz>
   # cd <cfe-6.0.1.src.dir>
   # mkdir build; cd build
   # CC=cc CXX=c++ CXXFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0" cmake -DBUILD_SHARED_LIBS=ON ..
   # cmake --build . -- -j <ncores>
   # cmake --build . --target install

7) manually install lld-6.0.1 (look for tar.gz here http://releases.llvm.org/download.html):
   # curl -O <...tar.gz>; tar xvf <...tar.gz>
   # cd <lld-6.0.1.src.dir>
   # mkdir build; cd build
   # CC=cc CXX=c++ CXXFLAGS="-D_GLIBCXX_USE_CXX11_ABI=0" cmake -DBUILD_SHARED_LIBS=ON ..
   # cmake --build . -- -j <ncores>
   # cmake --build . --target install

8) make workaround changes to file
./<clickhouse_src_dir>/contrib/boost/boost/predef/hardware/simd.h -
remove the following strings from it:
    #include <boost/predef/hardware/simd/x86.h>
    #include <boost/predef/hardware/simd/x86_amd.h>
    #include <boost/predef/hardware/simd/arm.h>
    ...
    [[`BOOST_HW_SIMD_X86_AVAILABLE`] [__predef_detection__]]
    [[`BOOST_HW_SIMD_X86_AMD_AVAILABLE`] [__predef_detection__]]
    [[`BOOST_HW_SIMD_ARM_AVAILABLE` ] [__predef_detection__]]
    ...
    [include ../include/boost/predef/hardware/simd/x86.h]
    [include ../include/boost/predef/hardware/simd/x86_amd.h]
    [include ../include/boost/predef/hardware/simd/arm.h]
    ...
    [section `BOOST_HW_SIMD_X86_*_VERSION`]
    [include ../include/boost/predef/hardware/simd/x86/versions.h]
    [endsect]
    [section `BOOST_HW_SIMD_X86_AMD_*_VERSION`]
    [include ../include/boost/predef/hardware/simd/x86_amd/versions.h]
    [endsect]
    [section `BOOST_HW_SIMD_ARM_*_VERSION`]
    [include ../include/boost/predef/hardware/simd/arm/versions.h]

9) patch the file ./<clickhouse_src_dir>/CMakeLists.txt:
< set (CMAKE_CXX_FLAGS_RELWITHDEBINFO "${CMAKE_CXX_FLAGS_RELWITHDEBINFO} -O3 ${CMAKE_CXX_FLAGS_ADD}")
---
> set (CMAKE_CXX_FLAGS_RELWITHDEBINFO "${CMAKE_CXX_FLAGS_RELWITHDEBINFO} -O2 -Wno-error -D_GLIBCXX_USE_CXX11_ABI=0 ${CMAKE_CXX_FLAGS_ADD}")
< set (CMAKE_C_FLAGS_RELWITHDEBINFO "${CMAKE_C_FLAGS_RELWITHDEBINFO} -O3 ${CMAKE_C_FLAGS_ADD}")
---
> set (CMAKE_C_FLAGS_RELWITHDEBINFO "${CMAKE_C_FLAGS_RELWITHDEBINFO} -O2 -Wno-error -D_GLIBCXX_USE_CXX11_ABI=0 ${CMAKE_C_FLAGS_ADD}")

10) patch the file ./<clickhouse_src_dir>/contrib/librdkafka-cmake/CMakeLists.txt -
add the following strings to the section "set(SRCS" of this file:
   ${RDKAFKA_SOURCE_DIR}/rdkafka_cert.c
   ${RDKAFKA_SOURCE_DIR}/rdkafka_ssl.c

11) patch the file ./<clickhouse_src_dir>/dbms/programs/clang/copy_headers.sh:
< for src_file in $(echo | $CLANG -M -xc++ -std=c++1z -Wall -Werror -msse2 -msse4 -mcx16 -mpopcnt -O3 -g -fPIC -fstack-protector -D_FORTIFY_SOURCE=2 \
---
> for src_file in $(echo | $CLANG -M -xc++ -std=c++1z -Wall -Werror -msse2 -msse4 -mcx16 -mpopcnt -O3 -g -fPIC -fstack-protector -D_FORTIFY_SOURCE=2 -Wno-unused-command-line-argument \

12) make workaround changes to the file ./<clickhouse_src_dir>/cmake/find_cpuid.cmake
(disable "libcpuid" module integration in Clickhouse build):
< if (NOT ARCH_ARM)
---
> if (ARCH_AMD64)
< elseif (NOT ARCH_ARM AND NOT MISSING_INTERNAL_CPUID_LIBRARY)
---
> elseif (ARCH_AMD64 AND NOT MISSING_INTERNAL_CPUID_LIBRARY)

13) configure the Clickhouse project:
   # cd <Clickhouse_src_dir>
   # mkdir build; cd build
   # CC=cc CXX=c++ CXXFLAGS="-I/usr/local/include/" cmake -DUSE_INTERNAL_ZLIB_LIBRARY=FALSE -DUSE_INTERNAL_LIBGSASL_LIBRARY=OFF -DENABLE_HDFS=0 ..

14) build the Clickhouse project:
   # ninja

.

dmkoltsov · 2020-03-10T15:54:02Z

@alexey-milovidov , could this workaround above be transformed on your side to live patch for some modern branch of clickhouse (and upstreamed), e.g. 19.17, 19.18, 20.x.x ?

dmkoltsov · 2020-03-10T16:00:46Z

btw: in Clickhouse 19.8 ppc64le build with the workaround above - I've already met the bug (which, may be, fixed in later versions of Clickhouse) - it's the bug with clickhouse-client program.

When using first query from tutorial:

host :) SELECT
    StartURL AS URL,
    AVG(Duration) AS AvgDuration
FROM tutorial.visits_v1
WHERE StartDate BETWEEN '2014-03-23' AND '2014-03-30'
GROUP BY URL
ORDER BY AvgDuration DESC
LIMIT 10:-] :-] :-] :-] :-] :-] 
SELECT 
    StartURL AS URL, 
    AVG(Duration) AS AvgDuration
FROM tutorial.visits_v1 
WHERE (StartDate >= '2014-03-23') AND (StartDate <= '2014-03-30')
GROUP BY URL
ORDER BY AvgDuration DESC

┌─URL────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┬────────AvgDuration─┐
│ http://itpalanija-pri-patrivative=0&ads_app_user                                                                                                                                                                                                           │              60127 │
│ http://renaul-myd-ukraine                                                                                                                                                                                                                                  │              58938 │
│ http://e.mail=on&default?abid=2061&scd=yes&option?r=city_inter.com/menu&site-zaferio.ru/c/m.ensor.net/ru/login=false&orderStage.php?Brandidatamalystyle/20Mar2014%2F007%2F94dc8d2e06e56ed56bbdd                                                            │              51378 │
│ https://moda/vyikrorable.com/notification                                                                                                                                                                                                                  │            48828.6 │
│ http://karta/Futbol/dynas.com/haberler.ru/messages.yandsearchives/494503_lte_13800200319                                                                                                                                                                   │ 48484.666666666664 │
│ http://karta/Futbol/dynamo.kiev.ua/kawaica.su/648                                                                                                                                                                                                          │            46272.2 │
│ https://moda/vyikroforum1/top.ru/moscow/delo-product/trend_sms/multitryaset/news/2014/03/201000                                                                                                                                                            │ 41531.666666666664 │
│ http:%2F%2Fallback/angleNews                                                                                                                                                                                                                               │  38878.29268292683 │
│ http://xmusic/vstreatings of speeds                                                                                                                                                                                                                        │              36925 │
│ http://bashmelnykh-metode.net/video/#!/video/emberkas.ru/detskij-yazi.com/iframe/default.aspx?id=760928&noreask=1&source                                                                                                                                   │              34323 │
│ http://censonhaber/547-popalientLog=0&strizhki-petro%3D&comeback=search?lr=213&text                                                                                                                                                                        │              31773 │
...
│ http://pogoda.yandsearchives/42231125665656-1-englijskij_bilos-u-zhdeniya_finansy_v_evromani                                                                                                                                                               │                912 │
│ http://e.mail.ru/articles/2014/1928-131606628/1/187236-adress.ru/yandsearch                                                                                                                                                                                │                912 │
│ http://e.mail.ru/article&id=609413&text=Из кино 2014/guns.artings.ru/embed/68399.1395432913                                                                                                                                                                │                912 │
│ http://e.mail.ru/articles/shortcut=true&dir=http://yandex.kz/yandex.html/ru/node/3642                                                                                                                                                                      │                912 │
│ http://pogoda-bir-and-Crime.ru/recept-maski                                                                                                                                                                                                                │                912 │
│ http://ru.golosy-gorba.html?api_id=0&is_vative=0&parent&task/5150.html/ru/?clid=22622/politics                                                                                                                                                             │                912 │
│ http://search?clid=19554009918446                                                                                                                                                                                                                          │                912 │
│ https://arom_smotret_onlajn/2014/03/23/02/03/22/79/haired-foto-ot-joomidor-plus.ru/new/list                                                                                                                                                                │                912 │
│ http://e.mail.ru/yandex.ru/devushki                                                                                                                                                                                                                        │              911.5 │
│ http://news%2F20%26ad%3D13876004/velico-942058936                                                                                                                                                                                                          │  911.3333333333334 │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┴────────────────────┘
  Showed first 10000.

100852 rows in set. Elapsed: 1.694 sec. Processed 1.48 million rows, 19.13 MB (876.17 thousand rows/s., 11.29 MB/s.) 

host:) 

Syntax error: failed at position 1:

LIMIT 10

Expected one of: Query, DETACH, DROP, ATTACH, SYSTEM query, KILL, SELECT subquery, DESC, WITH, list of elements, ALTER query, ALTER TABLE, CREATE TABLE or ATTACH TABLE query, CREATE, Query with output, SHOW PROCESSLIST query, SHOW PROCESSLIST, AST, ANALYZE, RENAME query, RENAME TABLE, SELECT query, possibly with UNION, SET query, SET, SHOW [TEMPORARY] TABLES|DATABASES [[NOT] LIKE 'str'], SHOW, EXISTS or SHOW CREATE query, EXISTS, DESCRIBE, SELECT query, subquery, possibly with UNION, USE query, USE, CHECK TABLE, DESCRIBE query, DROP query, TRUNCATE, INSERT query, INSERT INTO, SELECT, KILL QUERY query, OPTIMIZE query, OPTIMIZE TABLE, SELECT query

host :)

Second query:

host :) SELECT
    sum(Sign) AS visits,
    sumIf(Sign, has(Goals.ID, 1105530)) AS goal_visits,
    (100. * goal_visits) / visits AS goal_percent
FROM tutorial.visits_v1
WHERE (CounterID = 912887) AND (toYYYYMM(StartDate) = 201403) AND (domain(StartURL:-] ) = 'yandex.ru'):-] :-] :-] 
SELECT 
    sum(Sign) AS visits, 
    sumIf(Sign, has(Goals.ID, 1105530)) AS goal_visits, 
    (100. * goal_visits) / visits AS goal_percent
FROM tutorial.visits_v1 

┌──visits─┬─goal_visits─┬──────goal_percent─┐
│ 1676581 │        8554 │ 0.510204994569305 │
└─────────┴─────────────┴───────────────────┘

1 rows in set. Elapsed: 0.032 sec. Processed 1.68 million rows, 19.39 MB (51.76 million rows/s., 597.07 MB/s.) 

host :) 

Syntax error: failed at position 1:

WHERE (CounterID = 912887) AND (toYYYYMM(StartDate) = 201403) AND (domain(StartURL) = 'yandex.ru')

Expected one of: Query, DETACH, DROP, ATTACH, SYSTEM query, KILL, SELECT subquery, DESC, WITH, list of elements, ALTER query, ALTER TABLE, CREATE TABLE or ATTACH TABLE query, CREATE, Query with output, SHOW PROCESSLIST query, SHOW PROCESSLIST, AST, ANALYZE, RENAME query, RENAME TABLE, SELECT query, possibly with UNION, SET query, SET, SHOW [TEMPORARY] TABLES|DATABASES [[NOT] LIKE 'str'], SHOW, EXISTS or SHOW CREATE query, EXISTS, DESCRIBE, SELECT query, subquery, possibly with UNION, USE query, USE, CHECK TABLE, DESCRIBE query, DROP query, TRUNCATE, INSERT query, INSERT INTO, SELECT, KILL QUERY query, OPTIMIZE query, OPTIMIZE TABLE, SELECT query

host :)

So, our further small experiments with SQL-syntax showed - it seems that there is a bug with "interpretation" of last string of such SQL-queries in clickhouse-client.

alexey-milovidov · 2020-03-10T17:11:15Z

@dmkoltsov It should be fixed in the latest version, as we moved from readline to replxx.

danlark1 added 5 commits January 23, 2019 11:32

Docs for multi string search

8bc6eef

Merge branch 'master' of https://github.com/yandex/ClickHouse

e41467d

added powerpc build support

7f4b0f3

fix

f5f5674

fix

f6e6f99

alexey-milovidov added the can be tested label Jan 23, 2019

proller reviewed Jan 23, 2019

View reviewed changes

cmake/arch.cmake Outdated Show resolved Hide resolved

fix of ppc64le definitions

784c3e5

alexey-milovidov merged commit e1c78a3 into ClickHouse:master Jan 23, 2019

alexey-milovidov mentioned this pull request Mar 10, 2020

Add PowerPC build to CI. #9589

Closed

qoega added the no-docs-needed label Mar 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add powerpc build support #4132

Add powerpc build support #4132

danlark1 commented Jan 23, 2019

alexey-milovidov commented Jan 23, 2019

alexey-milovidov commented Jan 23, 2019

dmkoltsov commented Mar 4, 2020 •

edited

Loading

alexey-milovidov commented Mar 4, 2020

dmkoltsov commented Mar 4, 2020 •

edited

Loading

alexey-milovidov commented Mar 4, 2020

dmkoltsov commented Mar 4, 2020

dmkoltsov commented Mar 4, 2020 •

edited

Loading

alexey-milovidov commented Mar 4, 2020

alexey-milovidov commented Mar 4, 2020 •

edited

Loading

alexey-milovidov commented Mar 4, 2020

alexey-milovidov commented Mar 4, 2020

dmkoltsov commented Mar 4, 2020 •

edited

Loading

alexey-milovidov commented Mar 4, 2020

alexey-milovidov commented Mar 5, 2020

dmkoltsov commented Mar 5, 2020 •

edited

Loading

alexey-milovidov commented Mar 5, 2020

dmkoltsov commented Mar 10, 2020 •

edited

Loading

dmkoltsov commented Mar 10, 2020 •

edited

Loading

dmkoltsov commented Mar 10, 2020 •

edited

Loading

alexey-milovidov commented Mar 10, 2020

Add powerpc build support #4132

Add powerpc build support #4132

Conversation

danlark1 commented Jan 23, 2019

alexey-milovidov commented Jan 23, 2019

alexey-milovidov commented Jan 23, 2019

dmkoltsov commented Mar 4, 2020 • edited Loading

alexey-milovidov commented Mar 4, 2020

dmkoltsov commented Mar 4, 2020 • edited Loading

alexey-milovidov commented Mar 4, 2020

dmkoltsov commented Mar 4, 2020

dmkoltsov commented Mar 4, 2020 • edited Loading

alexey-milovidov commented Mar 4, 2020

alexey-milovidov commented Mar 4, 2020 • edited Loading

alexey-milovidov commented Mar 4, 2020

alexey-milovidov commented Mar 4, 2020

dmkoltsov commented Mar 4, 2020 • edited Loading

alexey-milovidov commented Mar 4, 2020

alexey-milovidov commented Mar 5, 2020

dmkoltsov commented Mar 5, 2020 • edited Loading

alexey-milovidov commented Mar 5, 2020

dmkoltsov commented Mar 10, 2020 • edited Loading

dmkoltsov commented Mar 10, 2020 • edited Loading

dmkoltsov commented Mar 10, 2020 • edited Loading

alexey-milovidov commented Mar 10, 2020

dmkoltsov commented Mar 4, 2020 •

edited

Loading

dmkoltsov commented Mar 4, 2020 •

edited

Loading

dmkoltsov commented Mar 4, 2020 •

edited

Loading

alexey-milovidov commented Mar 4, 2020 •

edited

Loading

dmkoltsov commented Mar 4, 2020 •

edited

Loading

dmkoltsov commented Mar 5, 2020 •

edited

Loading

dmkoltsov commented Mar 10, 2020 •

edited

Loading

dmkoltsov commented Mar 10, 2020 •

edited

Loading

dmkoltsov commented Mar 10, 2020 •

edited

Loading