-
Notifications
You must be signed in to change notification settings - Fork 56
Building ScyllaDB
The instructions provided below specify the steps to build ScyllaDB version 5.4.6 on Linux on IBM Z for the following distributions:
- RHEL (8.8, 8.10, 9.2, 9.4)
- Ubuntu (20.04, 22.04)
Ensure that the kernel you are using is the latest available.
General Notes:
- ScyllaDB and its dependencies are large projects and take a significant amount of time to build. The build can be sped up on large systems by increasing the
-j
parameter passed to make and/or ninja. The build was tested on a machine with 8 vCPUs and 32GB/16GB RAM. When building on a machine with 16GB RAM or less, a large swap file is needed. - Building ScyllaDB takes a large amount of disk space. Ensure you have at least 75GB available on the filesystem you are using. Significantly more space will be necessary if you wish to add debug information to the test binaries.
- A directory
/<source_root>/
will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it. - An installation directory
/<prefix>/
is referred to in these instructions, by default this is/usr/local
. - A target architecture
/<target>/
is referred to in these instructions - this is the value passed to the GCC compiler.
If you want to build ScyllaDB manually, go to STEP 2.
Use the following commands to build ScyllaDB using the build script. Please make sure you have wget installed.
wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ScyllaDB/5.4.6/build_scylladb.sh
bash build_scylladb.sh [-t] [-y] [-z <target>]
Provide -t option for executing build with tests.
Provide -y to install required packages without confirmation.
Provide -z option to select target architecture for build e.g. z13, z14. Defaults to native architecture.
If the build completes successfully, go to STEP 9. In case of error, check logs
for more details or go to STEP 2 to follow manual build steps.
export SOURCE_ROOT=/<source_root>/
export TARGET=native # z13, z14, native
export PREFIX=/usr/local # /<prefix>/
export PATCH_URL=https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ScyllaDB/5.4.6/patch/
-
RHEL 8.X
sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm sudo yum install -y --allowerasing gcc gcc-c++ libatomic libatomic_ops-devel \ java-1.8.0-openjdk-devel lksctp-tools-devel xfsprogs-devel snappy-devel libyaml-devel \ openssl-devel libevent-devel libtasn1-devel libmpcdec libidn2-devel numactl-devel c-ares-devel \ gnutls-devel gnutls-c++ gnutls-dane perl-devel python38 python38-devel python38-pip python38-PyYAML \ python38-setuptools python38-requests make automake git maven ant ninja-build unzip bzip2 wget curl \ xz-devel texinfo libffi-devel libpciaccess-devel libxml2-devel libtool diffutils libtool-ltdl-devel \ trousers-devel p11-kit-devel libunistring-devel libicu-devel readline-devel lua-devel patch systemd-devel \ valgrind-devel cmake hwloc hwloc-devel cryptopp cryptopp-devel lz4 lz4-devel jsoncpp jsoncpp-devel protobuf \ rapidjson-devel stow yaml-cpp yaml-cpp-devel ragel langpacks-en glibc-all-langpacks libdeflate libdeflate-devel \ file abseil-cpp-devel rapidxml-devel doxygen
sudo yum install -y gcc-toolset-12 gcc-toolset-12-libatomic-devel
-
RHEL 9.x
sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm sudo yum install -y --allowerasing gcc gcc-c++ \ libatomic libatomic_ops-devel java-1.8.0-openjdk-devel lksctp-tools-devel xfsprogs-devel \ snappy-devel libyaml-devel openssl-devel libevent-devel libtasn1-devel libmpcdec \ libidn2-devel numactl-devel c-ares-devel gnutls-devel gnutls-c++ gnutls-dane perl-devel \ python3.11 python3.11-devel python3.11-pip python3.11-pyyaml python3.11-setuptools python3.11-requests \ make automake git maven ant ninja-build unzip bzip2 wget curl xz-devel texinfo libffi-devel \ libpciaccess-devel libxml2-devel libtool diffutils libtool-ltdl-devel trousers-devel \ p11-kit-devel libunistring-devel libicu-devel readline-devel lua-devel patch systemd-devel \ valgrind-devel cmake hwloc hwloc-devel cryptopp cryptopp-devel lz4 lz4-devel jsoncpp \ jsoncpp-devel protobuf rapidjson-devel stow yaml-cpp yaml-cpp-devel ragel langpacks-en \ glibc-all-langpacks libdeflate libdeflate-devel abseil-cpp-devel doxygen thrift thrift-devel boost boost-devel \ xxhash xxhash-devel rapidxml-devel
sudo yum install -y gcc-toolset-12 gcc-toolset-12-libatomic-devel
-
Ubuntu (20.04, 22.04)
sudo apt-get update >/dev/null sudo apt-get install -y software-properties-common sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test sudo apt-get update >/dev/null sudo apt-get install -y --no-install-recommends gcc g++ sudo apt-get install -y openjdk-8-jdk libaio-dev systemtap-sdt-dev lksctp-tools xfsprogs \ libyaml-dev openssl libevent-dev libmpfr-dev libmpcdec-dev libssl-dev libsystemd-dev \ libsctp-dev libsnappy-dev libpciaccess-dev libxml2-dev xfslibs-dev libgnutls28-dev \ libiconv-hook-dev liblzma-dev libbz2-dev libxslt-dev libc-ares-dev libprotobuf-dev \ protobuf-compiler libcrypto++-dev libtool perl ant libffi-dev rapidjson-dev automake \ make git maven ninja-build unzip bzip2 wget curl xz-utils texinfo diffutils liblua5.3-dev \ libnuma-dev libunistring-dev python3 python3-pip pigz ragel stow patch locales valgrind \ libudev-dev libdeflate-dev zlib1g-dev doxygen librapidxml-dev
-
Ubuntu 22.04
sudo apt-get install -y libabsl-dev sudo apt-get install -y gcc-12 g++-12 sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-12 12 sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 12 sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-12 12 sudo update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++-12 12
cd "$SOURCE_ROOT"
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh /dev/stdin -y
export PATH="$HOME/.cargo/bin:$PATH"
echo "export PATH=$PATH" >> $ENV_VARS
cargo install cxxbridge-cmd --root $SOURCE_ROOT/cxxbridge
sudo cp -r $SOURCE_ROOT/cxxbridge/. /usr/local
rustup target add wasm32-wasip1
cargo install wasm-opt
export PATH=$PATH:$HOME/.rustup/toolchains/stable-s390x-unknown-linux-gnu/lib/rustlib/s390x-unknown-linux-gnu/bin/gcc-ld
#Only for Ubuntu
export RUSTPATH=~/.cargo/bin
export PATH=$RUSTPATH:$PATH
cd "$SOURCE_ROOT"
URL=https://ftp.gnu.org/gnu/binutils/binutils-2.38.tar.gz
curl -sSL $URL | tar xzf - || error "binutils 2.38"
cd binutils-2.38
mkdir objdir
cd objdir
CC=/usr/bin/gcc ../configure --prefix=/usr --build=s390x-linux-gnu
make -j 8
sudo make install
cd "$SOURCE_ROOT"
URL=https://ftp.gnu.org/gnu/gcc/gcc-12.1.0/gcc-12.1.0.tar.gz
curl -sSL $URL | tar xzf - || error "GCC 12.1.0"
cd gcc-12.1.0
./contrib/download_prerequisites
mkdir objdir
cd objdir
../configure --enable-languages=c,c++ --prefix=${PREFIX} \
--enable-shared --enable-threads=posix \
--disable-multilib --disable-libmpx \
--with-system-zlib --with-long-double-128 --with-arch=zEC12 \
--disable-libphobos --disable-werror \
--build=s390x-linux-gnu --host=s390x-linux-gnu --target=s390x-linux-gnu
make -j 8 bootstrap
sudo make install
export PATH=${PREFIX}/bin${PATH:+:${PATH}}
LD_LIBRARY_PATH=${PREFIX}/lib64${LD_LIBRARY_PATH:+:${LD_LIBRARY_PATH}}
LD_LIBRARY_PATH+=:${PREFIX}/lib
LD_LIBRARY_PATH+=:/usr/lib64
export LD_LIBRARY_PATH
LD_RUN_PATH=${PREFIX}/lib64${LD_RUN_PATH:+:${LD_RUN_PATH}}
LD_RUN_PATH+=:${PREFIX}/lib
LD_RUN_PATH+=:/usr/lib64
export LD_RUN_PATH
sudo rm /usr/bin/s390x-linux-gnu-ld /usr/bin/s390x-linux-gnu-ld.bfd
sudo ln -s /usr/bin/ld /usr/bin/s390x-linux-gnu-ld
sudo ln -s /usr/bin/ld.bfd /usr/bin/s390x-linux-gnu-ld.bfd
sudo ln -s ${PREFIX}/lib/gcc/s390x-linux-gnu/12.1.0 /usr/lib/gcc/s390x-linux-gnu/12.1.0
cd "$SOURCE_ROOT"
wget https://apt.llvm.org/llvm.sh
sed -i 's,add-apt-repository "${REPO_NAME}",add-apt-repository "${REPO_NAME}" -y,g' llvm.sh
chmod +x llvm.sh
sudo ./llvm.sh 17
rm ./llvm.sh
export CC=clang-17
export CXX=clang++-17
sudo ln -s /usr/bin/clang-17 /usr/bin/clang
sudo ln -s /usr/bin/clang++-17 /usr/bin/clang++
export ${SOURCE_ROOT}/llvm-project-llvmorg-17.0.6/llvm/utils/lit/tests/Inputs/lld-features:$PATH
cd "$SOURCE_ROOT"
URL="https://www.python.org/ftp/python/3.11.8/Python-3.11.8.tgz"
curl -sSL $URL | tar xzf -
cd Python-3.11.8
./configure
make
sudo make install
pip3 install --user --upgrade pip
pip3 install --user pyparsing colorama pyyaml boto3 requests pytest traceback-with-variables \
scylla-driver scylla-api-client aiohttp tabulate boto3 pytest-asyncio
- Cmake
cd "$SOURCE_ROOT"
URL=https://github.com/Kitware/CMake/releases/download/v3.17.4/cmake-3.17.4.tar.gz
curl -sSL $URL | tar xzf -
cd cmake-3.17.4
./bootstrap --prefix=/usr
make
sudo make install
- cryptopp
cd "$SOURCE_ROOT"
mkdir cryptopp
cd cryptopp
curl -ksSLO https://github.com/weidai11/cryptopp/archive/refs/tags/CRYPTOPP_8_2_0.zip
unzip CRYPTOPP_8_2_0.zip
cd cryptopp-CRYPTOPP_8_2_0
CXXFLAGS="-std=c++11 -g -O2" make
sudo make install
- hwloc
cd "$SOURCE_ROOT"
URL=https://download.open-mpi.org/release/hwloc/v2.4/hwloc-2.4.1.tar.gz
curl -sSL $URL | tar xzf - || error "hwloc 2.4.1"
cd hwloc-2.4.1
./configure
sudo make install
- yaml-cpp
cd "$SOURCE_ROOT"
URL=https://github.com/jbeder/yaml-cpp/archive/refs/tags/yaml-cpp-0.6.3.tar.gz
curl -sSL $URL | tar xzf -
cd yaml-cpp-yaml-cpp-0.6.3
mkdir build
cd build
cmake ..
make
sudo make install
- valgrind (Ubuntu 20.04 only)
cd "$SOURCE_ROOT"
URL=https://sourceware.org/pub/valgrind/valgrind-3.19.0.tar.bz2
curl -sSL $URL | tar -xj || error "Valgrind 3.19.0"
cd valgrind-3.19.0
./configure CC=gcc CXX=g++
make
sudo make install
- jsoncpp
cd "$SOURCE_ROOT"
URL=https://github.com/open-source-parsers/jsoncpp/archive/refs/tags/1.9.5.tar.gz
curl -sSL $URL | tar xzf -
cd jsoncpp-1.9.5
mkdir -p build/release
cd build/release
cmake ../..
make -j 8
sudo make install
- lz4
cd "$SOURCE_ROOT"
URL=https://github.com/lz4/lz4/archive/refs/tags/v1.9.3.tar.gz
curl -sSL $URL | tar xzf - || error "lz4 1.9.3"
cd lz4-1.9.3
sudo make install
cd "$SOURCE_ROOT"
URL=https://github.com/llvm/llvm-project/archive/refs/tags/llvmorg-17.0.6.tar.gz
curl -sSL $URL | tar xzf - || error "Clang 17.0.6"
cd llvm-project-llvmorg-17.0.6
mkdir build
cd build
cmake -DLLVM_ENABLE_PROJECTS=clang -DCMAKE_BUILD_TYPE="Release" -G "Unix Makefiles" -DGCC_INSTALL_PREFIX="/opt/rh/gcc-toolset-12/root/" -DCMAKE_C_COMPILER=/bin/gcc -DCMAKE_CXX_COMPILER=/bin/g++ ../llvm
make clang -j8
clangbuild=${SOURCE_ROOT}/llvm-project-llvmorg-17.0.6/build
ln -s clang++ ${SOURCE_ROOT}/llvm-project-llvmorg-17.0.6/build/bin/clang++-17
export PATH=$clangbuild/bin:${SOURCE_ROOT}/llvm-project-llvmorg-17.0.6/llvm/utils/lit/tests/Inputs/lld-features:$PATH
export LD_LIBRARY_PATH=$clangbuild/lib:$LD_LIBRARY_PATH
export CC=clang-17
export CXX=clang++
- fmt
cd "$SOURCE_ROOT"
git clone https://github.com/fmtlib/fmt.git
cd fmt
git checkout 9.1.0
cmake -DFMT_TEST=OFF \
-DCMAKE_CXX_STANDARD=17 \
-DCMAKE_BUILD_TYPE=RelWithDebInfo \
-DCMAKE_CXX_VISIBILITY_PRESET=hidden \
-DCMAKE_POSITION_INDEPENDENT_CODE=ON .
make
sudo make install
- Zstd
cd "$SOURCE_ROOT"
URL=https://github.com/facebook/zstd/releases/download/v1.4.5/zstd-1.4.5.tar.gz
curl -sSL $URL | tar xzf - || error "zstd 1.4.5"
cd zstd-1.4.5
curl -sSL ${PATCH_URL}/zstd.diff | patch -p1 || error "zstd.diff"
cd lib
make
sudo make install
- Antlr
cd "$SOURCE_ROOT"
URL=https://github.com/antlr/antlr3/archive/refs/tags/3.5.2.tar.gz
curl -sSL $URL | tar xzf -
cd antlr3-3.5.2
curl -sSL ${PATCH_URL}/antlr3.diff | patch -p1 || error "antlr3.diff"
sudo cp runtime/Cpp/include/antlr3* ${PREFIX}/include/
cd antlr-complete
MAVEN_OPTS="-Xmx4G" mvn
echo 'java -cp '"$(pwd)"'/target/antlr-complete-3.5.2.jar org.antlr.Tool $@' | sudo tee ${PREFIX}/bin/antlr3
sudo chmod +x ${PREFIX}/bin/antlr3
- Wabt
cd "$SOURCE_ROOT"
git clone --recursive https://github.com/WebAssembly/wabt
cd wabt
git submodule update --init
mkdir build
cd build
cmake ..
cmake --build .
export PATH=$PATH:${SOURCE_ROOT}/wabt/build
- xxHash
cd "$SOURCE_ROOT"
URL=https://github.com/Cyan4973/xxHash/archive/v0.8.0.tar.gz
curl -sSL $URL | tar xzf - || error "xxHash 0.8.0"
cd xxHash-0.8.0
sudo make install
#Fix for missing shared objects files (Only for RHEL 8.x)
sudo ln -s ${SOURCE_ROOT}/xxHash-0.8.0/libxxhash.so.0 /usr/lib64/libxxhash.so.0
- Boost
TOOLSET=gcc
CENV=(PATH=$PATH LD_LIBRARY_PATH=$LD_LIBRARY_PATH LD_RUN_PATH=$LD_RUN_PATH CC=$CC CXX=$CXX)
cd "$SOURCE_ROOT"
URL=https://sourceforge.net/projects/boost/files/boost/1.74.0/boost_1_74_0.tar.gz
curl -sSL $URL | tar xzf -
cd boost_1_74_0
sed -i 's/array\.hpp/array_wrapper.hpp/g' boost/numeric/ublas/matrix.hpp
sed -i 's/array\.hpp/array_wrapper.hpp/g' boost/numeric/ublas/storage.hpp
./bootstrap.sh
options=( toolset=$TOOLSET variant=release link=shared runtime-link=shared threading=multi --without-python )
./b2 ${options[@]} stage
sudo ${CENV[@]} ./b2 ${options[@]} install
#Fix for missing shared objects files (Only for RHEL 8.x)
sudo ln -s ${SOURCE_ROOT}/boost_1_74_0/stage/lib/libboost_program_options.so.1.74.0 /usr/lib64/libboost_program_options.so.1.74.0
sudo ln -s ${SOURCE_ROOT}/boost_1_74_0/stage/lib/libboost_thread.so.1.74.0 /usr/lib64/libboost_thread.so.1.74.0
sudo ln -s ${SOURCE_ROOT}/boost_1_74_0/stage/lib/libboost_date_time.so.1.74.0 /usr/lib64/libboost_date_time.so.1.74.0
sudo ln -s ${SOURCE_ROOT}/boost_1_74_0/stage/lib/libboost_regex.so.1.74.0 /usr/lib64/libboost_regex.so.1.74.0
sudo ln -s ${SOURCE_ROOT}/boost_1_74_0/stage/lib/libboost_system.so.1.74.0 /usr/lib64/libboost_system.so.1.74.0
sudo ln -s ${SOURCE_ROOT}/boost_1_74_0/stage/lib/libboost_unit_test_framework.so.1.74.0 /usr/lib64/libboost_unit_test_framework.so.1.74.0
- Thrift
cd "$SOURCE_ROOT"
URL=http://archive.apache.org/dist/thrift/0.13.0/thrift-0.13.0.tar.gz
curl -sSL $URL | tar xzf -
cd thrift-0.13.0
curl -sSL ${PATCH_URL}/thrift.diff | patch -p1 || echo "Error"
./configure --without-java --without-lua --without-go --disable-tests --disable-tutorial
make -j 8
sudo make install
#Fix for missing shared objects files (Only for RHEL 8.x)
sudo ln -s ${SOURCE_ROOT}/thrift-0.13.0/lib/cpp/.libs/libthrift-0.13.0.so /usr/lib64/libthrift-0.13.0.so
- Abseil (RHEL 8.x and Ubuntu 20.04 only)
cd "$SOURCE_ROOT"
git clone https://github.com/abseil/abseil-cpp.git
cd abseil-cpp
git checkout 20230125.3
cmake -Bbuild -H. -DCMAKE_INSTALL_PREFIX=/usr/local
sudo -E env "PATH=$PATH" cmake --build build --target install
cd "$SOURCE_ROOT"
sudo yum install -y xz
URL=https://gmplib.org/download/gmp/gmp-6.2.1.tar.xz
curl -sSL $URL | tar xJf - || error "gmplib 6.2.1"
cd gmp-6.2.1
./configure --prefix=${PREFIX} --with-pic
make
sudo make install
sudo mkdir -p /builddir/build/BUILD/gnutls-3.8.3/bundled_gmp/.libs/
sudo cp ${SOURCE_ROOT}/gmp-6.2.1/.libs/libgmp.a /builddir/build/BUILD/gnutls-3.8.3/bundled_gmp/.libs/
sudo locale-gen en_US.UTF-8
export LC_ALL=C
unset LANGUAGE
The following commands will clone the repository, initialize its submodules and apply the patches.
cd "$SOURCE_ROOT"
git clone https://github.com/scylladb/scylla.git
cd scylla
git checkout scylla-5.4.6
git submodule update --init --force --recursive
curl -sSL ${PATCH_URL}/seastar.diff | patch -d seastar -p1 || echo "Error"
curl -sSL ${PATCH_URL}/scylla_rh8.diff | patch -p1 || echo "Error" #For RHEL 8.x ditriburions
curl -sSL ${PATCH_URL}/scylla_rh9.diff | patch -p1 || echo "Error" #For RHEL 9.x ditriburions
curl -sSL ${PATCH_URL}/scylla_ub.diff | patch -p1 || echo "Error" #For Ubuntu ditriburions
find . -type f -exec sed -i 's,wasm32-wasi,wasm32-wasip1,g' {} +
The following commands will build a release binary that is optimized for, and will only run, on the specified TARGET:
export PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig:$PREFIX/lib64/pkgconfig${PKG_CONFIG_PATH:+:${PKG_CONFIG_PATH}}
export PATH=$PATH:~/.local/bin/:~/.cargo/bin
cflags="-I${PREFIX}/include -I${PREFIX}/include/boost"
./configure.py --mode="release" --target="${TARGET}" --debuginfo=1 \
--cflags="${cflags}" --compiler="${CXX}" --c-compiler="${CC}"
ninja build -j $(nproc)
Increase the request capacity in /proc/sys/fs/aio-max-nr for setting up Async I/O (only on Ubuntu 20.04 and Ubuntu 22.04):
echo "fs.aio-max-nr = 1048576" |& sudo tee /etc/sysctl.conf
sudo sysctl -p
Once ScyllaDB has been built you can run the tests. By default the tests will not contain debug information. To enable debug information add --tests-debuginfo 1
to the configure.py
parameters above. This will make the build significantly larger (10s of gigabytes).
cd "$SOURCE_ROOT/scylla"
./test.py --mode release
Note:
- Some testcases related to
topology
,cql-pytest
, andraft
components may fail in the first attempt but are successfull upon reruns. - As per this PR,
scylla-gdb
testcase is only supported onx86
architecture therefore its behaviour ons390x
is undefined. - Some testcases related to
s3
fails as they require AWS s3 socket connection
The performance tuning scripts that ship with ScyllaDB don't fully support LinuxONE at this stage. To maximize performance receive packet steering (RPS) should be enabled on network adapters.
The scylla
binary can be found at /<source_root>/scylla/build/release/scylla
.
$SOURCE_ROOT/scylla/build/release/scylla --help
For more information on how to run and test ScyllaDB see the links below.
The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.