Skip to content

Building ScyllaDB

Jump to bottom
linuxonz edited this page Jan 29, 2025 · 39 revisions

Building ScyllaDB

The instructions provided below specify the steps to build ScyllaDB version 6.1.1 on Linux on IBM Z for the following distributions:

  • RHEL (8.8, 8.10, 9.2, 9.4, 9.5)
  • Ubuntu (20.04, 22.04, 24.04)

Ensure that the kernel you are using is the latest available.

General Notes:

  • ScyllaDB and its dependencies are large projects and take a significant amount of time to build. The build can be sped up on large systems by increasing the -j parameter passed to make and/or ninja. The build was tested on a machine with 8 vCPUs and 32GB/16GB RAM. When building on a machine with 16GB RAM or less, a large swap file is needed.
  • Building ScyllaDB takes a large amount of disk space. Ensure you have at least 75GB available on the filesystem you are using. Significantly more space will be necessary if you wish to add debug information to the test binaries.
  • A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.
  • An installation directory /<prefix>/ is referred to in these instructions, by default this is /usr/local.
  • A target architecture /<target>/ is referred to in these instructions - this is the value passed to the GCC compiler.

1. Build using script

If you want to build ScyllaDB manually, go to STEP 2.

Use the following commands to build ScyllaDB using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ScyllaDB/6.1.1/build_scylladb.sh

bash build_scylladb.sh [-t] [-y] [-z <target>]

Provide -t option for executing build with tests.
Provide -y to install required packages without confirmation. Provide -z option to select target architecture for build e.g. z13, z14, z15, z16. Defaults to native architecture.

If the build completes successfully, go to STEP 6. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

2. Install Dependencies

export SOURCE_ROOT=/<source_root>/
export TARGET=native                      # z13, z14, z15, z16, native
export PREFIX=/usr/local                  # /<prefix>/
export PATCH_URL=https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/ScyllaDB/6.1.1/patch/

2.1. Install Packages

  • RHEL

      sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm # For RHEL8
  sudo dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm # For RHEL9

sudo yum install -y --allowerasing libatomic libatomic_ops-devel \
  java-1.8.0-openjdk-devel mpfr-devel llvm-toolset-17.0.6 \
  lksctp-tools-devel xfsprogs-devel snappy-devel libyaml-devel \
  openssl-devel libevent-devel libtasn1-devel libmpcdec libidn2-devel numactl-devel \
  c-ares-devel gnutls-devel gnutls-c++ gnutls-dane perl-devel \
  make automake git maven ant ninja-build unzip bzip2 wget curl xz-devel texinfo libffi-devel \
  libpciaccess-devel libxml2-devel libtool diffutils libtool-ltdl-devel trousers-devel \
  p11-kit-devel libunistring-devel libicu-devel readline-devel lua-devel patch systemd-devel \
  valgrind-devel cmake hwloc hwloc-devel cryptopp cryptopp-devel lz4 lz4-devel jsoncpp \
  jsoncpp-devel protobuf protobuf-devel rapidjson-devel stow yaml-cpp yaml-cpp-devel langpacks-en \
  glibc-all-langpacks libdeflate libdeflate-devel file abseil-cpp-devel rapidxml-devel doxygen net-tools \
  python3.12 python3.12-devel python3.12-pip gcc-toolset-12 gcc-toolset-12-libatomic-devel iproute

  • Ubuntu 20.04

      sudo apt-get update >/dev/null
  sudo apt-get install -y software-properties-common
  sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
  sudo apt-get update >/dev/null
  sudo apt-get install -y openjdk-8-jdk libaio-dev systemtap-sdt-dev lksctp-tools xfsprogs \
    libyaml-dev openssl libevent-dev libmpfr-dev libmpcdec-dev libssl-dev libsystemd-dev \
    libsctp-dev libsnappy-dev libpciaccess-dev libxml2-dev xfslibs-dev libgnutls28-dev \
    libiconv-hook-dev liblzma-dev libbz2-dev libxslt-dev libc-ares-dev libprotobuf-dev \
    protobuf-compiler libtool perl ant libffi-dev rapidjson-dev automake \
    make git maven ninja-build unzip bzip2 wget curl xz-utils texinfo diffutils liblua5.3-dev \
    libnuma-dev libunistring-dev pigz ragel stow patch locales \
    libudev-dev libdeflate-dev zlib1g-dev doxygen librapidxml-dev \
    libzstd-dev libxxhash-dev \
    liblz4-dev libhwloc-dev libyaml-cpp-dev antlr3 net-tools

  • Ubuntu (22.04, 24.04)

      sudo apt-get update >/dev/null
  sudo apt-get install -y software-properties-common
  sudo add-apt-repository -y ppa:ubuntu-toolchain-r/test
  sudo apt-get update >/dev/null
  sudo apt-get install -y openjdk-8-jdk libaio-dev systemtap-sdt-dev lksctp-tools xfsprogs \
    libyaml-dev openssl libevent-dev libmpfr-dev libmpcdec-dev libssl-dev libsystemd-dev \
    libsctp-dev libsnappy-dev libpciaccess-dev libxml2-dev xfslibs-dev libgnutls28-dev \
    libiconv-hook-dev liblzma-dev libbz2-dev libxslt-dev libc-ares-dev libprotobuf-dev \
    protobuf-compiler libcrypto++-dev libtool perl ant libffi-dev rapidjson-dev automake \
    make git maven ninja-build unzip bzip2 wget curl xz-utils texinfo diffutils liblua5.3-dev \
    libnuma-dev libunistring-dev pigz ragel stow patch locales valgrind \
    libudev-dev libdeflate-dev zlib1g-dev doxygen librapidxml-dev \
    libjsoncpp-dev libzstd-dev libxxhash-dev \
    liblz4-dev cmake libhwloc-dev libyaml-cpp-dev wabt antlr3 net-tools

2.2. Install GCC

  • Ubuntu 20.04

      #Install and build GCC 12.3.0 from source
  cd "$SOURCE_ROOT"
  sudo apt install -y gcc g++
  URL=https://ftp.gnu.org/gnu/gcc/gcc-12.3.0/gcc-12.3.0.tar.gz
  curl -sSL $URL | tar xzf - || error "GCC 12.3.0"
  cd gcc-12.3.0
  ./contrib/download_prerequisites
  mkdir objdir
  cd objdir

  ../configure --enable-languages=c,c++ --prefix=${PREFIX} \
    --enable-shared --enable-threads=posix \
    --disable-multilib --disable-libmpx \
    --with-system-zlib --with-long-double-128 --with-arch=zEC12 \
    --disable-libphobos --disable-werror \
    --build=s390x-linux-gnu --host=s390x-linux-gnu --target=s390x-linux-gnu

  make -j$(nproc) bootstrap
  sudo make install

  sudo update-alternatives --install /usr/bin/cc cc ${PREFIX}/bin/gcc 40
  sudo sed -i "1s,^,${PREFIX}/lib64\n," /etc/ld.so.conf.d/s390x-linux-gnu.conf
  sudo ldconfig

  sudo ln -s ${PREFIX}/lib/gcc/s390x-linux-gnu/12.3.0 /usr/lib/gcc/s390x-linux-gnu/12.3.0

  • Ubuntu 22.04

      sudo apt-get install -y gcc-12 g++-12
  sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-12 12
  sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 12
  sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-12 12
  sudo update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++-12 12

  • Ubuntu 24.04

      sudo apt install -y gcc-12 g++-12 libboost-all-dev
  sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-12 12
  sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-12 12
  sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-12 12
  sudo update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++-12 12
  export EXTRA_CFLAGS="--gcc-install-dir=/usr/bin/../lib/gcc/s390x-linux-gnu/12 "

  • RHEL 8.x/9.x

      sudo alternatives --install /usr/bin/ld ld /opt/rh/gcc-toolset-12/root/usr/bin/ld.bfd 100
  sudo ln -s /opt/rh/gcc-toolset-12/root/usr/bin/ld.bfd ${PREFIX}/bin
  sudo ln -s /opt/rh/gcc-toolset-12/root/usr/lib/gcc/s390x-redhat-linux/12/libatomic.so ${PREFIX}/lib/libatomic.so
  sudo ln -s /opt/rh/gcc-toolset-12/root/usr/lib/gcc/s390x-redhat-linux/12/libatomic.a ${PREFIX}/lib/libatomic.a
  sudo alternatives --install ${PREFIX}/bin/cc cc /opt/rh/gcc-toolset-12/root/usr/bin/gcc 30
  sudo alternatives --install ${PREFIX}/bin/gcc gcc /opt/rh/gcc-toolset-12/root/usr/bin/gcc 30
  sudo alternatives --install ${PREFIX}/bin/g++ g++ /opt/rh/gcc-toolset-12/root/usr/bin/g++ 30
  export EXTRA_CFLAGS="--gcc-install-dir=/opt/rh/gcc-toolset-12/root/usr/lib/gcc/s390x-redhat-linux/12 "
  export EXTRA_LDFLAGS="-B${PREFIX}/bin "

2.3. Install rust and rust packages

  cd "$SOURCE_ROOT"
  curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh /dev/stdin -y
  export PATH="$HOME/.cargo/bin:$PATH"
  cargo install cxxbridge-cmd
  cargo install wasm-opt
  rustup target add wasm32-wasip1

  #Only for RHEL
  export PATH=$PATH:$HOME/.rustup/toolchains/stable-s390x-unknown-linux-gnu/lib/rustlib/s390x-unknown-linux-gnu/bin/gcc-ld

2.4. Install Python 3.12

2.4.1. Install Python 3.12

  • RHEL 9.x

      sudo ln -s /usr/bin/python3.12 /usr/local/bin/python3
  sudo ln -s /usr/bin/pip3.12 /usr/local/bin/pip3

  • Ubuntu 24.04

      sudo apt install -y python3 python3-pip python3-aiohttp python3-magic \
    python3-colorama python3-tabulate python3-boto3 python3-pytest python3-pytest-asyncio \
    python3-redis python3-unidiff python3-humanfriendly python3-jinja2 python3-geomet python3-treelib gdb
  pip3 install --user --break-system-packages scylla-driver traceback-with-variables scylla-api-client allure-pytest
  export PATH="$HOME/.local/bin:$PATH"

2.4.2. Build Python 3.12.4 with modules (Ubuntu 20.04 and 22.04)

  cd "$SOURCE_ROOT"
  URL="https://www.python.org/ftp/python/3.12.4/Python-3.12.4.tgz"
  curl -sSL $URL | tar xzf - || error "Python 3.12.4"
  cd Python-3.12.4
  ./configure --enable-optimizations
  make
  sudo make install
  pip3 install --user --upgrade pip
  pip3 install --user pyparsing colorama pyyaml boto3 requests pytest traceback-with-variables \
  scylla-driver scylla-api-client aiohttp==3.9.5 tabulate boto3 pytest-asyncio \
  geomet treelib allure-pytest unidiff humanfriendly redis

  export PATH="$HOME/.local/bin:$PATH"

2.4.3. Install Python modules (RHEL)

  pip3 install --user --upgrade pip
  pip3 install --user pyparsing colorama pyyaml boto3 requests pytest traceback-with-variables \
    scylla-driver scylla-api-client aiohttp==3.9.5 tabulate boto3 pytest-asyncio \
    geomet treelib allure-pytest unidiff humanfriendly redis

  export PATH="$HOME/.local/bin:$PATH"

2.5. Set Environment (Ubuntu)

  sudo locale-gen en_US.UTF-8
  export LC_ALL=C
  unset LANGUAGE

2.6. Build and Install dependencies for required packages

2.6.1. RHEL

  • Antlr3
  cd "$SOURCE_ROOT"
  URL=https://github.com/antlr/antlr3/archive/3.5.2.tar.gz
  curl -sSL $URL | tar xzf - || error "antlr 3.5.2"
  cd antlr3-3.5.2
  curl -sSL ${PATCH_URL}/antlr3.diff | patch -p1 || error "antlr3.diff"
  sudo cp runtime/Cpp/include/antlr3* ${PREFIX}/include/
  cd antlr-complete
  MAVEN_OPTS="-Xmx4G" mvn
  echo 'java -cp '"$(pwd)"'/target/antlr-complete-3.5.2.jar org.antlr.Tool $@' | sudo tee ${PREFIX}/bin/antlr3
  sudo chmod +x ${PREFIX}/bin/antlr3
  • Ragel
  cd "$SOURCE_ROOT"
  URL=http://www.colm.net/files/ragel/ragel-6.10.tar.gz
  curl -sSL $URL | tar xzf -
  cd ragel-6.10
  ./configure
  make -j$(nproc)
  sudo make install
  • Zstd
  cd "$SOURCE_ROOT"
  URL=https://github.com/facebook/zstd/releases/download/v1.5.6/zstd-1.5.6.tar.gz
  curl -sSL $URL | tar xzf - || error "zstd 1.5.6"
  cd zstd-1.5.6
  cd lib
  make -j$(nproc)
  sudo make install
  • XxHash

Build XxHash for RHEL 8

  cd "$SOURCE_ROOT"
  URL=https://github.com/Cyan4973/xxHash/archive/v0.8.0.tar.gz
  curl -sSL $URL | tar xzf - || error "xxHash 0.8.0"
  cd xxHash-0.8.0
  sudo make install

Install XxHash package for RHEL 9

  sudo yum install -y xxhash-devel
  • Gmp(RHEL 9)
  cd "$SOURCE_ROOT"
  sudo yum install -y xz
  URL=https://gmplib.org/download/gmp/gmp-6.2.1.tar.xz
  curl -sSL $URL | tar xJf - || error "gmplib 6.2.1"
  cd gmp-6.2.1
  ./configure --prefix=${PREFIX} --with-pic
  make
  sudo make install

  sudo mkdir -p /builddir/build/BUILD/gnutls-3.8.3/bundled_gmp/.libs/
  sudo cp ${SOURCE_ROOT}/gmp-6.2.1/.libs/libgmp.a /builddir/build/BUILD/gnutls-3.8.3/bundled_gmp/.libs/

2.6.2. Ubuntu 20.04

  • CMake
  cd "$SOURCE_ROOT"
  URL=https://github.com/Kitware/CMake/releases/download/v3.27.9/cmake-3.27.9.tar.gz
  curl -sSL $URL | tar xzf - || error "cmake 3.27.9"
  cd cmake-3.27.9
  CC=/usr/bin/gcc CXX=/usr/bin/g++ ./bootstrap --parallel=$(nproc)
  make -j$(nproc)
  sudo make install
  cmake --version
  • Valgrind
  cd "$SOURCE_ROOT"
  URL=https://sourceware.org/pub/valgrind/valgrind-3.19.0.tar.bz2
  curl -sSL $URL | tar -xj || error "Valgrind 3.19.0"
  cd valgrind-3.19.0
  ./configure
  make -j$(nproc)
  sudo make install
  • Cryptopp
  cd "$SOURCE_ROOT"
  mkdir cryptopp
  cd cryptopp
  curl -ksSLO https://github.com/weidai11/cryptopp/archive/refs/tags/CRYPTOPP_8_2_0.zip
  unzip CRYPTOPP_8_2_0.zip
  cd cryptopp-CRYPTOPP_8_2_0
  CXXFLAGS="-std=c++11 -g -O2" make
  sudo make install
  • JsonCpp
  cd "$SOURCE_ROOT"
  URL=https://github.com/open-source-parsers/jsoncpp/archive/1.9.5.tar.gz
  curl -sSL $URL | tar xzf - || error "jsoncpp 1.9.5"
  cd jsoncpp-1.9.5
  mkdir -p build/release
  cd build/release
  cmake ../..
  make -j$(nproc)
  sudo make install

2.6.3. Dependencies common to distributions

  • Wabt (Ubuntu 20.04 and RHEL)
  cd "$SOURCE_ROOT"
  git clone -b 1.0.27 --recursive https://github.com/WebAssembly/wabt
  cd wabt
  git submodule update --init
  mkdir build
  cd build
  CC=gcc CXX=g++ cmake ..
  cmake --build .
  sudo make install
  • GDB (All distributions excluding Ubuntu 24.04)
  cd "$SOURCE_ROOT"
  wget "http://ftp.gnu.org/gnu/gdb/gdb-15.2.tar.gz"
  tar -xzf gdb-15.2.tar.gz
  cd gdb-15.2
  ./configure --with-python
  make -j$(nproc)
  sudo make install
  • Fmt
  cd "$SOURCE_ROOT"
  git clone https://github.com/fmtlib/fmt.git
  cd fmt
  git checkout 10.2.1
  cmake -DFMT_TEST=OFF \
    -DCMAKE_CXX_STANDARD=20 \
    -DCMAKE_POSITION_INDEPENDENT_CODE=ON \
	   -DFMT_DOC=OFF \
	   -DCMAKE_VISIBILITY_INLINES_HIDDEN=ON \
	   -DFMT_PEDANTIC=ON \
	   -DFMT_WERROR=ON \
	   -DBUILD_SHARED_LIBS=ON .
  make
  sudo make install
  • Antlr3 headers (Ubuntu)
  cd "$SOURCE_ROOT"
  git clone --depth 1 https://github.com/antlr/antlr3.git
  cd antlr3
  curl -sSL ${PATCH_URL}/antlr3.diff | patch -p1 || error "antlr3.diff"
  sudo cp -r runtime/Cpp/include/* ${PREFIX}/include/
  • Clang (Ubuntu)
  cd "$SOURCE_ROOT"
  wget https://apt.llvm.org/llvm.sh
  chmod +x llvm.sh
  sudo ./llvm.sh 17
  rm ./llvm.sh
  sudo ln -sf /usr/bin/clang-17 /usr/bin/clang
  sudo ln -sf /usr/bin/clang++-17 /usr/bin/clang++
  • Boost (All distributions excluding Ubuntu 24.04)
  cd "$SOURCE_ROOT"
  URL=https://archives.boost.io/release/1.83.0/source/boost_1_83_0.tar.gz
  curl -sSL $URL | tar xzf -
  cd boost_1_83_0
  ./bootstrap.sh
  sudo ./b2 variant=release link=shared runtime-link=shared threading=multi install

  # For RHEL only
  sudo bash -c "echo -e ''"$PREFIX"'/lib\n'"$PREFIX"'/lib64' > /etc/ld.so.conf.d/scylla.conf"

  sudo ldconfig
  • Minio
  sudo wget https://dl.minio.io/server/minio/release/linux-s390x/minio -P ${PREFIX}/bin
  sudo wget https://dl.min.io/client/mc/release/linux-s390x/mc -P ${PREFIX}/bin
  sudo chmod +x ${PREFIX}/bin/mc ${PREFIX}/bin/minio

3. Build ScyllaDB

3.1. Checkout and Patch

The following commands will clone the repository, initialize its submodules and apply the patches.

  cd "$SOURCE_ROOT"
  git clone -b scylla-6.1.1 https://github.com/scylladb/scylla.git --depth 1
  cd scylla
  git submodule update --init --force --recursive
  curl -sSL ${PATCH_URL}/scylladb.diff | patch -p1 || echo "ScyllaDB Patch Error"

3.2. Configure and compile ScyllaDB

The following commands will build a release binary that is optimized for, and will only run, on the specified TARGET:

  ./configure.py --mode="release" --target=${TARGET} \
    --cflags="${EXTRA_CFLAGS}-I${PREFIX}/include" --ldflags="${EXTRA_LDFLAGS}-L${PREFIX}/lib"

  ninja build -j$(nproc)

Increase the request capacity in /proc/sys/fs/aio-max-nr for setting up Async I/O (only on Ubuntu distributions):

echo "fs.aio-max-nr = 1048576" |& sudo tee /etc/sysctl.conf
sudo sysctl -p

4. Test ScyllaDB

Once ScyllaDB has been built you can run the tests. By default the tests will not contain debug information. To enable debug information add --tests-debuginfo 1 to the configure.py parameters above. This will make the build significantly larger (10s of gigabytes).

  cd "$SOURCE_ROOT/scylla"
  ./test.py --mode release

Note:

  • cql-pytest/test_wasm test case may fail in the first attempt but is successfull upon reruns.
  • cql/lwt_test and raft/fsm_test test case fails on RHEL 8 on both Intel and S390x systems.

5. Performance Tuning

The performance tuning scripts that ship with ScyllaDB don't fully support LinuxONE at this stage. To maximize performance receive packet steering (RPS) should be enabled on network adapters.

6. Run ScyllaDB

The scylla binary can be found at /<source_root>/scylla/build/release/scylla.

$SOURCE_ROOT/scylla/build/release/scylla --help

For more information on how to run and test ScyllaDB see the links below.

References

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally