Skip to content

Building Kibana

Jump to bottom
aborkar-ibm edited this page Jul 10, 2023 · 52 revisions

Building Kibana

The instructions provided below specify the steps to install Kibana 8.8.1 on Linux on IBM Z for following distributions:

  • RHEL (7.8, 7.9, 8.6, 8.8, 9.0, 9.2)
  • SLES (12 SP5, 15 SP4)
  • Ubuntu (20.04, 22.04)

General Notes:

  • When following the steps below please use a standard permission user unless otherwise specified.
  • A directory /<source_root>/ will be referred to in these instructions, this is a temporary writable directory anywhere you'd like to place it.

Build and Install Kibana

1) Build using script

If you want to build Kibana using manual steps, go to STEP 2.

Use the following commands to build Kibana using the build script. Please make sure you have wget installed.

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Kibana/8.8.1/build_kibana.sh

# Build Kibana
bash build_kibana.sh

If the build completes successfully, go to STEP 13. In case of error, check logs for more details or go to STEP 2 to follow manual build steps.

2) Install build dependencies

export SOURCE_ROOT=/<source_root>/

Please note EPEL is required on RHEL.

  • RHEL (7.8, 7.9)

    sudo yum install -y curl git devtoolset-7-gcc-c++ devtoolset-7-gcc gzip make python3 java-11-openjdk-devel unzip zip tar wget patch xz pkgconfig expat-devel glib2-devel
sudo yum install -y python3-bind9.16 bind9.16-chroot coreutils-single ed expect file gnupg2 iproute iproute-devel iputils lcov less openssl-devel redhat-lsb netcat python2-devel python3-devel python3-pip python3-requests python3-setuptools python3-six python3-wheel python3-pyyaml zlib-devel
scl enable devtoolset-7 bash

  • RHEL (8.6)

    sudo yum install -y curl git gcc-c++ gzip make python2 python3 java-11-openjdk-devel unzip zip tar wget patch xz pkg-config expat-devel glib2-devel
sudo yum install -y python3-bind9.16 bind9.16-chroot coreutils-single ed expect file gnupg2 iproute iproute-devel iputils lcov less openssl-devel redhat-lsb netcat python2-devel python3-devel python3-pip python3-requests python3-setuptools python3-six python3-wheel python3-pyyaml zlib-devel  
sudo ln -sf /usr/bin/python3 /usr/bin/python

  • RHEL (8.8)

    sudo yum install -y curl git gcc-c++ gzip make python2 python3 java-11-openjdk-devel unzip zip tar wget patch xz pkg-config expat-devel glib2-devel
sudo yum install -y python3-bind bind-chroot coreutils ed expect file gnupg2 iproute iproute-devel iputils lcov less openssl-devel redhat-lsb netcat python2-devel python3-devel python3-pip python3-requests python3-setuptools python3-six python3-wheel python3-pyyaml zlib-devel
sudo ln -sf /usr/bin/python3 /usr/bin/python

  • RHEL (9.0)

    sudo yum install -y curl git gcc-c++ gzip make python3 java-11-openjdk-devel unzip zip tar wget patch xz pkg-config expat-devel glib2-devel
sudo yum install -y python3-bind bind-chroot coreutils-single ed expect file gnupg2 iproute iproute-devel iputils lcov less openssl-devel netcat python3-devel python3-pip python3-requests python3-setuptools python3-six python3-wheel python3-pyyaml zlib-devel  
sudo ln -sf /usr/bin/python3 /usr/bin/python

  • RHEL (9.2)

    sudo yum install -y curl git gcc-c++ gzip make python3 java-11-openjdk-devel unzip zip tar wget patch xz pkg-config expat-devel glib2-devel
sudo yum install -y python3-bind bind-chroot coreutils ed expect file gnupg2 iproute iproute-devel iputils lcov less openssl-devel netcat python3-devel python3-pip python3-requests python3-setuptools python3-six python3-wheel python3-pyyaml zlib-devel
sudo ln -sf /usr/bin/python3 /usr/bin/python

  • SLES 12 SP5

    sudo zypper install -y curl libnghttp2-devel git gcc-c++ gcc11 gcc11-c++ gzip make python python3 java-11-openjdk-devel unzip zip tar wget patch xz which gawk pkg-config glib2-devel libexpat-devel
sudo zypper install -y python-bind bind-chrootenv coreutils curl ed expect file iproute2 iputils less libopenssl-devel netcat python2 python2-devel python3 python3-devel python3-pip python3-requests python3-setuptools python3-six python3-wheel unzip zlib-devel python3-PyYAML
sudo update-alternatives --install /usr/bin/cc cc /usr/bin/gcc-11 11
sudo update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-11 11
sudo update-alternatives --install /usr/bin/g++ g++ /usr/bin/g++-11 11
sudo update-alternatives --install /usr/bin/c++ c++ /usr/bin/g++-11 11
sudo ln -sf /usr/bin/gcc /usr/bin/s390x-linux-gnu-gcc
sudo ln -sf /usr/bin/cpp-11 /usr/bin/cpp

# Install Python 3.7 
wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Python3/3.7.4/build_python3.sh
bash build_python3.sh -y

  • SLES 15 SP4

    sudo zypper install -y curl git gcc-c++ gzip make python python3 java-11-openjdk-devel unzip zip tar wget patch xz which gawk pkg-config glib2-devel libexpat-devel
sudo zypper install -y python3-bind bind-chrootenv coreutils curl ed expect file iproute2  iputils lcov less libopenssl-devel netcat python2 python2-devel python3 python3-devel python3-pip python3-requests python3-setuptools python3-six python3-wheel unzip zlib-devel python3-python-gnupg python3-PyYAML

  • Ubuntu 20.04

sudo apt-get update
sudo apt-get install -y curl git g++ gzip make python python3 openjdk-11-jdk unzip zip tar wget patch xz-utils build-essential pkg-config libglib2.0-dev libexpat1-dev
  • Ubuntu 22.04
sudo apt-get update
sudo apt-get install -y curl git g++ gzip make python2 python3 openjdk-11-jdk unzip zip tar wget patch xz-utils build-essential pkg-config libglib2.0-dev libexpat1-dev

3) Set JAVA_HOME environment variable

export JAVA_HOME=<path to java>
export PATH=$JAVA_HOME/bin:$PATH

4) Install Bazel

  • RHEL and SLES
mkdir bazel && cd bazel
wget https://github.com/bazelbuild/bazel/releases/download/5.1.1/bazel-5.1.1-dist.zip
mkdir -p dist/bazel && cd dist/bazel
unzip ../../bazel-5.1.1-dist.zip
chmod -R +w .
curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/5.1.1/patch/dist-md5.patch | git apply
env EXTRA_BAZEL_ARGS="--tool_java_runtime_version=local_jdk" bash ./compile.sh
cd $SOURCE_ROOT/bazel
git clone https://github.com/bazelbuild/bazel.git
cd bazel
git checkout 5.1.1
curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/5.1.1/patch/bazel.patch | git apply
cd $SOURCE_ROOT/bazel/bazel
$SOURCE_ROOT/bazel/dist/bazel/output/bazel build -c opt --stamp --embed_label "5.1.1" //src:bazel //src:bazel_jdk_minimal //src:test_repos
mkdir output
cp bazel-bin/src/bazel output/bazel
./output/bazel build  -c opt --stamp --embed_label "5.1.1" //src:bazel //src:bazel_jdk_minimal //src:test_repos
export PATH=$PATH:$SOURCE_ROOT/bazel/output/
export USE_BAZEL_VERSION=$SOURCE_ROOT/bazel/output/bazel
  • Ubuntu
wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Bazel/5.1.1/build_bazel.sh
bash build_bazel.sh -y
export PATH=$PATH:$SOURCE_ROOT/bazel/output/
export USE_BAZEL_VERSION=$SOURCE_ROOT/bazel/output/bazel

5) Install Go v1.18.4

wget -q https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Go/1.18.4/build_go.sh
bash build_go.sh -y

6) Build Bazelisk binary

cd $SOURCE_ROOT
git clone https://github.com/bazelbuild/bazelisk.git
cd bazelisk
git checkout v1.12.1
curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Kibana/8.8.1/patch/bazelisk_patch.diff | git apply --ignore-whitespace
go build && ./bazelisk build --config=release //:bazelisk-linux-s390x

7) Install Node.js

cd $SOURCE_ROOT
sudo mkdir -p /usr/local/lib/nodejs
wget https://nodejs.org/dist/v16.19.1/node-v16.19.1-linux-s390x.tar.xz
sudo tar xf node-v16.19.1-linux-s390x.tar.xz -C /usr/local/lib/nodejs
export PATH=/usr/local/lib/nodejs/node-v16.19.1-linux-s390x/bin:$PATH

8) Install Yarn and patch Bazelisk

sudo chmod ugo+w -R /usr/local/lib/nodejs/node-v16.19.1-linux-s390x
npm install -g yarn @bazel/bazelisk@1.12.1
BAZELISK_DIR=/usr/local/lib/nodejs/node-v16.19.1-linux-s390x/lib/node_modules/@bazel/bazelisk
curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Kibana/8.8.1/patch/bazelisk.js.diff | patch $BAZELISK_DIR/bazelisk.js
cp $SOURCE_ROOT/bazelisk/bazel-out/s390x-opt-*/bin/bazelisk-linux_s390x $BAZELISK_DIR

9) Install libvips

cd $SOURCE_ROOT
wget https://github.com/libvips/libvips/releases/download/v8.13.2/vips-8.13.2.tar.gz
tar xf vips-8.13.2.tar.gz
cd vips-8.13.2/
./configure
make && sudo make install
mkdir $SOURCE_ROOT/libvips-8.13.2-linux-s390x && cd $SOURCE_ROOT/libvips-8.13.2-linux-s390x
wget https://github.com/lovell/sharp-libvips/releases/download/v8.13.2/libvips-8.13.2-linux-x64.tar.gz
tar -xzf libvips-8.13.2-linux-x64.tar.gz
rm -rf libvips-8.13.2-linux-x64.tar.gz
cp /usr/local/lib/libvips-cpp.so.42 $SOURCE_ROOT/libvips-8.13.2-linux-s390x/lib/
sed -i 's/linux-x64/linux-s390x/g' $SOURCE_ROOT/libvips-8.13.2-linux-s390x/platform.json
cd $SOURCE_ROOT
tar -czf libvips-8.13.2-linux-s390x.tar.br $SOURCE_ROOT/libvips-8.13.2-linux-s390x
npm config set sharp_libvips_local_prebuilds "$SOURCE_ROOT/libvips-8.13.2-linux-s390x.tar.br"

10) Build and Install Kibana

  • Download Kibana

    cd $SOURCE_ROOT
git clone https://github.com/elastic/kibana.git
cd kibana
git checkout v8.8.1

  • Build re2

    cd $SOURCE_ROOT
git clone https://github.com/uhop/node-re2.git
cd node-re2 && git checkout 1.17.4
git submodule update --init --recursive
npm install
gzip -c build/Release/re2.node > $SOURCE_ROOT/linux-s390x-93.gz
mkdir -p $SOURCE_ROOT/kibana/.native_modules/re2/
cp $SOURCE_ROOT/linux-s390x-93.gz $SOURCE_ROOT/kibana/.native_modules/re2/

  • Build chromedriver

    cd $SOURCE_ROOT
git clone https://github.com/giggio/node-chromedriver.git
cd node-chromedriver && git checkout 113.0.0
sed -i "s#process.arch === 'arm64' || process.arch === 'x64'#process.arch === 'arm64' || process.arch === 's390x' || process.arch === 'x64'#g" install.js
npm install

  • Apply source code patch:

    cd $SOURCE_ROOT/kibana
curl -sSL https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Kibana/8.8.1/patch/kibana_patch.diff | git apply

  • Bootstrap Kibana

    yarn kbn bootstrap

  • Build Kibana

    cd $SOURCE_ROOT/kibana
export NODE_OPTIONS="--max_old_space_size=4096"
yarn build --skip-os-packages

11) Test (Optional)

  • Unit tests

    cd $SOURCE_ROOT/kibana
wget https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Kibana/8.8.1/patch/unittest.sh
bash unittest.sh

  • Integration tests

    cd $SOURCE_ROOT/kibana
wget https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Kibana/8.8.1/patch/integrationtest.sh
bash integrationtest.sh

  • Functional tests

    export LD_LIBRARY_PATH=/usr/local/lib:$LD_LIBRARY_PATH
sudo ldconfig
  	
cd $SOURCE_ROOT/kibana
wget https://raw.githubusercontent.com/linux-on-ibm-z/scripts/master/Kibana/8.8.1/patch/functionaltest.sh
bash functionaltest.sh

Note:

  • The following test case failures can be ignored as they are seen on x86 also:

    packages/kbn-type-summarizer/src/tests/integration_tests/summarize_package.test.ts packages/kbn-type-summarizer/src/tests/integration_tests/ast_indexer.test.ts

  • The following test case failures are not related to functionality and can be safely ignored:

    src/cli_plugin/install/zip.test.js src/dev/build/lib/config.test.ts src/dev/build/tasks/nodejs/download_node_builds_task.test.ts src/dev/build/tasks/nodejs/extract_node_builds_task.test.ts src/dev/build/tasks/nodejs/verify_existing_node_builds_task.test.ts src/dev/build/lib/integration_tests/fs.test.ts src/dev/build/lib/integration_tests/scan_copy.test.ts

  • Some tests pass with below flags:

    TEST_BROWSER_HEADLESS=1 FORCE_COLOR=1 DISABLE_BOOTSTRAP_VALIDATION=true BROWSERSLIST_IGNORE_OLD_DATA=true

  • Some integration/functional tests fail as they download and start Intel specific ElasticSearch binary

  • Some X-Pack test cases will fail as X-Pack plugins are not supported on s390x

  • Some failing test cases pass when executed individually.

12) Install Kibana

cd $SOURCE_ROOT/kibana
sudo mkdir /usr/share/kibana/
sudo tar -xzf target/kibana-8.8.1-SNAPSHOT-linux-s390x.tar.gz -C /usr/share/kibana --strip-components 1
sudo ln -sf /usr/share/kibana/bin/* /usr/bin/

sudo groupadd [username]
sudo chown [username]:elastic -R /usr/share/kibana/

13) Start Kibana

Kibana requires an Elasticsearch instance to be running.

Set Kibana home directory export KIBANA_HOME=/usr/share/kibana/. You may start Kibana and also update the Kibana configuration file $KIBANA_HOME/config/kibana.yml.

kibana

Open your browser and go to https://<HOST_IP>:5601 to make sure the Kibana installation works.

Note:

  • When Kibana is run for the first time, it generates a unique link in your terminal which can be used to enroll your Kibana instance with Elasticsearch. See here for more information on starting Kibana.
  • In order to access Kibana server through web browser, you might need to change server.host field in $KIBANA_HOME/config/kibana.yml to 0.0.0.0. See here for more information.
  • For more information regarding configuring Kibana, please visit here.

References

The information provided in this article is accurate at the time of writing, but on-going development in the open-source projects involved may make the information incorrect or obsolete. Please open issue or contact us on IBM Z Community if you have any questions or feedback.

Clone this wiki locally