Skip to content

Building HPCC

Jump to bottom Edit New page
Michael Gardner edited this page Jul 12, 2024 · 141 revisions

Assuming you already have your git repository set up (see Git-step-by-step), you can build and install the HPCCSystems Platform and clienttools.

We force 'out-of-source' builds. The developers environment should have a build directory alongside the cloned HPCC-Platform git repository. Once vcpkg has generated it's build tools and libraries, we will have a vcpkg_installed directory alongside as well.

Prerequisites

The build process requires some additional packages. We now utilize vcpkg for library and major build tool dependencies.

All environments

NodeJS: We recommend the current LTS version (20.x.x at time of writing): see installing NodeJS for more information.

Ubuntu Distributions

Ubuntu 24.04:

### install build utilities
sudo apt-get update && sudo apt-get install -y --no-install-recommends apt-transport-https autoconf autoconf-archive automake autotools-dev binutils-dev bison build-essential ca-certificates curl dirmngr flex git gnupg groff-base libtool pkg-config software-properties-common tar unzip uuid-dev zip mono-complete cmake

Ubuntu 22.04:

### install build utilities
sudo apt-get update && sudo apt-get install -y --no-install-recommends apt-transport-https autoconf autoconf-archive automake autotools-dev binutils-dev bison build-essential ca-certificates curl dirmngr flex git gnupg groff-base libtool pkg-config software-properties-common tar unzip uuid-dev zip cmake
## install mono for vcpkg
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
sudo sh -c 'echo "deb https://download.mono-project.com/repo/ubuntu stable-focal main" > /etc/apt/sources.list.d/mono-official-stable.list'
sudo apt-get update && sudo apt-get install -y mono-complete

Ubuntu 20.04:

### install build utilities
sudo apt-get update && sudo apt-get install -y --no-install-recommends apt-transport-https autoconf autoconf-archive automake autotools-dev binutils-dev bison build-essential ca-certificates curl dirmngr flex git gnupg groff-base libtool pkg-config software-properties-common tar unzip uuid-dev zip
### install mono for vcpkg
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF
sudo sh -c 'echo "deb https://download.mono-project.com/repo/ubuntu stable-focal main" > /etc/apt/sources.list.d/mono-official-stable.list'
sudo apt-get update && sudo apt-get install -y mono-complete

CentOS Builds

CentOS 7 (EOL on June 30, 2024):

### install build utilities
sudo yum update -y
sudo yum install -y centos-release-scl https://packages.endpointdev.com/rhel/7/os/x86_64/endpoint-repo.x86_64.rpm
sudo yum group install -y "Development Tools"
sudo yum install -y autoconf autoconf-archive automake curl git kernel-devel libtool perl-IPC-Cmd python3 tar unzip yum-utils zip
sudo yum -y clean all
sudo yum install -y devtoolset-11
### set devtoolset-11 for future shell environments
echo "source /opt/rh/devtoolset-11/enable" | sudo tee -a /etc/bashrc
source /opt/rh/devtoolset-11/enable

cd ~

### pkg-config
curl -o pkg-config-0.29.2.tar.gz https://pkg-config.freedesktop.org/releases/pkg-config-0.29.2.tar.gz && tar xvfz pkg-config-0.29.2.tar.gz
pushd pkg-config-0.29.2
./configure --prefix=/usr/local/pkg_config/0_29_2 --with-internal-glib
make -j$(nproc)
sudo make install
sudo ln -s /usr/local/pkg_config/0_29_2/bin/pkg-config /usr/local/bin/
sudo mkdir /usr/local/share/aclocal
sudo ln -s /usr/local/pkg_config/0_29_2/share/aclocal/pkg.m4 /usr/local/share/aclocal/
popd

### autoconf
curl -o autoconf-2.71.tar.gz http://ftp.gnu.org/gnu/autoconf/autoconf-2.71.tar.gz && gunzip autoconf-2.71.tar.gz && tar xvf autoconf-2.71.tar && pushd autoconf-2.71 && ./configure && make -j$(nproc) && sudo make install

### autoconf-archive
curl -o autoconf-archive-2021.02.19.tar.xz http://ftp.gnu.org/gnu/autoconf-archive/autoconf-archive-2021.02.19.tar.xz && xz -d -v autoconf-archive-2021.02.19.tar.xz && tar xvf autoconf-archive-2021.02.19.tar && pushd autoconf-archive-2021.02.19 && ./configure && make -j$(nproc) && sudo make install && popd

### automake
curl -o automake-1.16.5.tar.gz http://ftp.gnu.org/gnu/automake/automake-1.16.5.tar.gz && tar xvzf automake-1.16.5.tar.gz && pushd automake-1.16.5 && ./configure && make -j$(nproc) && sudo make install && popd

### libtool
curl -o libtool-2.4.6.tar.gz http://ftp.jaist.ac.jp/pub/GNU/libtool/libtool-2.4.6.tar.gz && tar xvfz libtool-2.4.6.tar.gz && pushd libtool-2.4.6 && ./configure --prefix=/usr/local/libtool/2_4_6 && make -j$(nproc) && sudo make install && popd

### symlinks
sudo ln -s /usr/local/libtool/2_4_6/bin/libtool /usr/local/bin/ && sudo ln -s /usr/local/libtool/2_4_6/bin/libtoolize /usr/local/bin/ && sudo ln -s /usr/local/libtool/2_4_6/include/libltdl /usr/local/include/ && sudo ln -s /usr/local/libtool/2_4_6/include/ltdl.h /usr/local/include/ && sudo ln -s /usr/local/libtool/2_4_6/lib/libltdl.a /usr/local/lib/ && sudo ln -s /usr/local/libtool/2_4_6/lib/libltdl.la /usr/local/lib/ && sudo ln -s /usr/local/libtool/2_4_6/lib/libltdl.so /usr/local/lib/ && sudo ln -s /usr/local/libtool/2_4_6/lib/libltdl.so.7 /usr/local/lib/ && sudo ln -s /usr/local/libtool/2_4_6/lib/libltdl.so.7.3.1 /usr/local/lib/ && sudo ln -s /usr/local/libtool/2_4_6/share/aclocal/libtool.m4 /usr/local/share/aclocal/ && sudo ln -s /usr/local/libtool/2_4_6/share/aclocal/ltargz.m4 /usr/local/share/aclocal/ && sudo ln -s /usr/local/libtool/2_4_6/share/aclocal/ltdl.m4 /usr/local/share/aclocal/ && sudo ln -s /usr/local/libtool/2_4_6/share/aclocal/lt~obsolete.m4 /usr/local/share/aclocal/ && sudo ln -s /usr/local/libtool/2_4_6/share/aclocal/ltoptions.m4 /usr/local/share/aclocal/ && sudo ln -s /usr/local/libtool/2_4_6/share/aclocal/ltsugar.m4 /usr/local/share/aclocal/ && sudo ln -s /usr/local/libtool/2_4_6/share/aclocal/ltversion.m4 /usr/local/share/aclocal/

### fetch mono
sudo yum-config-manager --add-repo http://download.mono-project.com/repo/centos/ && sudo yum clean all && sudo yum makecache && sudo rpm --import "http://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF" && sudo yum install -y mono-complete && sudo yum -y clean all

### extras with epel-release
sudo yum makecache && sudo yum install -y wget epel-release java-11-openjdk-devel python3-devel wget && sudo yum update -y && sudo yum install -y ccache R-core-devel && sudo yum -y clean all

### R dependencies
export Rcpp_package=Rcpp_0.12.19.tar.gz && export RInside_package=RInside_0.2.12.tar.gz && wget https://cran.r-project.org/src/contrib/Archive/Rcpp/${Rcpp_package} && wget https://cran.r-project.org/src/contrib/Archive/RInside/${RInside_package} && sudo R CMD INSTALL ${Rcpp_package} ${RInside_package} && rm -f ${Rcpp_package} ${RInside_package}

Rockylinux 8:

### install build utilities
sudo dnf install -y 'dnf-command(config-manager)' && sudo dnf config-manager --set-enabled powertools && sudo dnf update -y && sudo dnf install -y --allowerasing autoconf autoconf-archive automake bison curl flex git libtool perl-IPC-Cmd python3 tar unzip zip && sudo dnf clean all

sudo dnf install -y gcc-toolset-12-gcc gcc-toolset-12-gcc-c++ gcc-toolset-12-binutils gcc-toolset-12-annobin-annocheck gcc-toolset-12-annobin-plugin-gcc

echo "source /opt/rh/gcc-toolset-12/enable" | sudo tee -a /etc/bashrc
source /opt/rh/gcc-toolset-12/enable

### install mono for vcpkg
sudo rpmkeys --import "http://keyserver.ubuntu.com/pks/lookup?op=get&search=0x3FA7E0328081BFF6A14DA29AA6A19B38D3D831EF" && sudo dnf config-manager --add-repo http://download.mono-project.com/repo/centos8-stable/ && sudo dnf clean all && sudo dnf makecache && sudo dnf install -y mono-complete

sudo yum remove -y python3.11 java-1.* && yum install -y java-11-openjdk-devel python3-devel epel-release && sudo yum update -y && sudo yum install -y ccache R-core-devel R-Rcpp-devel R-RInside-devel && sudo yum -y clean all

Setup for Clienttools Only builds

On OSX (macos-12):

brew install bison flex pkg-config autoconf autoconf-archive automake libtool cmake openjdk@11

On Windows:

placeholder

Get Latest Sources

(See Git-step-by-step for full instructions)

git clone [-b <branch name>] --recursive https://github.com/hpcc-systems/HPCC-Platform.git

Where [ ] denotes an optional argument.

Build vcpkg libraries and tools

Within the freshly fetched HPCC-Platform repository, there is submodule of vcpkg. Using this submodule we will create our build libraries for linking, and build tools.

In this example we will be working as a non-root user, under our home directory. We will have a HPCC-Platform repository cloned with the appropriate branch that we want to build checked out, and have run a git submodule update --init --recursive to ensure that the vcpkg submodule within that branch is set to the correct HEAD.

The following commands will create a ~/vcpkg_installed directory containing libraries for linking in our final build. They will be referenced later when invoking cmake.

### setup environment variables
export VCPKG_BINARY_SOURCES="clear;nuget,Github,readwrite"
export VCPKG_NUGET_REPOSITORY=https://github.com/hpcc-systems/vcpkg

pushd ~/HPCC-Platform/vcpkg
./bootstrap-vcpkg.sh
mono `./vcpkg fetch nuget | tail -n 1` sources add -name "Github" -source "https://nuget.pkg.github.com/hpcc-systems/index.json"
./vcpkg install --x-abi-tools-use-exact-versions --x-install-root=~/vcpkg_installed --triplet=x64-linux-dynamic

To generate build tools such as cmake, ninja and node:

pushd ~/HPCC-Platform/vcpkg
mkdir -p ~/tools/{cmake,ninja,node}
cp -r $(dirname $(dirname `./vcpkg fetch cmake | tail -n 1`))/* ~/tools/cmake
cp -r $(dirname $(dirname `./vcpkg fetch ninja | tail -n 1`))/* ~/tools/ninja
cp -r $(dirname $(dirname `./vcpkg fetch node | tail -n 1`))/* ~/tools/node

### manual install on system
sudo cp -rs ~/tools/cmake/bin /usr/local/ && sudo cp -rs ~/tools/cmake/share /usr/local/ && sudo ln -s ~/tools/ninja/ninja /usr/local/bin/ninja && sudo cp -rs ~/tools/node/bin /usr/local/ && sudo cp -rs ~/tools/node/include /usr/local/ && sudo cp -rs ~/tools/node/lib /usr/local/ && sudo cp -rs ~/tools/node/share /usr/local/

Build from source

An out of source build is required. Setup a build directory for the generated build files and artifacts.

All named variables are preceeded by -D. So for example, -DCMAKE_BUILD_TYPE=Debug -S is the source code directory -B build location VCPKG_FILES_DIR is the directory vcpkg_installed is located in. In this tutorial it is assumed to be the home directory CMAKE_BUILD_TYPE by default is RelWithDebInfo. Options are Release, RelWithDebInfo, Debug VCPKG_TARGET_TRIPLET is the library triplet for your os. Default is x64-linux-dynamic. Centos7 is centos-7-dynamic. AmazonLinux is x64-amazonlinux-dynamic. CONTAINERIZED uses different logic for startup, with optimizations for running in cloud environments. Default is OFF CPACK_STRIP_FILES can strip excess symbols from files for packaging. Default is OFF

An example invocation of cmake

mkdir -p ~/build
cmake -S ~/HPCC-Platform -B ~/build -DVCPKG_FILES_DIR=~ -DUSE_OPTIONAL=OFF -DCMAKE_BUILD_TYPE=RelWithDebInfo -DCONTAINERIZED=OFF -DCPACK_STRIP_FILES=OFF
#make all targets and output package, with all available threads
cmake --build ~/build --parallel $(nproc) --target package

plugins

To build an individual plugin package, set the -DPLUGIN= variable. To include all plugins in a platform build, set the -DINCLUDE_PLUGINS=ON variable. To include individual plugins in a platform build, set -DINCLUDE_=ON for each desired plugin. Example: -DINCLUDE_JAVAEMBED=ON.

These are the allowable names for plugin names;

  • CASSANDRAEMBED
  • COUCHBASEEMBED
  • ECLBLAS
  • H3
  • JAVAEMBED
  • KAFKA
  • MEMCACHED
  • MONGODBEMBED
  • MYSQLEMBED
  • NLP
  • PARQUETEMBED
  • REDIS
  • REMBED
  • SQLITE3EMBED
  • SQS
  • WASMEMBED

Clienttools

To build the client tools only, do:

cd ~/hpcc/build
cmake -DCLIENTTOOLS_ONLY=1 ../src

To enable signing of the ecl standard library, insure you have a gpg private key loaded into your gpg keychain and do:

cd ~/hpcc/build
# Add -DSIGN_MODULES_KEYID and -DSIGN_MODULES_PASSPHRASE if applicable
cmake -DSIGN_MODULES=ON ../src

In some cases, users have found that when packaging for ubuntu, the dpkg-shlibdeps portion of the packaging adds an exceptional amount of time to the build process. This package will likely not run properly if installed on another system. Only use locally. To turn this off (and to create a package without dynamic dependency generation) do:

cd ~/hpcc/build
cmake -DUSE_SHLIBDEPS=OFF ../src

CMake will check for necessary dependencies like binutils, boost regex, cppunit, pthreads, etc. If everything is correct, it'll create the necessary Makefiles and you're ready to build HPCC.

####NOTE: We default to using libxslt in place of Xalan for xslt support. Should you prefer to use libxalan, you can specify -DUSE_LIBXALAN on the cmake command line.

Building

It doesn't take too long to compile HPCC, but it's highly recommended that you use all your CPUs to do so. When building complex projects (where I/O plays an important part), it's often recommended to use 50% more threads than the number of CPUs you have.

So, on a quad-core machine, the command you should type is:

make -j6

Alternatively you can call a build system agnostic variant (works with make, ninja, XCode, Visual Studio etc.)

cmake --build .

That will make all binaries, libraries and scripts necessary to create the package.

Creating a package

The recommended way to install HPCC in your machine (even for testing) is to use distro packages. CMake has already detected your system, so it know whether to generate TGZ files, DEB or RPM packages.

Just type:

make package

Alternatively you can use the build system agnostic variant:

cmake --build . --target package

and it'll create the appropriate package for you to install. The package will be inside the build directory.

In my system (Ubuntu 11.04), it created a package called hpccsystems-platform-community_3.3.0-1trunk-Debugnatty_amd64.deb.

Installing the package

The HPCC package has dependencies built in, and your system might not have them all (you can have the source package in order to build, but not the library package installed by your package manager, for example).

Install the package:

sudo dpkg -i hpccsystems-platform-community_6.0.0-trunk0trusty_amd64.deb

(note that the name of the package you just built will depend on the branch you checked out, the distro, and other options).

Now fix any missing dependencies:

sudo apt-get -f install

(see here for Ubuntu based installation).

Starting the services

If your installation succeeded, you should see the service in /etc/init.d (or /etc/rc.d/init.d on RPM based distros) called hpcc-init. This service follows the standard Linux service interface, where you can use commands like:

sudo /etc/init.d/hpcc-init status
sudo /etc/init.d/hpcc-init start
sudo /etc/init.d/hpcc-init stop

Use hpcc-init start and make sure there are no error messages. If everything went ok, check that the cluster is up and running by visiting the admin page on your browser.

To create a development installation on Linux

Building a development installation for quick startup and debugging is very similar to the normal build process. The only major difference are the flags passed to the invocation of cmake to build your make files, and instead of doing a make package you run make install and make configure. There are two key cmake variables that must be set, and some optional variables depending on the environment you're building in.

DESTDIR=/home/user/path
RUNTIME_USER=<my_user>
RUNTIME_GROUP=<my_alternate_group> (optional)
HOME_DIR=/path/to/alternate/home/directory (optional)

Destdir is used for the new root installation path of your build. For instance if you set DESTDIR=/home/user/runtime, then everything will be laid down there as though it was root. Then in order to start the platform, you simply need to call the hpcc-init script directly from /home/user/runtime/etc/init.d/hpcc-init. It is important that you do not set the DESTDIR environment variable in your shell. Allow cmake to handle DESTDIR, not the shell environment.

Runtime User is generally your user name. This will allow you to run the entire platform as your user instead of as root. Running as root will mess up your runtime environment as certain files will be owned by root on creation instead of ${USER}. Never run as root when doing a development build.

Runtime group is optional and is for environments like OSx where your Runtime User and Runtime Group aren't both set to your user name.

The Home directory is a useful variable if you have the home path for your environment set to a non-standard location such as /mnt/disk1/home/my_user. It is important to remember that the DESTDIR should most likely match this path in most situations. If the HOME_DIR is standard, this will be done by default for you. Otherwise look to manipulate this variable.

Example:

cmake -DDESTDIR=$HOME/runtime -DRUNTIME_USER=$USER ../HPCC-Platform/
make install #instead of make package, will also include configure target

TODO

  • How to cross compile to other architectures
  • More Windows details

7 Series and Earlier builds

For older builds, please refer to this point in the documents history.

Add a custom footer
Add a custom sidebar
Clone this wiki locally