Testing on Ares Cluster

When testing on Ares cluster, make sure to change the hostfile at hcl/test/hostfile. It should have two mpi processes on the first node, and two on the second, as in:

ares-comp-01:2
ares-comp-02:2

You can either run tests using ctest or mpirun. If you aren't familiar with the tests, ctest is preferred.

ctest -N lists the tests you can run.

ctest -V -R ares_unordered_map_test_MPI_4_2_500_1000_1_0 runs the unordered_map test.

ctest -V runs all tests available.

The command to run a test without ctest is:

LD_PRELOAD=`pwd`/libhcl.so mpirun -f test/hostfile -n 4
test/unordered_map_test 2 500 1000 1 0

The arguments are ranks_per_server, num_requests, size_of_request, server_on_node, and debug. The variable size_of_request should be equal to TEST_REQUEST_SIZE in include/hcl/common/constants.h, since we have not yet found a way to make dynamically configurable request sizes (due to serialization).

Configure

$ mkdir $HOME/hcl_build
$ cd $HOME/hcl_build
$ $HOME/software/install/bin/cmake \
-DCMAKE_BUILD_TYPE=Debug \
-DCMAKE_C_COMPILER=/opt/ohpc/pub/compiler/gcc/7.3.0/bin/gcc \
-DCMAKE_CXX_COMPILER=/opt/ohpc/pub/compiler/gcc/7.3.0/bin/g++ \
"-DCMAKE_CXX_FLAGS=-I${HOME}/software/install/include -L${HOME}/software/install/lib"\
-G "CodeBlocks - Unix Makefiles" $SRC_DIR

where $SRC_DIR = source directory (e.g. /tmp/tmp.bYNfITLGMr)

Compile

$ $HOME/software/install/bin/cmake --build ./ --target all -- -j 8

Run

$ cd $HOME/hcl_build/test
$ ctest -V

Patching Mercury 1.0.1 to work with RoCE

For HCL to work with Mercury 1.0.1 with RoCE, it needs a patched version of Mercury. The patch allows you to specify a domain to be used when connecting to a verbs interface (rather than looking up the domain), which is the only practical way to use RoCE. This patch will not work with other versions of Mercury, and it certainly won't work with previous versions, since it changes lookup syntax, and therefore assumes that lookup syntax is as in Mercury 1.0.1 to begin with (lookup syntax was actually changed in Mercury 1.0.1, so this certainly shouldn't work with older versions).

Assuming that Mercury 1.0.1 has been extracted to ./mercury-1.0.1 run our patch mercury-1.0.1-RoCE.patch in the RoCE_Patch folder as such

$ patch  -p0 <  mercury-1.0.1-RoCE.patch