Skip to content

Commit

Permalink
Merge pull request #11 from hpc-io/qiao_develop
Browse files Browse the repository at this point in the history 
Merge several bug fixes and tests
  • Loading branch information
@houjun
houjun authored Mar 11, 2021
2 parents 3e79a79 + 1a9edf0 commit 0e79d1b
Show file tree
Hide file tree
Showing 50 changed files with 5,215 additions and 833 deletions.
197 changes: 72 additions & 125 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,134 +1,81 @@
What is PDC?
============

Proactive Data Containers (PDC) is a Novel data abstraction for storing data in an
object-centric manner, where data objects are first-class citizens.

Visit https://sdm.lbl.gov/pdc for full details and publications discussing the
PDC concepts and designs. The remainder of this README discusses software dependencies
and installation.


Required libraries
======
1 libfabric
```sh
Download libfabric library from https://ofiwg.github.io/libfabric/
Compile and install following the instructions from https://github.com/ofiwg/libfabric
## Installation for PDC
This instruction is for installing PDC on Linux and Cray machines. Make sure you have GCC version at least 7 and MPI installed before you proceed.
For MPI, the recommended version is MPICH. Follow the procedures in https://www.mpich.org/static/downloads/3.4.1/mpich-3.4.1-installguide.pdf
PDC depends on libfabric and mercury. We are going to install libfabric, mercury, and PDC step by step.
Make sure you record the environmental variables (lines that contains the export commands). They are needed for running PDC and make the libraries again.
# Install libfabric
```

2 Mercury
```sh
git clone https://github.com/mercury-hpc/mercury && cd mercury
git submodule update --init
mkdir build
cd build
ccmake .. (where ".." is the relative path to the mercury-X directory)
0. wget https://github.com/ofiwg/libfabric/archive/v1.11.2.tar.gz
1. tar xvzf v1.11.2.tar.gz
2. cd libfabric-1.11.2
3. mkdir install
4. export LIBFABRIC_DIR=$(pwd)/install
5. ./autogen.sh
6. ./configure --prefix=$LIBFABRIC_DIR CC=gcc CFLAG="-O2"
7. make -j8
8. make install
9. export LD_LIBRARY_PATH="$LIBFABRIC_DIR/lib:$LD_LIBRARY_PATH"
10. export PATH="$LIBFABRIC_DIR/include:$LIBFABRIC_DIR/lib:$PATH"
```

Type 'c' multiple times and choose suitable options. Recommended options are:

BUILD_SHARED_LIBS ON (or OFF if the library you link
against requires static libraries)
BUILD_TESTING ON
CMAKE_INSTALL_PREFIX /path/to/install/directory
MERCURY_ENABLE_PARALLEL_TESTING OFF (ON requires running on HPC compute nodes)
MERCURY_USE_BOOST_PP OFF
MERCURY_USE_XDR OFF
MERCURY_USE_OPA OFF
NA_USE_BMI OFF
NA_USE_OFI ON
NA_USE_MPI OFF
NA_USE_CCI OFF
NA_USE_SM OFF

OFI_INCLUDE_DIR LIBFABRIC_PATH/include
OFI_LIBRARY LIBFABRIC_PATH/libbmi.so

CMAKE_C_FLAGS for both FLAGS, add -dynamic on NERSC machines
CMAKE_CXX_FLAGS if there you see errors: "/usr/bin/ld: attempted
static link of dynamic object `../bin/libmercury_hl.so' "


Setting include directory and library paths may require you to toggle to
the advanced mode by typing 't'. Once you are done and do not see any
errors, type 'g' to generate makefiles. Once you exit the CMake
configuration screen and are ready to build the targets, do:

make

To test Mercury is successfully built, run

make test

Look for Test #1: mercury_rpc_ofi_tcp, Test #2: mercury_bulk_ofi_tcp, etc.

3 BMI (Optional)
```sh
git clone git://git.mcs.anl.gov/bmi && cd bmi
# If you are building BMI on an OSX platform, then apply the following patch:
# patch -p1 < patches/bmi-osx.patch
./prepare && ./configure --enable-shared --enable-bmi-only
make && make install
# Install mercury
Make sure the ctest passes. Otherwise PDC will not work.
Step 2 is not required. It is a stable commit I am using when I write this instruction. You may skip it if you believe the current master branch of mercury works.
```

4 CCI (Optional)
```sh
git clone https://github.com/CCI/cci && cd cci
./autogen.pl
./configure (on Cori add --without-verbs for successful make)
make && make install
0. git clone https://github.com/mercury-hpc/mercury.git
1. cd mercury
2. git checkout e741051fbe6347087171f33119d57c48cb438438
3. git submodule update --init
4. export MERCURY_DIR=$(pwd)/install
5. mkdir install
6. cd install
7. cmake ../ -DCMAKE_INSTALL_PREFIX=$MERCURY_DIR -DCMAKE_C_COMPILER=gcc -DBUILD_SHARED_LIBS=ON -DBUILD_TESTING=ON -DNA_USE_OFI=ON -DNA_USE_SM=OFF
8. make
9. make install
10. ctest
11. export LD_LIBRARY_PATH="$MERCURY_DIR/lib:$LD_LIBRARY_PATH"
12. export PATH="$MERCURY_DIR/include:$MERCURY_DIR/lib:$PATH"
```


Building
====
git clone git@bitbucket.org:berkeleylab/pdc.git
cd pdc/src
mkdir build
cd build
ccmake .. (where ".." is the relative path to the PDC directory)

Similar to previous Mercury building process, type 'c' multiple times and choose
suitable options, and toggle to advanced mode by typing 't'. Recommended options are:

BUILD_SHARED_LIBS ON (or OFF if the library you link
against requires static libraries)
CFLAGS -dynamic (this is required on NERSC machines)
ENABLE_CHECKPOINT ON (or OFF if no metadata checkpoint is needed)
ENABLE_LUSTRE ON
ENABLE_MPI ON (or OFF if only used in serial mode)
ENABLE_MULTITHREAD OFF
ENABLE_TIMING OFF (or ON to enable PDC timer)


Once you are done and do not see any errors, type 'g' to generate makefiles.
Once you exit the CMake configuration screen and are ready to build the targets, do:

make


Testing
====
On NERSC machines (e.g. Cori), do the following:
----
* Job allocation (e.g. use 4 nodes)
```sh
salloc -C haswell -N 4 -t 01:00:00 -q interactive --gres=craynetwork:2
# Install PDC
You can replace mpicc to whatever MPI compilers you are using. For example, on Cori, you may need to use cc.
The ctest contains both sequential and MPI tests for our settings. These regression tests should work.
```
Run PDC create object test
----
* Set pdc temporary directory for server config file and checkpoint file (optional, if not set, the server and client will create and write/read under ./pdc_tmp)
```sh
export PDC_TMPDIR=/path/to/the/pdc/tmp/dir
0. git clone https://github.com/hpc-io/pdc.git
1. cd pdc
2. git checkout qiao_develop
3. cd src
4. mkdir install
5. cd install
6. export PDC_DIR=$(pwd)
7. cmake ../ -DBUILD_MPI_TESTING=ON -DBUILD_SHARED_LIBS=ON -DBUILD_TESTING=ON -DCMAKE_INSTALL_PREFIX=$PDC_DIR -DPDC_ENABLE_MPI=ON -DMERCURY_DIR=$MERCURY_DIR -DCMAKE_C_COMPILER=mpicc
8. make
9. make -j8
10. ctest
```

* Run 4 server processes, each on one node in background:
```sh
srun -N 4 -n 4 -c 2 --mem=25600 --cpu_bind=cores --gres=craynetwork:1 ./bin/pdc_server.exe &
# Environmental variables
During installation, we have set some environmental variables. These variables will disappear when you close the current shell.
I recommend adding the following lines to ~/.bashrc. (you can also manually execute them when you login).
The MERCURY_DIR and LIBFABRIC_DIR should be identical to the values you set during your installations of Mercury and Libfabric.
Remember, the install path is the path containing bin and lib directory, instead of the one containing source code.
```

* Run 64 client processes that concurrently create 1000 objects in total:
```sh
srun -N 4 -n 64 -c 2 --mem=25600 --cpu_bind=cores --gres=craynetwork:1 ./bin/create_obj_scale -r 1000
export PDC_DIR="where/you/installed/your/pdc"
export MERCURY_DIR="where/you/installed/your/mercury"
export LIBFABRIC_DIR="where/you/installed/your/libfabric"
export LD_LIBRARY_PATH="$LIBFABRIC_DIR/lib:$MERCURY_DIR/lib:$LD_LIBRARY_PATH"
export PATH="$LIBFABRIC_DIR/include:$LIBFABRIC_DIR/lib:$MERCURY_DIR/include:$MERCURY_DIR/lib:$PATH"
```
You can also manage the path with Spack, which is a lot more easier to load and unload these libraries.
## Running PDC
The ctest under PDC install folder runs PDC examples using PDC APIs.
PDC needs to run at least two applications. First, you need to start servers. Then, you can run client programs that send I/O request to servers as mercury RPCs.
For example, you can do the following. On Cori, you need to change the mpiexec argument to srun. On Theta, it is aprun. On Summit, it is jsrun.
```
cd $PDC_DIR/bin
./mpi_test.sh ./pdc_init mpiexec 2 4
```
This is test will start 2 processes for PDC servers. The client program ./pdc_init will start 4 processes. Similarly, you can run any of the client examples in ctest.
These source code will provide you some knowledge of how to use PDC. For more reference, you may check the documentation folder in this repository.
# PDC on Cori.
Installation on Cori is not very different from a regular linux machine. Simply replace all gcc/mpicc with the default cc compiler on Cori. Add options -DCMAKE_C_FLAGS="-dynamic" to the cmake line of PDC. Add -DCMAKE_C_FLAGS="-dynamic" -DCMAKE_CXX_FLAGS="-dynamic" at the end of the cmake line for mercury as well. Finally, "-DMPI_RUN_CMD=srun" is needed for ctest command later. Sometimes you may need to unload darshan before installation.


Loading

0 comments on commit 0e79d1b

Please sign in to comment.