Skip to content

Building on Hera

Jump to bottom
Cory Martin edited this page Aug 18, 2021 · 2 revisions

Modules

The JCSDA maintains JEDI modules on Hera for compiling/running JEDI executables.

First, you must set the JEDI_OPT environment variable. This must be set both at compile and at runtime.

If using bash:

export JEDI_OPT=/scratch1/NCEPDEV/jcsda/jedipara/opt/modules

If using csh:

setenv JEDI_OPT /scratch1/NCEPDEV/jcsda/jedipara/opt/modules

Next, you need to add these JEDI modules for use

module use $JEDI_OPT/modulefiles/core

It is recommended that you run module purge before loading the JEDI environment modules, to prevent conflicts. To load the complete set of libraries needed to build and run JEDI executables, choose Intel or GNU options based on below: For the Intel compiler suite:

module load jedi/intel-impi/2020.2

and for the GNU compilers:

module load jedi/gnu-openmpi

Other Handy Environment Variables

To run the ctests properly, many require MPI. Hera admins will tar and feather you if you run a MPI job on a login node. But ctest can be smart and submit the test as a batch job, if the correct environment variables are set. For example:

export SLURM_ACCOUNT=da-cpu
export SALLOC_ACCOUNT=$SLURM_ACCOUNT
export SBATCH_ACCOUNT=$SLURM_ACCOUNT
export SLURM_QOS=debug

Please change this as appropriate if you are using csh, and ensure that SLURM_ACCOUNT is set to a HPC project that you have access to the queue for.

ecbuild commands

The ecbuild command that you use will depend on what you are attempting to build.

For the most basic case, you should always have this:

ecbuild -DMPIEXEC_EXECUTABLE=`which srun` -DMPIEXEC_NUMPROC_FLAG="-n" /path/to/bundle

The first option:

-DMPIEXEC_EXECUTABLE=`which srun`

tells ecbuild that for ctests that need MPI, how to submit it to the job queue using srun instead of mpirun. The second option:

-DMPIEXEC_NUMPROC_FLAG="-n"

is also needed to tell ecbuild how to specify the number of processors needed for each test.

For some other cases you may need to add extra options before the /path/to/bundle. These include:

  • To build the IODA Python API: -DBUILD_PYTHON_BINDINGS=ON
  • To build the IODA-Converters in the IODA-bundle: -DBUILD_IODA_CONVERTERS=ON

For example, to build the ioda-bundle with the python API and the ioda-converters:

ecbuild -DMPIEXEC_EXECUTABLE=`which srun` -DMPIEXEC_NUMPROC_FLAG="-n" -DBUILD_PYTHON_BINDINGS=ON -DBUILD_IODA_CONVERTERS=ON /path/to/bundle
Clone this wiki locally