Skip to content

Getting Started

Jump to bottom
Gustavo Marques edited this page Sep 20, 2021 · 7 revisions

Getting Started

For detailed instructions, see NOAA-GFDL wiki.

The steps of running MOM6 cases:

  1. Downloading NCAR/MOM6-cases
  2. Setting up the compiler environment
  3. Compiling cases
  4. Running cases

Downloading NCAR MOM6 cases

Clone NCAR/MOM6-cases git repository as follows:

git clone --recursive https://github.com/NCAR/MOM6-cases.git

Note that the option --recursive is to clone the submodules simultaneously. In the rest of this wiki, your local NCAR/MOM6 clone directory will be referred to as $MOM6-cases/

Setting up the compiler environment

Cheyenne & Intel compiler:

Load the modules as follows:

module load ncarenv
module load intel
module load netcdf
module load ncarcompilers
module load mpt/2.19

To automate this step, see section Cheyenne Compiler Environment

Hobart & Intel compiler:

Load the following module:

module load compiler/intel/default

CU Boulder - Summit

module load intel/17.4 impi netcdf

Other machines and compilers

see NOAA-GFDL wiki

Compiling cases

1) Cheyenne

Overview

NCAR/MOM6-cases currently depends on two external libraries:

  • FMS: a framework for constructing and executing cases
  • mkmf: a tool to construct case makefiles

These are saved under src/ directory along with NCAR/MOM6 submodule. (These two libraries will eventually be replaced by NCAR/MOM6- and CESM-specific packages.)

Work Flow

As described in NOAA-GFDL wiki, the common phases of compiling FMS (shared library) and MOM6 executable are:

  1. Using the list_path tool that comes with mkmf, create a path_names file that contains the relative path to the source files.

  2. Using mkmf script, create a makefile.

  3. Compile the shared library or MOM6 executable.

Compiling the FMS shared library

Before compiling MOM6, compile the FMS shared code.

First, generate the makefile for FMS. In the MOM6-cases/ directory:

cd $MOM6-cases/
mkdir -p build/intel/shared/repro/
cd build/intel/shared/repro/
rm -f path_names
../../../../src/mkmf/bin/list_paths ../../../../src/FMS
../../../../src/mkmf/bin/mkmf -t ../../../../src/mkmf/templates/cheyenne-intel.mk -p libfms.a -c "-Duse_libMPI -Duse_netCDF -DSPMD" path_names

Build FMS (libfms.a):

make NETCDF=3 REPRO=1 libfms.a -j

Compiling MOM6 in ocean-only mode:

With the FMS library compiled, generate the makefile for NCAR/MOM6:

cd $MOM6-cases/
mkdir -p build/intel/ocean_only/repro/
cd build/intel/ocean_only/repro/
rm -f path_names
../../../../src/mkmf/bin/list_paths -l ./ ../../../../src/MOM6/{config_src/infra/FMS1,config_src/memory/dynamic_symmetric,config_src/drivers/solo_driver,config_src/external,src/{*,*/*}}/ ; \
../../../../src/mkmf/bin/mkmf -t ../../../../src/mkmf/templates/cheyenne-intel.mk -o '-I../../shared/repro' -p MOM6 -l '-L../../shared/repro -lfms' -c '-Duse_libMPI -Duse_netCDF -DSPMD' path_names

Finally, compile NCAR/MOM6:

make NETCDF=3 REPRO=1 MOM6 -j

2) Hobart

Repeat the steps above but replace "cheyenne-intel.mk" to "hobart-mpi.mk"

3) CU Boulder - Summit

Repeat the steps above but replace "cheyenne-intel.mk" to "summit-intel.mk" or simply add the following to the hobart-mpi.mk around line 78

CFLAGS += $(shell nc-config --cflags)

Running cases

On Cheyenne:

  1. First, compile FMS and MOM6. (see above)
  2. Create a batch script to submit the job. 
    #!/bin/bash
### Job Name
#PBS -N double_gyre
#PBS -A [YOUR-PROJECT-CODE]
#PBS -l walltime=00:01:00
#PBS -q regular
#PBS -j oe
### Select 1 node with 36 CPUs 
#PBS -l select=1:ncpus=36:mpiprocs=36
### Send email on abort and end
#PBS -m ae
#PBS -M [YOUR-EMAIL-ADDRESS]

### Run the executable
mpiexec_mpt dplace -s 1 ../../build/intel/ocean_only/repro/MOM6
  3. Submit your job through PBS Pro workload management system. For example: 
    qsub script_name

On Hobart:

  1. First, compile FMS and MOM6. (see above)
  2. Create a batch script to submit the job. 
    #!/bin/tcsh
### Job Name
#PBS -N double_gyre
#PBS -l walltime=00:05:00
#PBS -q short
### Select 1 nodes with 32 CPUs 
#PBS -l nodes=1:ppn=32
### Send email on abort, begin and end
#PBS -m abe

### Load correct environment 
source ~/MOM6-cases/build/intel/env

### Run the executable
mpirun -np 32 ~/MOM6-cases/build/intel/ocean_only/repro/MOM6
  3. Submit your job through PBS Pro workload management system. For example: 
    qsub script_name

On CU Boulder - Summit

This process is similar to the two above.

  1. First, compile FMS and MOM6. (see above)
  2. Navigate to the folder of the example you want to run. Double gyre is small enough to run on a single core.
  3. Create a batch script to submit the job. 
    #!/bin/bash

#SBATCH --nodes=1
#SBATCH --ntasks=16
#SBATCH --time=00:10:00
#SBATCH --partition=shas-testing
#SBATCH --output=sample-%j.out

module purge

module load intel/17.4 impi netcdf
echo "== This is the scripting step! =="
### Run the executable
    mpirun -np 16 ~/path/to/MOM6-cases/build/intel/ocean_only/repro/MOM6
echo "== End of Job =="
  4. Submit your job 
    sbatch script_name
Clone this wiki locally