Skip to content

Commit

Permalink
restructured WRF documentation for Maui to Mahuika migration (#357)
Browse files Browse the repository at this point in the history
  • Loading branch information
@Johnryder23
Johnryder23 authored Oct 4, 2024
2 parents 4493680 + 39b5543 commit b7bef25
Showing 1 changed file with 91 additions and 113 deletions.
204 changes: 91 additions & 113 deletions docs/Scientific_Computing/Supported_Applications/WRF.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,135 +16,59 @@ architecture supporting parallel computation and system extensibility.
The model serves a wide range of meteorological applications across
scales from tens of meters to thousands of kilometres.

Download WRF:

``` sh
cd /nesi/project/<your_project_code>
wget https://github.com/wrf-model/WRF/archive/v4.1.1.tar.gz
tar xf v4.1.1.tar.gz
```

This guide is based on WRF 4.1.1 and WPS 4.2 (it was also tested with
Polar WRF 4.1.1 but for that, you would need to apply the Polar WRF and
WPS patches before compiling).

WRF needs NetCDF, NetCDF-Fortran, PnetCDF and their dependencies to be
installed. On Māui, these are available as modules. On Mahuika, we
recommend to download these packages and build them by hand
(instructions are provided below).

This guide is based on WRF 4.6.0 and WPS 4.6.0
## WRF on Mahuika
### Building WRF on Mahuika

### Environment on Mahuika

We'll use the Intel compiler and Intel MPI library.

The following script will run through the complete install procedure of WRF on Mahuika. Run the script with `bash` *script\_name.sh*:
``` sh
module purge
module load HDF5/1.12.2-iimpi-2022a
```

Although NeSI has NetCDF modules installed, WRF wants the C and Fortran
NetCDF libraries, include files and modules all installed under the same
root directory. Hence we build those by hand.

### Building WRF dependencies on Mahuika

Copy-paste the commands below to build WRF on Mahuika:

``` sh
# create build directory for WRF's dependencies
export WRF_DEPS_DIR=$PWD/wrf-deps
mkdir $WRF_DEPS_DIR
cd $WRF_DEPS_DIR

# required for building and running
export LD_LIBRARY_PATH=$WRF_DEPS_DIR/lib:$LD_LIBRARY_PATH

# to access the nc-config anf nf-config commands
export PATH=$WRF_DEPS_DIR/bin:$PATH

# set the compilers
export CC=icc
export CXX=icc
export FC=ifort
export F77=ifort
export MPICC=mpiicc
export MPICXX=mpiicc
export MPIF77=mpiifort
export MPIF90=mpiifort

# now install the dependencies

# netcdf-c
wget https://downloads.unidata.ucar.edu/netcdf-c/4.9.2/netcdf-c-4.9.2.tar.gz
tar xf netcdf-c-4.9.2.tar.gz
cd netcdf-c-4.9.2/
./configure --prefix=$WRF_DEPS_DIR
make & make install
cd ..

# netcdf-fortran
wget https://downloads.unidata.ucar.edu/netcdf-fortran/4.6.1/netcdf-fortran-4.6.1.tar.gz
tar xf netcdf-fortran-4.6.1.tar.gz
cd netcdf-fortran-4.6.1
CPPFLAGS="$(../bin/nc-config --cflags)" LDFLAGS="$(../bin/nc-config --libs)" \
./configure --prefix=$WRF_DEPS_DIR
make & make install
cd ..

# pnetcdf
wget https://parallel-netcdf.github.io/Release/pnetcdf-1.12.3.tar.gz
tar xf pnetcdf-1.12.3.tar.gz
cd pnetcdf-1.12.3
./configure --prefix=$WRF_DEPS_DIR --with-netcdf4=$WRF_DEPS_DIR
make & make install
cd ..

# back to the top directory
cd ..
```

Then proceed to configure WRF by setting

``` sh
export NETCDF=$WRF_DEPS_DIR
export HDF5=$WRF_DEPS_DIR
export PNETCDF=$WRF_DEPS_DIR
```

``` sh
cd WRF-4.1.1
#!/bin/bash

module purge >& /dev/null
module load netCDF-Fortran/4.6.1-gompi-2023a

# set the compilers and supress warnings
export fallow_argument=-fallow-argument-mismatch
export boz_argument=-fallow-invalid-boz
export FFLAGS="$fallow_argument $boz_argument -m64"
export FCFLAGS="$fallow_argument $boz_argument -m64"
export CC=gcc
export CXX=gcc
export MPICC=mpicc
export MPICXX=mpicxx
export NETCDF=$EBROOTNETCDFMINFORTRAN

git clone --recurse-submodule https://github.com/wrf-model/WRF.git --depth 1 --branch v4.6.0

cd WRF
echo -e "\n\n\033[31m============If you are on Mahuika please select option 34 below============\033[0m"
./configure
[select 15]
```

and build the code with
echo -e "\n\n\033[31m============Done with configure step. Now compiling WRF. Build log in './WRF/wrf_build.log'============\033[0m"
export J="-j 12"
./compile em_real >& wrf_build.log

``` sh
./compile em_real >& log.compile
```
!!! Note
Please select option 34 (dmpar gfortran/gccGNU) when asked `Please select from among the following Linux x86_64 options`.

This may take several hours to compile. Check the log file to ensure
that the compilation was successful.
It will take some time for WRF to compile (~30 minutes). You may wish to run this from a [tmux](https://docs.nesi.org.nz/Getting_Started/Cheat_Sheets/tmux-Reference_sheet/) session to minimise the risk of disconnecting. Check the `wrf_build.log` file for any error or warning messages when finished.

### Running WRF on Mahuika

An example Slurm script for running WRF on Mahuika extension, which can
be submitted with *sbatch name\_of\_script.sl*:
An example Slurm job script for WRF on Mahuika is given below. The job can be submitted with `sbatch` *name\_of\_script.sl*`

``` sl
#!/bin/bash -e
#SBATCH --job-name=wrf
#SBATCH --time=01:00:00
#SBATCH --ntasks=240
#SBATCH --ntasks=36
#SBATCH --hint=nomultithread
#SBATCH --partition=milan
module purge
module load HDF5/1.12.2-iimpi-2022a
module purge 2> /dev/null
module load netCDF-Fortran/4.6.1-gompi-2023a
module list
export LD_LIBRARY_PATH=$WRF_DEPS_DIR/lib:$LD_LIBRARY_PATH
# run real
srun --kill-on-bad-exit --output=real.log ./real.exe
Expand All @@ -158,9 +82,62 @@ if any individual task fails. Without this option, the WRF job will stay alive
until the wall limit is reached but won't actually do anything.


### Building WPS on Mahuika
The following script will build serial WPS on Mahuika. Like the WRF build process, this will ask you to specify the compiler from a list of options:

``` sh
#!/bin/bash

wget https://github.com/wrf-model/WPS/archive/refs/tags/v4.6.0.tar.gz
tar xf v4.6.0.tar.gz
cd WPS-4.6.0

module purge 2> /dev/null
module load netCDF-Fortran/4.6.1-gompi-2023a
module load JasPer/2.0.33-GCC-12.3.0

export NETCDF=$EBROOTNETCDFMINFORTRAN
export NETCDFF=$EBROOTNETCDFMINFORTRAN
export HDF5=$EBROOTHDF5
export JASPERLIB=$EBROOTJASPER/lib64/libjasper.so
export JASPERINC=$EBROOTJASPER/include/jasper/jasper.h
export WRF_DIR='path/to/WRF/directory'

./clean > /dev/null 2>&1

echo -e "\n\033[31m=============On Mahuika, please choose option 1 below===========\033[0m"
./configure

echo -e "\n\033[31m=============Now compiling WPS. log file is './WPS-4.6.0/WPS_build.log'===========\033[0m"
./compile >& WPS_build.log

```
!!! Note
Change the `WRF_DIR` directory to the *full path* where you built WRF. Also, please **choose option 1** (`Linux x86_64, gfortran (serial)`) for the compiler.

WPS will compile much faster than WRF. Most WPS jobs can be run from the command line on the login node. If you wish to submit a WPS job (`geogrid.exe` for example) to a compute node, it can be done via the following Slurm script:
```
#!/bin/bash -e
#SBATCH --job-name=wps
#SBATCH --time=01:00:00
#SBATCH --cpus-per-task=1
#SBATCH --hint=nomultithread
module purge 2> /dev/null
module load netCDF-Fortran/4.6.1-gompi-2023a
module load JasPer/2.0.33-GCC-12.3.0
export WRF_DIR='path/to/WRF/build'
# run geogrid
./geogrid.exe
```
Note the required module environments if you wish to run `geogrid.exe` from the login node.

## WRF on Māui

### Building WRF
### Building WRF Māui

Load environment modules and set up the build environment (here we will
build with the Intel compiler):
Expand Down Expand Up @@ -210,7 +187,7 @@ message saying compilation was successful. You could also check the
timestamps of the executables in the *main* subdirectory
(*main/wrf.exe*, *main/real.exe*, etc).

### Running WRF
### Running WRF on Māui

An example Slurm script for running WRF on Māui, which can be submitted
with *sbatch name\_of\_script.sl*:
Expand Down Expand Up @@ -250,7 +227,8 @@ to 11 will use parallel netCDF instead. It is also recommended (possibly
required) to set *nocolons = .true.* in the *time\_control* section of
the namelist when using parallel IO.

### Building WPS

### Building WPS on Māui

Download WPS:

Expand Down Expand Up @@ -303,7 +281,7 @@ Now compile WPS:
Check that the executables were successfully created: *geogrid.exe*,
*ungrib.exe* and *metgrid.exe*.

### Running WPS
### Running WPS on Māui

Both *geogrid* and *metgrid* are parallel applications and can be run on
Māui compute nodes. However, *ungrib* is serial and should not be run on
Expand Down

0 comments on commit b7bef25

Please sign in to comment.