BLOM iHAMOCC regression testing with NorESM

Regression testing for BLOM/iHAMOCC development can be done with the buildt-in CIME Case Control System (CCS) that is provided with the NorESM source code. Basic funtionality is included in the NorESM documentation section on system testing with NorESM.

Regression test system

The regression test system is located under cime/scripts, and called by running

./create_test TESTNAME

which will execute case.setup, case.build and case.submit.

Available regression tests for BLOM

$ ./query_testlists --xml-category aux_blom_noresm
aux_blom_noresm: SMS_D_Ld1.T62_tn14.NOINYOC.betzy_intel
aux_blom_noresm: ERS_Ld5.T62_tn14.NOINYOC.betzy_intel
aux_blom_noresm: ERS_Ld5.T62_tn14.NOINY.betzy_intel
aux_blom_noresm: ERS_Ld5.T62_tn14_wtn14nw.NOINY_WW3.betzy_intel
aux_blom_noresm: ERS_Ld5.T62_tn14_wtn14nw.NOINY_WW3.betzy_intel.blom-wavice

Run a single test on betzy

./create_test 
    SMS_D_Ld1.T62_tn14.NOINYOC.betzy_intel     
    --project <your_project>
or

./create_test
    SMS_D_Ld1.T62_tn14.NOINYOC.betzy_intel.blom-hamocc2 
    --project <your_project>

Will bring in xmlchanges and user_nl changes from 
   $SRCROOT/components/blom/cime_config/testdefs/testmods/blom/hamocc1/

Run a pre-defined test suite on betzy

./create_test 
    --xml-category aux_blom_noresm
    --xml-machine betzy
    --xml-compiler intel 
    --project <your_project>

Generate new baselines

NOTE: The default path (on betzy) for NorESM baseline tests is /cluster/shared/noresm/noresm_baselines, whereas baselines for BLOM development should go under /cluster/shared/noresm/noresm_baselines/blom_develop. Naming convention for BLOM development baselines is --generate <NorESM-tag>_<BLOM-development-tag>, where the BLOM development tag typically will be on the format dev#.#.#.#.

./create_test 
    --xml-category aux_blom_noresm
    --xml-machine betzy
    --xml-compiler intel 
    --generate <your_baseline_directory_name>
    --baseline-root /cluster/shared/noresm/noresm_baselines 
    --project <your_project>

Comparing regression test with existing baseline

./create_test 
    --xml-category aux_blom_noresm
    --xml-machine betzy
    --xml-compiler intel 
    --compare <your_baseline_directory_name>
    --baseline-root /cluster/shared/noresm/noresm_baselines 
    --project <your_project>

Example test file

A test file, e.g. BLOM_testfile.txt can be used to specify a set of regression tests, which may be easier to use than aux_blom_noresm when adjusting for specific test needs (e.g. testing a development branch),

SMS_D_Ld1.T62_tn14.NOINYOC.betzy_intel
ERS_Ld3.T62_tn14.NOINYOC.betzy_intel
ERS_Ld3.T62_tn14.NOINYOC.betzy_intel.blom-hamocc1
ERS_Ld3.T62_tn14.NOINYOC.betzy_intel.blom-hamocc2

Sample test script

#!/usr/bin/env bash
set -eu -o pipefail; shopt -s failglob

## Experiment settings
PROJECT="<my_nn####k_project>"
## TEST_TYPE :  "c" : XML_CATEGORY
##              "f" : FILE
##              "s" : single test
TEST_TYPE=${TEST_TYPE:-"f"}
## See $SRCROOT/cime/scripts/query_testlists for all available options.
## NorESM categories have the suffix "_noresm"
XML_CATEGORY=${XML_CATEGORY:-"aux_blom_noresm"}
## See .../query_testlists --list machine
XML_MACHINE=${XML_MACHINE:-"betzy"}
## See .../query_testlists --list compiler
XML_COMPILER=${XML_COMPILER:-"intel"}
## Default test file
TESTFILE=${TESTFILE:-"${PWD}/BLOM_testfile.txt"}
## Default test run
TESTNAME=${TESTNAME:-"SMS_D_Ld1.T62_tn14.NOINYOC.betzy_intel"}

## Model settings
TAG=${TAG:-"release-noresm2.1.1"}
LABEL=${LABEL:-"noresm2.1.1"}

## Component settings
## BLOM_TAG: This can be either a tag or a branch
##    tag: "<tag_name>"
##    branch: "develop/<branch_name>"
## BLOM_REPO: source for development version, either url to remote
##            repository or local path
BLOM_DEVELOP="yes"
if [ "${BLOM_DEVELOP}" == "yes" ]; then
    BLOM_LABEL=${BLOM_LABEL:-"master"}
    #BLOM_TAG=${BLOM_TAG:-"${BLOM_LABEL}"}           # Use this for tags
    BLOM_TAG=${BLOM_TAG:-"develop/${BLOM_LABEL}"}   # Use this for branches
    BLOM_REPO=${BLOM_REPO:-"https://github.com/<my_BLOM_fork>"}
    LABEL="${LABEL}_${BLOM_LABEL}"
fi

## Decide if you will generate baselines
GENERATE_BASELINES="no"
if [ "${GENERATE_BASELINES}" == "yes" ]; then
    if [ "${BLOM_DEVELOP}" == "yes" ]; then
        BASELINE_ROOT=${BASELINE_ROOT:-"/cluster/shared/noresm/noresm_baselines/blom_develop"}
        BASELINE_NAME=${LABEL}
    else
        BASELINE_ROOT=${BASELINE_ROOT:-"/cluster/shared/noresm/noresm_baselines"}
        BASELINE_NAME=${TAG}
    fi
fi

## Decide if you will compare against baselines
COMPARE_BASELINES="yes"
if [ "${COMPARE_BASELINES}" == "yes" ]; then
    BASELINE_ROOT=${BASELINE_ROOT:-"/cluster/shared/noresm/noresm_baselines/blom_develop"}
    COMPARE_NAME=${COMPARE_NAME:-"noresm2.1.1_dev1.5.1.3"}
fi

## Decide if you run model in debug mode
## - Debug flags enabled
RUN_DEBUG_MODE="no"

## Decied if we run with the --run-unsupported flag
## This should be used for compsets for which there are no
## science supported experiments available
RUN_UNSUPPORTED="yes"

## Decide whether or not to run manage_externals/checkout_externals
## While this is not perfect, it works if used consistently
## NB: If you check out the target tag from outside the script,
## be sure to run checkout_externals yourself, this script might
## miss it.
RUN_CHECKOUT="yes"

## Source and case directories
REPO=${REPO:-"https://github.com/NorESMhub/NorESM"}
SRCROOT="/cluster/projects/${PROJECT}/${USER}/NORESM/NorESM"


##--- Function: Check for execution errors ---
perror() {
    ## Print an error message and exit if a non-zero error code is passed
    if [ $1 -ne 0 ]; then
        echo "ERROR (${1}): ${2}"
        exit $1
    fi
}

##--- Checkout model source code ---
## (make sure that clone exists, otherwise, clone REPO)
if [ ! -d "${SRCROOT}" ]; then
    git clone -o NorESM ${REPO} ${SRCROOT}
    perror $? "running 'git clone -o NorESM ${REPO} ${SRCROOT}'"
    RUN_CHECKOUT="yes"
fi

## Ensure correct source is checked out
cd ${SRCROOT}
if [ "$( git describe )" != "${TAG}" ]; then
    git checkout ${TAG}
    perror $? "running 'git checkout ${TAG}'"
    RUN_CHECKOUT="yes"
fi
if [ "${RUN_CHECKOUT}" != "no" ]; then
    ./manage_externals/checkout_externals
    perror $? "running './manage_externals/checkout_externals'"
fi


##--- Checkout BLOM development version ---
if [ "${BLOM_DEVELOP}" == "yes" ]; then
    cd components/blom
    ## Remove old "develop" repo if present
    if git config remote.develop.url > /dev/null; then
        git remote rm develop
    fi
    git remote add develop ${BLOM_REPO}
    git fetch --all
    git fetch --tags --all
    git checkout --detach ${BLOM_TAG}
    cd ${SRCROOT}
fi


##--- Create test case
cn_args=" --project ${PROJECT} --xml-machine ${XML_MACHINE} --xml-compiler ${XML_COMPILER}"
cn_args="${cn_args} --output-root ${USERWORK}/noresm/${LABEL}"

case ${TEST_TYPE} in
    "c")
        cn_args="${cn_args} --xml-category ${XML_CATEGORY}"
        ;;
    "f")
        cn_args="${cn_args} --testfile ${TESTFILE}"
        ;;
    "s")
        cn_args="${cn_args} ${TESTNAME}"
        ;;
    *)
        echo "ERROR: Unknown TEST_TYPE ${TEST_TYPE}"
        exit
        ;;
esac

if [ "${GENERATE_BASELINES}" == "yes" ]; then
    cn_args="${cn_args} --baseline-root ${BASELINE_ROOT} --generate ${BASELINE_NAME}"
fi

if [ "${COMPARE_BASELINES}" == "yes" ]; then
    cn_args="${cn_args} --baseline-root ${BASELINE_ROOT} --compare ${COMPARE_NAME}"
fi

./cime/scripts/create_test ${cn_args}
perror $? "running './cime/scripts/create_test ${cn_args}'"