How to use matrix.comp to compare regtests with develop

How to use matrix.comp to compare regtests with develop/main

1. Have a "base" and "comp" directories of WW3

First, you will want two different directories your "base" which is likely whatever branch you are working on and then a copy of the current main for your "comp". For example this might be:

 git clone https://github.com/fork/WW3  ww3_develop 
 cd ww3_develop 
 sh model/bin/ww3_from_ftp.sh     # to get the binary regtest files needed 
 cd ../
 cp -r ww3_develop ww3_branch 
 cd ww3_branch
 git checkout --track origin/branch   #where "branch" is the branch you are working on 

2. Run desired regression tests

Then you will want to run the desired regression tests in both the "base" and "comp" directories. This might be running the whole matrix of regression tests via a wrapper around matrix.base such as from the regtests directory using the bin/matrix_ncep or the bin/matrix_datarmor wrappers for matrix.base for a particular machine.

3. Run matrix.comp script

Then, you are ready to run the script, so from the ww3_branch/regtests directory:

./bin/matrix.comp all basedir compdir

to compare all of the regression tests, or to just compare one regression test

./bin/matrix.comp ww3_tp1.1 

4. Look at results

The output will be two files "matrixCompFull.out" and "matrixCompSummary.out" which will list the tests that are identical or not and which files differ. Not all of WW3 is bit-identical, meaning we expect some tests to come out with differences. For now (as of 4/18/2023 commit d609a49), the known list of not bit-identical tests from matrix.base are:

mww3_test_03/./work_PR1_MPI_e                     (1 files differ)
mww3_test_03/./work_PR2_UNO_MPI_d2                     (17 files differ)
mww3_test_03/./work_PR1_MPI_d2                     (10 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2_c                     (11 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2_c                     (13 files differ)
mww3_test_03/./work_PR3_UNO_MPI_d2                     (13 files differ)
mww3_test_03/./work_PR2_UQ_MPI_d2                     (16 files differ)
mww3_test_03/./work_PR3_UQ_MPI_e                     (1 files differ)
mww3_test_03/./work_PR3_UQ_MPI_d2                     (15 files differ)
ww3_ta1/./work_UPD0F_U                     (0 files differ)
ww3_tp2.10/./work_MPI_OMPH                     (7 files differ)
ww3_tp2.16/./work_MPI_OMPH                     (4 files differ)
ww3_ufs1.3/./work_a                     (3 files differ)

The issue with the mww3_test_03 grid set d2 is described in Issue #144 (See VLab issue #56247 for updates on reducing the list above.) The issues with tp2.10 and tp2.16 are documented in issue #321