Update with Dougie's config

[adele-morrison]

I think the decision at yesterday's TWG meeting was to move @dougiesquire's CESM config here, is that right?

Then the ACCESS-NRI ocean/sea ice working group can start evaluating the output and we can get things moving forward.

[dougiesquire]

Thank @adele157. I've transferred these to the COSIMA org here:

• https://github.com/COSIMA/gmom_jra_wd
• https://github.com/COSIMA/gmom_jra
• https://github.com/COSIMA/d_jra_wd

I've added a little detail to the READMEs, but in transferring I've lost permission to edit the repo names and descriptions. @aekiss / @micaeljtoliveira would one of you be able to make these a little less opaque?

Also, the input data currently lives in /g/data/tm70 and I don't have permission to copy it to ik11. Could someone copy across /g/data/tm70/cime/inputdata and update the input paths in the config.yamls accordingly?

[aekiss]

Thanks @dougiesquire I can copy the inputs to ik11. However most of the exes are missing from /scratch/tm70/ds0092/cime/scratch/*/bld/, perhaps because of scratch auto-cleanup - were these built using https://github.com/ACCESS-NRI/CESM as you describe in https://forum.access-hive.org.au/t/cesm-configurations-on-gadi-using-cime/115 ?

[dougiesquire]

Ah bugger. The GMOM_JRA_WD and GMOM_JRA executables were built using CIME as described in the forum post. For D_JRA_WD, I built the shared libraries using CIME and then did the rest of the build using the kluged scripts in https://github.com/COSIMA/CICE6-WW3/tree/main/build.

I can recreate these executables and put them somewhere more permanent.

[dougiesquire]

@aekiss, I don't have permission to write to /g/data/ik11 so I'll let you know once the builds are finished and you can copy the exes to where ever you put the input data. Not very good from a provenance perspective, but it will have to do for now...

[dougiesquire]

@aekiss, the exectuables for the different configs can be found here:

• MOM6-CICE6-WW3 : /scratch/tm70/ds0092/cime/scratch/GMOM_JRA_WD/bld/cesm.exe
• MOM6-CICE6 : /scratch/tm70/ds0092/cime/scratch/GMOM_JRA/bld/cesm.exe
• CICE6-WW3 : /scratch/tm70/ds0092/cime/scratch/D_JRA_WD/bld/cesm.exe

(i.e. at the paths currently listed in the config.yamls). Re provenance, I've also added some info to the READMEs on how these executables were generated.

[adele-morrison]

Thanks for sharing @dougiesquire! Is there existing output available for the MOM6-CICE6 config (and if so where)? Or do we need to run again / longer to start doing evaluation?

[dougiesquire]

I did have a small amount of output from each configuration on scratch, but it looks like that's started to get flushed so we may need to rerun. That should be simple, but we'll need to use the version of Payu in this branch (at least until we get it merged into the main repo)

[aekiss]

Thanks @dougiesquire I've put those exes here, with READMEs and symlinks to your bld dirs for some (temporary) provenance, and updated the config.yamls:

/g/data/ik11/inputs/cime/bin
├── CICE6-WW3
│   └── 2023-03-10
│       ├── bld -> /scratch/tm70/ds0092/cime/scratch/D_JRA_WD/bld
│       ├── cesm.exe
│       └── README
├── MOM6-CICE6
│   └── 2023-03-10
│       ├── bld -> /scratch/tm70/ds0092/cime/scratch/GMOM_JRA/bld
│       ├── cesm.exe
│       └── README
└── MOM6-CICE6-WW3
    └── 2023-03-10
        ├── bld -> /scratch/tm70/ds0092/cime/scratch/GMOM_JRA_WD/bld
        ├── cesm.exe
        └── README

[dougiesquire]

To install the version of Payu in my branch into a conda environment:

• Following the instructions under "Creating personal environments" here to create a conda environment. E.g. after editing your .condarc as described:

  conda create -n my_env python
  conda activate my_env

• pip install directly from by branch on github:

  pip install git+https://github.com/dougiesquire/payu.git@cesm_cmeps

[aekiss]

Thanks @dougiesquire I'm giving it a whirl.

I copied all the inputs from /g/data/tm70/cime/inputdata to /g/data/ik11/inputs/cime/inputdata/2023-03-10/ and updated the config.yamls

[adele-morrison]

Thanks @dougiesquire I'm giving it a whirl.

Yay, thanks @aekiss!

[aekiss]

The 3 exes depend on a libesmf.so shared library in /scratch/tm70 so require membership of tm70 to run, and will break with scratch cleanup

$ ldd /g/data/ik11/inputs/cime/bin/*/2023-03-10/cesm.exe | grep tm70
	libesmf.so => /scratch/tm70/mrd599/esmf-8.3.0/lib/libg/Linux.intel.x86_64_medium.openmpi.default/libesmf.so (0x00007f7b1f747000)
	libesmf.so => /scratch/tm70/mrd599/esmf-8.3.0/lib/libg/Linux.intel.x86_64_medium.openmpi.default/libesmf.so (0x00007f4408b67000)
	libesmf.so => /scratch/tm70/mrd599/esmf-8.3.0/lib/libg/Linux.intel.x86_64_medium.openmpi.default/libesmf.so (0x00007f38db7a7000)

[aekiss]

Micael suggested static linking instead, using
/scratch/tm70/mrd599/esmf-8.3.0/lib/libg/Linux.intel.x86_64_medium.openmpi.default/libesmf.a

I also have esmf-8.3.1 in a somewhat better location (though ik11 would be better)
/g/data/v45/aek156/spack/share/spack/modules/linux-rocky8-cascadelake/esmf-8.3.1-intel-2021.6.0-fdatj3f

but this will be a non-issue once we've got a complete build process in this repo, so maybe not worthwhile fixing now...

[dougiesquire]

Just noting that there is an access-om3 driver in the latest version of Payu that should be available in the hh5 conda environment from analysis3-23.04 (currently analysis3-unstable)

[aekiss]

Thanks @dougiesquire - I've updated the quick start instructions to reflect this.

[dougiesquire]

Closing this issue as the configurations have been moved into the COSIMA github org and are forming the basis for our ongoing development