Support for initial conditions files

[anton-seaice]

Its been highlighted that the current behaviour of copying initial conditions files to every payu work directory, even when they are not longer needed because the model has already run and created restart files, is confusing.

The restart: option in config.yaml to start a 'new' experiment from an existing restart directory could provide a better place to put initial conditions files, however it's not perfect:

• We have been specifying input files as individual files, instead of directories (e.g. https://github.com/ACCESS-NRI/access-om2-configs/blob/fce24e35bdf35720a15b6b03fa3616d3486f50b4/config.yaml#L64C1-L67C117)
• What happens if you do a payu clone --restart but there are already values in the restart: section - do they get overwritten ?

(related: ACCESS-NRI/access-om2-configs#155)

[jo-basevi]

I think with the current restart: option, it would require a directory structure with directories the same names as sub-models in the config.yaml, e.g. ocean, atmosphere, ice. So maybe to specify a individual files for a sub-model, we could add a restart option to the sub-model config, e.g.:

- name: ocean
  model: mom
  exe: /g/data/vk83/apps/spack/0.20/release/linux-rocky8-x86_64/intel-19.0.5.281/mom5-git.2023.11.09=2023.11.09-qji4nlmr6utrribaiyhewe4je6mifguz/bin/fms_ACCESS-OM.x
  input:
      - /g/data/vk83/experiments/inputs/access-om2/ocean/grids/mosaic/global.1deg/2020.05.30/grid_spec.nc
          # ...
  restart:
     - /g/data/vk83/experiments/inputs/access-om2/ocean/initial_conditions/global.1deg/2020.10.22/ocean_temp_salt.res.nc  

Though that might be confusing when there's a directory defined under the top-level restart: as that probably should take precedence..

What happens if you do a payu clone --restart but there are already values in the restart: section - do they get overwritten ?

Yes, currently payu overwrites any existing values in the restart: section.

[anton-seaice]

Though that might be confusing when there's a directory defined under the top-level restart: as that probably should take precedence..

If the new section was named initial_conditions: instead of restart it could be slightly clearer ?

Its probably as clear as the current setup ? Where the files in input: and restart: folder can be serving the same purpose

I think with the current restart: option, it would require a directory structure with directories the same names as sub-models in the config.yaml

Why wouldn't we just add support for a list of files in the current restart option ?

[anton-seaice]

pinging @blimlim because he's setup esm1.5 with the warm-start directory and might have thoughts on what is clearest ?

[jo-basevi]

Why wouldn't we just add support for a list of files in the current restart option?

I think (?) payu needs to know what model each restart file relates to, as they get copied/symlinked to the model subdirectory in the payu work directory by the model driver - E.g. work/ocean/INPUT/ocean_temp_salt.res.nc is symlinked to the input file in config.yaml, and similarly work/ocean/INPUT/ocean_temp_salt.res.nc.0000 for the restart files.

[anton-seaice]

I think (?) payu needs to know what model each restart file relates to, as they get copied/symlinked to the model subdirectory in the payu work directory by the model driver - E.g. work/ocean/INPUT/ocean_temp_salt.res.nc is symlinked to the input file in config.yaml, and similarly work/ocean/INPUT/ocean_temp_salt.res.nc.0000 for the restart files.

Oh yes - that is true for the ACCESS 1/2 models :)

[blimlim]

If the new section was named initial_conditions: instead of restart it could be slightly clearer ?

I agree that this could be a clearer option!

I get a bit confused about initial vs restart runs in MOM and CICE. Is it correct that for the first run, OM2 uses these initial conditions files to construct a starting state rather than using a full restart?

If so, I think it makes sense to call the new section initial_conditions to distinguish between the two, and to also avoid interacting with things like the prior_restart_path logic, e.g. in CICE.

[anton-seaice]

Yes - OM2 uses an initial state for ocean temperature and salinity which is derived from an reanalysis product. It then generates some initial sea-ice distribution from the ocean temperature. (There are also some ocean BGC initial conditions dervied from "something"). These initial conditions are a (small) subset of the fields which are captured in a full model restart.

I guess as ESM1.5/1.6 uses a full restart for "initial_conditions", maybe this would only be a useful feature for the ocean models ?

[blimlim]

Thanks!

I guess as ESM1.5/1.6 uses a full restart for "initial_conditions", maybe this would only be a useful feature for the ocean models ?

Yeah I think this would be most relevant for the ocean models