pio testing in standalone CICE

[apcraig]

Just note that we haven't decided whether we want to do this or not, but open an issue.

[dabail10]

I am doing some of this now with the history precision changes #270. On cheyenne, the io_pio directory doesn't build. I think this is because it needs to point to PIO version 1. The io_pio2 area builds fine with PIO2. However, it sets the restart_format to 'nc' and it should be 'pio'.

[dabail10]

Actually, it is not building with io_pio2. This defaults back to netcdf.

[dabail10]

I have a bit of a conundrum here. I need IO_TYPE to determine what modules to load. Does the build and env.machine_compiler have access to cice.settings? I assume so?

[phil-blain]

@dabail10 yes, cice.settings is sourced in cice.build, and then env.machine_compiler is sourced :
https://github.com/CICE-Consortium/CICE/blob/master/configuration/scripts/cice.build#L96-L97

[apcraig]

@dabail10, I'm glad this work is moving along. I assume you are using a prebuilt version of pio on cheyenne? All of the cice.settings env variables are available during the script build and run phases, so you should have what you need. I assume you are trying to figure out how to modify the Macros to pick up the correct pio version? We probably need to have a set_env.pio1 and set_nml.pio1 (and similar for pio2) to support testing. If I can help with anything, let me know. My hope is that both io_pio and io_pio2 build and run as currently implemented. We're using io_pio in RASM, and I think io_pio2 was used in CESM2 initial testing. If we need to modify io_pio or io_pio2, we'll want to be careful that we don't break RASM or other models.

[dabail10]

I've got a prototype for this. I can now get the io_pio2 to build and run on cheyenne. However, I am having problems with io_pio. The version of PIO1 we have on cheyenne is 1.10.1. I've got the current version on a branch of ESCOMP/CICE.

https://github.com/ESCOMP/CICE/tree/precision

The problem is that the pio1 lib does not have shr_kind_mod, shr_sys_mod or shr_pio_mod. Can you check your version of PIO for RASM and compare the ice_pio.F90 module with this branch?

[apcraig]

The CICE6 code in RASM is identical to the CICE6 code in the Consortium. We do have the shr code stuff in ice_pio.F90 in RASM. It seems clear this has to be updated to work with standalone CICE. Up to now, I guess all usage/testing always included CESM. It will be good to clean this up. I believe io_pio2 has the same issue. Have you already fixed this in io_pio2? It looks like the same mods should work in io_pio. I'm happy to test changes in RASM as part of development. Let me know if/how I can help. I'm even happy to take on the shr code cleanup if you'd like and we can then apply it to both io_pio and io_pio2. It should not be difficult.

[dabail10]

It builds and runs fine with io_pio2. So, I guess the PIO2 library must have some of this built in already?

[apcraig]

I just checked out ParallelIO and did a grep and am not all that impressed with the implementation. pio2 includes gptl which uses "shr" code but it doesn't call it shr code. Have a look in src/gptl/perf_utils.F90. I think this is very bad form to reuse the same names in the pio gptl implementation that exist in CESM. It can only create this kind of chaos. In addition, it looks like gptl can be turned off in pio with the TIMING cpp, so there is no guarantee that the "fake shr code in pio2" exists in the pio library.

I believe we should update ice_pio.F90 in both io_pio and io_pio2 to move away from all the "share" code cleanly. We don't want CICE to be dependent on this implementation.

[dabail10]

I agree this is bad. We can definitely replace shr_kind_mod and shr_sys_mod calls, but we still need the shr_pio_mod calls. I can't find where this is buried. In CESM, this is part of CIME src/share/util.

[apcraig]

I cannot tell where your io_pio2 standalone test is picking up the shr_pio stuff. I can't find it in ParallelIO or in CICE either. We probably just need to copy in the CIME src/share/util code and rename it in io_pio(2) area, but I'm curious about how your current test is building. Any chance you can track it down? It's probably good just to be careful.

[dabail10]

I see. I tried to set ICE_IOTYPE to pio2, but I guess it didn't take. My logic in the scripts is to use the io_pio2 directory when ICE_IOTYPE is set to pio2. Then it also should load the correct modules for PIO2. So, I was mistaken. Neither the io_pio or the io_pio2 is building.

[apcraig]

Since we haven't really tested the io_pio or io_pio2 in the standalone scripts, there could be some bugs. But I think what you are doing is the correct approach. Whatever is broken should be easy to fix too. It looks like there is a bunch of progress in your precision branch. Again, happy to help test and implement. Just let me know.

[dabail10]

Unfortunately, the code in ice_pio.F90 relies on subroutines from shr_pio_mod.F90. If we include shr_pio_mod.F90, this goes down a rabbit hole as it relies on other csm_share code. At some point I found a working version of ice_pio.F90 from CICE5.1 that @eclare108213 had put together. Not sure why this was not incorporated into CICE6.

[eclare108213]

Really, it works? I don't remember even getting pio to work in my own testing.

[dabail10]

This is the code you added and I was able to get it working with CICE5.1. It requires some PIO namelist additions:

!echmod ice_pio_subsystem => shr_pio_getiosys(inst_name)
!echmod pio_iotype = shr_pio_getiotype(inst_name)
!echmod: equivalent to shr code
!echmod: for now, assume all processors write output (numProcs)
! DAB: Let PIO choose.
integer (int_kind) :: numProcs ! number of processor participating
integer (int_kind) :: ierr

numProcs = distrb_info%nprocs

call t_initf('ice_in')

call PIO_init(my_task , & ! local task
MPI_COMM_ICE , & ! MPI communicator for ice comms
numProcs/istride , & ! number of I/O processors
0 , & ! aggregator
istride , & ! stride
PIO_rearr_box, & ! PIO-specific
ice_pio_subsystem) ! I/O system for ice

pio_iotype = PIO_iotype_pnetcdf ! type of parallel I/O

call PIO_setdebuglevel(3)

! pio_iotype = PIO_iotype_netcdf ! type of parallel I/O
!echmod

[apcraig]

Ultimately, we have to go down the rabbit hole if we want to be able to test/run standalone CICE with pio. It's likely/possible that we can simplify the depth and width of that hole as we pull things up. I don't know anything about a prior work-around. I think what's there now was based on the latest cice5 + columnphysics implementation. The io_pio came from what was working in RASM which was using the latest code prior to updating to CICE6. The io_pio2 probably came from what was working in CESM2 with CICE5.

Again, I'm happy to help with this task. Just let me know. I definitely think it's worth doing, even if we can only test on a small subset of machines where pio is pre-installed.

[apcraig]

@dabail10, can you provide an update on the status of this effort. I have decided it's important to have sooner than later. We need to be able to test pio and pio2 in standalone mode. The fact that these libs are built on cheyenne is a big help. I'd like to take this on if it's not already partly or fully implemented. Maybe addressing this first will help #270? If you'd like me to create a PR with this, let me know. If you'd like me to help push along a branch that is started but stagnated, let me know too.