LILAC test failing in ctsm5.1.dev095 with cime/share/pio update

[ekluzek]

Brief summary of bug

LILACSMOKE test fails

General bug information

CTSM version you are using: ctsm5.1.dev096

Does this bug cause significantly incorrect results in the model's science? No

Configurations affected: o

LILACSMOKE_D_Ld2.f10_f10_mg37.I2000Ctsm50NwpSpAsRs.cheyenne_intel.clm-lilac Fails

Details of bug

Build fails. It looks like it's due to the shr_pio_init1 and shr_pio_init2 interfaces.

I don't get why this happens with the latest cime update and doesn't show up in the ctsm5.1.dev095 tag.

[ekluzek]

Applying similar changes that went into CMEPS I was able to get it to build at this point, but it doesn't run yet.

[jedwards4b]

@ekluzek In cmeps we moved the pio input parameters from namelists to nuopc.runconfig
I think that in lilac you will need to read the namelist and set NUOPC_CompAttributes that can then be read in
shr_pio_init.

[billsacks]

Thanks a lot for your reply to this @jedwards4b . We don't use NUOPC in LILAC (we use non-NUOPC-based ESMF) so the implementation will need to differ... hopefully it will still be doable....

[jedwards4b]

You use ESMF but not NUOPC? strange. Another approach might be to change the FilePath to use the older version of shr_pio_mod from the share directory.

[jedwards4b]

ESCOMP/CESM_share#33
I'm not quite sure how to distinguish a lilac build, but that issue should fix the problem.

[jedwards4b]

Turns out this also requires a modification to nuopc_shr_methods.F90 removing shr_pio_log_comp_settings

But I have more extensive changes to the pio interface in cmeps branch (in my fork) jedwards/asyncio
I'm not sure that the changes I am suggesting here will continue to work once those are brought into cmeps.

[wwieder]

I put this back as next so we can figure out when it may be fixed?

[ekluzek]

@wwieder I had talked with @billsacks about it before he left on vacation. We thought it would be best to wait until he got back and have him work on it. If needed @negin513 or I could work on it, but @billsacks will likely be able to fix it quicker.

[billsacks]

I have discussed this further with @jedwards4b (via discussion in ESCOMP/CESM_share#33) and @mvertens. Motivating points are:

1. We can't use the NUOPC-based version of shr_pio_mod from LILAC (since we don't use NUOPC in LILAC)
2. We want to keep some / much of the functionality of shr_pio_mod, since it deals with the initialization of PIO based on read-in settings
3. The non-NUOPC version of shr_pio_mod (in the share repository) will probably be removed once MCT is removed
4. We need an ability for ncdio_pio to obtain some values that are stored in shr_pio_mod, and we need this to be reasonably consistent for a LILAC and non-LILAC run.

Given these points, it feels like we'll need to maintain our own version of shr_pio_mod for use in LILAC. @mvertens and I feel that the simplest way forward would be to copy the current version of shr_pio_mod from the share repository into the CTSM source code: even though we probably don't need quite everything from there, we need a lot of it in order to read the PIO settings from a file and initialize PIO appropriately. To avoid name collisions, I propose the following:

• Change the (limited) uses of shr_pio in ncdio_pio to instead reference a new module, ctsm_shr_pio (so we'll have ctsm_shr_pio_getiosys, etc.)
• Create new files named ctsm_shr_pio_mod.F90 in each of CTSM's three driver directories (mct, lilac and nuopc)
  • The versions in the mct and nuopc directories will be small files that simply use shr_pio_mod with renames (so shr_pio_getiosys is renamed to ctsm_shr_pio_getiosys, etc.) (Note, though, that due to the different file paths used for mct vs. nuopc, they will end up wrapping different modules.)
  • The version in lilac will be a copy of the current shr_pio_mod in the share repository, but with all instances of shr_pio changed to ctsm_shr_pio. We will maintain this version for LILAC moving forward.

@ekluzek @jedwards4b @mvertens let me know if you have any thoughts on this plan. If people approve, then I will move forward with it next week.

[ekluzek]

I think I understand and I think it makes sense. Let me repeat to make sure I've got it. They'll be three versions fo ctsm_shr_pio_mod.F90

1.) NUOPC version that's a wrapper to the shr_pio_mod.F90 version in CMEPS.
Since it's a wrapper it'll be small and likely won't change much.
2.) MCT version that's a wrapper to shr_pio_mod.F90 version for MCT
Same as above for NUOPC, but it will go away eventually.
3.) LILAC version that's a copy of current shr_pio_mod.F90 from MCT
This will be a full file, but it's required for us to maintain it because the MCT version will go away.
Changes that go into the NUOPC shr_pio_mod may have to go into our version. And new PIO
features will need to go in as well.

It's a bit of a pain to have our own copy, but I don't see a way around it either. Maintaining the wrapper for NUOPC won't be a big deal though.

[billsacks]

Yes, that's right; thanks for the summary @ekluzek. In general, I imagine that the file for (3) will only need to change if we want to support new PIO features in LILAC operations; I'm not sure how often that will be the case.

I'll give others (@jedwards4b @mvertens ) a little more time to chime in if they have any thoughts objections; if I haven't heard any, I'll move forward with this plan.

[billsacks]

Well, shoot, I don't think my plan in #1759 (comment) is going to work: the LILAC test fails at runtime in some code that is invoking CDEPS. In retrospect that makes sense: CDEPS also uses shr_pio_mod, and with my plan, the variables in the actual shr_pio_mod never get initialized with LILAC (only those in the new ctsm_shr_pio_mod get initialized).

So I think we'll need to go with @jedwards4b 's initial proposal (ESCOMP/CESM_share#33 (comment)) after all.

[jedwards4b]

Will that work? Lilac uses cdeps and cdeps uses nuopc and the nuopc version of shr_pio_mod. I'm not sure you can use both at once?

[billsacks]

Will CDEPS only work with the nuopc version of shr_pio_mod? Based on the use statements in CDEPS (shr_pio_getiosys, shr_pio_getiotype, shr_pio_getioformat), it looks like it just needs to access some settings that – at a glance – are equivalent between the nuopc and non-nuopc versions of shr_pio_mod. But are there some subtle or implicit dependencies of CDEPS on the nuopc version of this module?

More broadly, it looks like there are two aspects to shr_pio_mod:

(1) Reading configuration files and initializing PIO appropriately

(2) Storing information about PIO (io system descriptors, io types, io formats) and providing an interface to query this information

It looks like (1) is only used by the driver and now – given the changes from a few months ago – is driver-specific. Piece (2) is the piece that actually needs to be shared between different components and – from a quick look – seems to be effectively identical between the NUOPC and non-NUOPC versions.

This leads me to another idea for how we could handle this LILAC issue: We could separate these two pieces into two different files. Piece (2) would live in the share code and would be used regardless of the driver. Piece (1) would be driver-specific, so for now we would have three versions of that piece: one in CMEPS (created by extracting the initialization pieces from cmeps/cesm/nuopc_cap_share/shr_pio_mod.F90), one in the cpl7 repo (created by extracting the initialization pieces from share/src/shr_pio_mod.F90), and one in CTSM's LILAC directory (which would be identical to the one in the cpl7 repo). Piece (2) – the actual share code piece – would be used by components (their use statements could stay exactly as they are now) as well as by piece (1) (which would be responsible for setting the module-level variables in piece (2)).

I feel like this design would make it less likely that we would run into future problems with LILAC in this respect – by clearly separating the shared pieces of this module from the driver-specific pieces. I'm on the fence, though, about whether it's worth the extra day or so of time that I think it would take to put this in place and do the necessary testing. @mvertens @ekluzek @jedwards4b let me know if you have feelings on this.

Of course, neither of these solutions will work if CDEPS truly depends on the NUOPC version of shr_pio_mod, so @jedwards4b I would like to hear if there are issues you're aware of that I may not be seeing.

[billsacks]

I spent a couple of hours doing the separation/consolidation I proposed in my last comment. I haven't tested yet, so I shouldn't say anything, but with that (major) caveat, I'll cautiously say that I think I have managed to do this to first order. I need to wrap up for the night, but I can show it to people tomorrow and you can let me know if you think this is a good path forward.

[billsacks]

tl;dr of this comment: I discussed an alternative approach with @mvertens and @jedwards4b that would allow us to use the existing shr_pio_mod from LILAC, but I still prefer my approach described above of letting each driver initialize PIO however it wants via splitting shr_pio_mod into two logical pieces.

I talked about this yesterday with @mvertens and @jedwards4b . A third possible implementation emerged from this discussion: It seems possible to use the recent NUOPC-based shr_pio_mod from LILAC if we change LILAC to:

(a) Set the required PIO settings as attributes on various ESMF components. For a PIO setting foo, LILAC would execute an ESMF_InfoSet call, setting attribute /NUOPC/Instance/foo. Note the namespacing into the /NUOPC/Instance namespace: this allows shr_pio_mod to extract the setting using a NUOPC_AttributeGet call. i.e., LILAC fakes that it's using NUOPC_AttributeSet to set these attributes.

(b) Rework the shr_pio initialization calls to fit with the new initialization interface in NUOPC's shr_pio_mod.

A benefit of doing this is that, as new PIO features are introduced in NUOPC's shr_pio_mod, we could leverage them from LILAC. However, this comes at a cost, and my view is that the cost of trying to reuse this same shr_pio_mod routine is greater than the benefits. The downsides I see are:

(1) I believe there will be a greater long-term maintenance cost of trying to reuse shr_pio_mod: any time there is a change to the PIO-related attributes in nuopc.runconfig, I believe we will need to make a similar change in LILAC. Granted, in some cases this could give us access to new features, but I'm not sure that benefit is worth the cost. Given the relatively limited use of LILAC, I feel like stability – i.e., low maintenance cost – is more important than keeping up with the latest and greatest new PIO features.

(2) The NUOPC shr_pio_mod interface expects a driver gridded component. For LILAC, we don't have a driver gridded component. We could probably figure out a way to make this work, but to some extent I feel like we're trying to squish a square peg into a round hole.

(3) This approach uses what I believe is an undocumented translation between ESMF_InfoGet and NUOPC_AttributeGet. This makes me a bit nervous in terms of the long-term potential for breakage.

(4) This approach relies on LILAC being the one to initialize PIO. If the host model uses PIO, I can envision a situation where the host model would want to use its own logic for initializing the PIO subsystem for the land. I think this would be hard to make work if we are trying to reuse NUOPC's shr_pio_mod, and would be easier to make work if LILAC were in more control of this process (as in my previous proposal). (That said, I'm not sure how likely this situation would be to arise in practice.)

What this really boils down to is a question of whether we want to keep LILAC's PIO initialization closely tied to that of CESM or allow it to diverge. There are costs and benefits of both, but I believe that we'll be better off long-term allowing it to diverge.

Stepping back from LILAC and considering the bigger picture of the architecture, I can also see the potential for my previous proposal – i.e., splitting shr_pio_mod into two pieces – providing a path forward for reusing CDEPS and other components in other systems without being tied to the same PIO initialization logic that CESM uses.