Bug fixes to 3D diagnostic tendencies; add more tendencies

[SamuelTrahanNOAA]

Bug fixes are added so the 3D diagnostic tendency output works properly. There is a new qdiag3d flag to turn on 3D tracer tendencies. Also, there are now tendencies for the model, ccpp, and total. The later two are for debugging the tendency calculations. The timestep_init and timestep_final stages of CCPP are implemented.

This is fully tested with gfs v16 beta, GSD v0, and GFS v15p2 suites.

However, there is an unresolved question of how to handle the problem of precision.

The Precision Problem

I can say definitively that double precision floating point is not enough to accumulate tendencies across the orders of magnitude typically found in the atmosphere. This was determined by adding a fourth tendency, now disabled in the code. Every time the ccpp or model tendencies were accumulated, that fourth tendency was accumulated by the same amount. In that case, the tendencies of: temperature had a precision of about 0.1 degrees, water vapor mixing ratio of 1e-5, and wind of .1 meters per second. Those are per-3-hour tendencies from a C768 global model. A higher-resolution run would have much larger imprecision due to the larger number of timesteps.

Those can be viewed as the best possible tendency errors.

Right now, the error in the tendencies of CCPP are about 2.5 times the best possible. It is hard to tell whether it is due to further precision issues (eight times as many additions per timestep) or incorrect tendency logic.

The precision issue could be dealt with using quadruple precision (128 bit) floating-point. That is a part of the Fortran 2008 standard, and it is an optional part of the MPI standard. It may be feasible to implement it, but that could cause portability issues.

[DomHeinzeller]

@llpcarson @grantfirl - for your information, the latest version of the 3D tendencies PR for NOAA-GSD/ccpp-physics.

[llpcarson]

Using "ccpp" here seems like the wrong terminology, or at least, it's not clear to me. Is this "outside of physics", i.e. dycore? Or, ?

[DomHeinzeller]

Firstly, apologies that it took me so long to review these PRs. I do have a few concerns about the implementation, which I also ran past the developers of the next-generation CCPP framework code generator - to make sure that I am not giving wrong directions wrt future development. The following is based on my understanding of what the CCPP is required/expected to do, and I may have missed important details.

The issue I am seeing is that code is added to the ccpp-physics repository that is responsible for calculating model tendencies before/after the actual physics are run. This is not the responsibility of the CCPP. The CCPP is "subordinate" to the host model and does what it is supposed to do, run the physics, return updated quantities and tendencies of so requested. It is the host model's responsibility to calculate its own tendencies from the dycore and other processes, and use the tendencies from CCPP for output or whatever else it wants to do.

As such, I believe that the code in physics/model*.F90 does not belong inside the CCPP. This code belongs to the host model and could be coded up in atmos_model.F90 (or maybe CCPP_driver.F90, but I will need to think about this a little more). This also means that the newly added timestep_init and timestep_final groups are not necessary, and some of the variables added to GFS_typedefs.F90 (e.g. the bookkeeping array) may not be required either.

Regarding the changes to the actual physics schemes, I think this is really good. The idea to use flags to signal of developers have already put the code to populate the 3d physics tendency arrays inside their schemes or not is much better than "if gf then ... else if sas then ... end if". For the moment this is hard-coded in GFS_typedefs.F90, which is ok - the current CCPP framework does not provide any other way. With cap_gen.py, we may be able to obtain this information from the CCPP itself (metadata, suite definition file) and remove the hard-coded section in GFS_typedefs.F90. Again, much better than having to weed through the physics schemes and take out if blocks based on which scheme is run.

I would like to get your feedback (in particular Sam) and, if what I am envisioning is ok, permission to propose changes on top of these PRs so that all the good work that went into the existing physics schemes gets used.

@llpcarson @grantfirl @ligiabernardet @gold2718 @SamuelTrahanNOAA

[SamuelTrahanNOAA]

I added the model, ccpp, and total tendencies for my own purposes, to debug the tendency calculations. I don't see their addition to the master as being critical. It is quite possible it isn't useful to anyone but me.

I suggest they be entirely removed from this PR. If they're deemed needed in the future, I can make a new PR.

[DomHeinzeller]

I added the model, ccpp, and total tendencies for my own purposes, to debug the tendency calculations. I don't see their addition to the master as being critical. It is quite possible it isn't useful to anyone but me.

I suggest they be entirely removed from this PR. If they're deemed needed in the future, I can make a new PR.

Ok, I will work on this while you are busy with the RAP implementation changes, and ask you to review the combined/amended PRs once they are ready. Thanks!