COSP output appears to be broken

[brianpm]

What happened?

Recent development runs with COSP turned on are producing monthly output that is all zeros.

I have previously reported something similar, but I don't think this is directly related to #880.

I checked 13 recent simulations: /glade/derecho/scratch/hannay/archive/b.e23_alpha16g.BLT1850.ne30_t232.*

I looked at CLMODIS, FISCCP1_COSP, and CLD_MISR in the last month of each of those runs, and they are all zeros.

What are the steps to reproduce the bug?

Look at COSP outputs from recent CAM runs.

What CAM tag were you using?

whatever cecile has been using

What machine were you running CAM on?

CISL machine (e.g. cheyenne)

What compiler were you using?

Intel

Path to a case directory, if applicable

/glade/p/cesmdata/cseg/runs/cesm2_0/b.e23_alpha16g.BLT1850.ne30_t232.072

Will you be addressing this bug yourself?

No

Extra info

I can try to help diagnose what is going on, but I think I need help getting this fixed.

[brian-eaton]

@brianpm and @cecilehannay. I've looked into this issue and have found two problems, both related to the fact that the simulation is using interpolated output.

1. For some reason that I haven't been able to figure out the COSP fields that have two mdims (the dimensions between the horizontal grid dimensions and the time dimension) are ending up as zero in the history file. COSP fields with zero or one mdims are output correctly. If the fields are output on the native SE grid then there is no problem.
2. The interpolation scheme is not treating fillvalues correctly. This results in field values that are just a bit different from the fillvalue, and thus is not recognized as a fillvalue by post-processing software.

I think each of these problems will require substantial effort to resolve. My recommendation is to output the COSP fields on the native SE grid in a separate history file.

[brian-eaton]

Closing this now. Recommended workaround is to output COSP fields on native SE grid.