How to treat time-unit "yrPt"?

[nierad]

Hi again,

I have another request on how to deal with the temporal unit "yrPt" (see e.g. here: http://clipc-services.ceda.ac.uk/dreq/u/fa23fd12-267c-11e7-9bed-ac72891c3257.html). This is not an annual mean but the value of a certain time-step, in CMIP6_Eyr.json it is described in the notes as the end-of-year value. For LPJG we have so far not cmorized values with this "frequency", because we only noticed this because they were missing. So I looked into data on the ESGF from some other providers. This didn't help very much, some use
time = 365 and time_bnds = [365, 365]
or just time (no bnds) in another temporal unit.
I think, since it is the "average" of only one daily timestep in LPJG, it should be treated as
time = 364.5 and time_bnds =[364, 365]
One thing to keep in mind is that time=365 in all the other EC-Earth outputs already lies in the following year.
Edit: The reason for it being in the following year is a matter of how we treat it.

I'd really like to hear your opinion on this.
Thanks, Lars

[treerink]

As a side comment:

For LPJG we have so far not cmorized values with this "frequency", because we only noticed this because they were missing.

Rather recently I did add in 386a750 the yrPt for the LPJG component of ece2cmor3 as part of #645 for the LAMACLIMA project. I am still waiting for Iris to repeat her run, so we can validate the final setup.

[nierad]

Hi Thomas,
this was the reason, they were skipped in the first place. So we added a

if frequency == "yr" or frequency == "yrPt":

case to lpjg2cmor.py. It "works" such that the data is cmorized and the files can be used. They do NOT pass the qa-dkrz test, though.

[nierad]

I have checked other model's output and found that they are actually using day 365 for the instantaneous value, which is Dec 31 24:00 o'clock of year Y , or Jan 1st 0 am (or pm ? Whatever, modnight, beginning of day) of year Y+1. This is what ece2cmor3 creates, too. As we only cmorize single years this would mean, the values for "end-of-year Y" would be "start-of-year Y+1" instead. Same point in time, different file-label. This isn't particularly nice, because there will now be a shift between the "yrPt" and other annual variables and it doesn't work with qa-dkrz.
This is what the data looks like for cVeg_Eyr_EC-Earth3-Veg_land-cCO2_r1i1p1f1_gr_1851-1851.nc

netcdf cVeg_Eyr_EC-Earth3-Veg_land-cCO2_r1i1p1f1_gr_1851-1851 {
dimensions:
        time = UNLIMITED ; // (1 currently)
        lat = 256 ;
        lon = 512 ;
        bnds = 2 ;
variables:
        double time(time) ;
                time:bounds = "time_bnds" ;
                time:units = "days since 1850-01-01" ;
                time:calendar = "proleptic_gregorian" ;
                time:axis = "T" ;
                time:long_name = "time" ;
                time:standard_name = "time" ;
        double time_bnds(time, bnds) ;
        double lat(lat) ;
....

data:

 time = 365 ;

and

 time_bnds =
  365, 365 ;
}

This is the error from qa-dkrz (check_logs/Annotations/EC-Earth-Consortium_EC-Earth3-Veg_land-cCO2_r1i1p1f1.json):

{
    "QA_conclusion": "FAIL",
    "QA_logfile": "/proj/s-cmip/users/lu/x_larni/cmorized/CCO2//QC/qa_Eyr/check_logs/EC-Earth-Consortium_EC-Earth3-Veg_land-cCO2_r1i1p1f1.log",
    "project": "CMIP6",
    "data_path": [ "/proj/s-cmip/users/lu/x_larni/cmorized/CCO2" ],
    "selection": [ "CMIP6/LUMIP/EC-Earth-Consortium/EC-Earth3-Veg/land-cCO2/r1i1p1f1/Eyr/*" ],
    "DRS_0": "CMIP6",
    "DRS_1": "LUMIP",
    "DRS_2": "EC-Earth-Consortium",
    "DRS_3": "EC-Earth3-Veg",
    "DRS_4": "land-cCO2",
    "DRS_5": "r1i1p1f1",
    "DRS_6": "Eyr",
    "DRS_7": "*",
    "DRS_8": "gr",
    "DRS_9": "v20210128",
    "annotation":
    [
        {
            "DRS_7": [ "cLitter", "cLitterLut", "cProduct", "cProductLut", "cSoil", "cSoilLut", "cVeg", "cVegLut", "fracLut" ],
            "annotation": "Negative/zero time-bounds range.",
            "tag": "R8",
            "severity": "Error"
        },
        {
            "DRS_7": [ "cLitter", "cLitterLut", "cProduct", "cProductLut", "cSoil", "cSoilLut", "cVeg", "cVegLut", "fracLut" ],
            "annotation": "Variable <time_bnds> declaration is inconsistent with cell_methods time: point.",
            "tag": "T_3c",
            "severity": "Warning"
        },
 }

The questions are:

• Will we use a different time-step (like 364 or 364.5)?
• Will we remove time_bnds for "yrPt" variables entirely (Some other models don't provide it)?
• Do we want to modify qa-dkrz or ece2cmor3? Is there even a way for the files to pass qa-dkrz?

Thanks again for your help!

[plesager]

So you mean that when cmorizing year Y, you create a file with Y+1 in the name? If you prefer to have step 365 file Y (which I would also expect, understanding "days since YYYY-01-01" starts with 1 pinned to YYYY-01-01, right?), then rename the files (you or ece2cmor3).

I suggest picking up the last available value of the year that you cmorize, set it to t=365 or 366, and without time bounds. I would advise against "time = 364.5 and time_bnds =[364, 365] ", because that's not a time point.

Does your data pass the nctime test?

[nierad]

So you mean that when cmorizing year Y, you create a file with Y+1 in the name? If you prefer to have step 365 file Y (which I would also expect, understanding "days since YYYY-01-01" starts with 1 pinned to YYYY-01-01, right?), then rename the files (you or ece2cmor3).

That could easily be done, but wouldn't that create issues somewhere else?

I suggest picking up the last available value of the year that you cmorize, set it to t=365 or 366, and without time bounds. I would advise against "time = 364.5 and time_bnds =[364, 365] ", because that's not a time point.

There is only one value per year. And that is the last timestep's value. For LPJG that runs on a daily time-step 364.5 would not be unappropriate, but I get your point (and qa-dkrz disallows it, too). My question is: How to get rid of time_bnds?

Does your data pass the nctime test?

I can't make the nctime-test run on our machine...rolleyes

[plesager]

So you mean that when cmorizing year Y, you create a file with Y+1 in the name? If you prefer to have step 365 in file Y (which I would also expect, understanding "days since YYYY-01-01" starts with 1 pinned to YYYY-01-01, right?), then rename the files (you or ece2cmor3).

That could easily be done, but wouldn't that create issues somewhere else?

If done correctly, that should not create problems elsewhere. Looks like an issue for ece2cmor3 to resolve. But a simple post-ece2cmor3 script would do the trick... and we already have plenty of them.

[...] My question is: How to get rid of time_bnds?

A simple call to ncatted should work.

I would first do a qa-dkrz test on files that you have renamed from Y+1 to Y, and from which you have removed the time_bounds.

[uwefladrich]

[...] I can't make the nctime-test run on our machine...rolleyes

Are you on tetralith? Because Prashanth has built a container with nctime, look here. There is another container with the cmor-fixer. Haven't tested either of them yet, but sounds cool.

[etiennesky]

I am unsure what is the status of the LPJG YrPt files in current version, is it supported?

[etiennesky]

also - do we need to apply any changes to LPJG and add the corresponding entries to the .ins files for this to work? I see that the commit 386a750 adds them

[nierad]

@etienesky: yrPt files don't yet pass the test, but I will address this again now.
Re changes to LPJG and ins-files: I don't know if I understand correctly, but no, I don't think there need to be any changes to LPJG. When lpjg2cmor was done, yrPt was just missed as a possible frequency in ece2cmor.

[treerink]

I have seen that the LPJG YrPt technically could be produced, so on that level it is working now. I have no idea whether the quality is ok , but that is what Lars is being commenting on and now is taking up again.

[nierad]

Hi all,
technically, I have resolved the time_bnds issue (and yrPt variables do not require time_bnd) and am left with the point-in-time issue.
qa-dkrz tells me the following, after I renamed the files to the correct years:

- date: 2021-04-26T15:59:49
   file: cVeg_Eyr_EC-Earth3-Veg_land-cCO2_r1i1p1f1_gr_1851-1851.nc
   data_path: /proj/s-cmip/users/lu/x_larni/cmorized/CCO2/CMIP6/LUMIP/EC-Earth-Consortium/EC-Earth3-Veg/land-cCO2/r1i1
p1f1/Eyr/cVeg/gr/v20210426
   result_path: /proj/s-cmip/users/lu/x_larni/cmorized/CCO2//QC/qa_Eyr/data/CMIP6/LUMIP/EC-Earth-Consortium/EC-Earth3-
Veg/land-cCO2/r1i1p1f1/Eyr/cVeg/gr/v20210426
   tracking_id: hdl:21.14100/f1b15eb3-87af-4abb-bf45-df66f059c77e
   period:
    begin: 1852-01-01T00:00:00
    end: 1852-01-01T00:00:00
   conclusion: CF: PASS, CNSTY: FAIL, CV: disabled, DATA: PASS, DRS: PASS, TIME: FAIL
   events:
    - event:
       annotation: 'Gap between time values across files.'
       impact: L1
       tag: 6_13
       info:
        -  t(last) of previous file=365 (1851-01-01T00:00:00), t(1st) of current file=730 (1852-01-01T00:00:00)
    - event:
       annotation: 'Auxiliary variable <time_bnds> is missing across sub-temporal files.'
       impact: L1
       tag: 8_4a
    - event:
       annotation: 'Attribute <time>:bounds is missing across sub-temporal files.'
       impact: L1
       tag: 8_7a
    - event:
       annotation: 'CMOR Error: Your filename <cVeg_Eyr_EC-Earth3-Veg_land-cCO2_r1i1p1f1_gr_1851-1851.nc> does not mat
ch the CMIP6 requirement.'
       impact: L1
       tag: ab45
       info:
        -  Your output filename should be:
        - <cVeg_Eyr_EC-Earth3-Veg_land-cCO2_r1i1p1f1_gr_1852-1852.nc> and should follow this template:
        - <<variable_id><table><source_id><experiment_id><member_id><grid_label>> See your Control Vocabulary
        - file.(/proj/s-cmip/users/lu/x_larni/cmorized/CCO2/QC/qa_Eyr/tables/cmip6-cmor-tables/Tables/CMIP6_CV.json)
   status: 1

qa-dkrz now complains about:

• missing time_bnds causing the CNSTY:FAIL
• a time gap between to succeeding files, causing TIME:FAIL

Apart from that my limited knowledge of how qa-dkrz works tells me, that all these are only "L1" issues, that I interpret as warnings. Further, I still think that the point in time 365 is wrong, because it belongs to the following year (and qa-dkrz treats it rightly so).
However, a quick test with 365-1 delivers the same set of warnings, of course without the "Wrong filename"-warning.

No idea, how I should proceed. Any suggestions welcome!

[nierad]

Ok. I did use the nctime-checks in the container that @ufladrich pointed me to. This worked quite nicely. The outcome was, that it seems to be correct, to use timestep 365 and have ece2cmor produce files for the following year, i.e. instead of having files from 1850-2014, there will be 1851-2015 instead. Also, setting time_bnd = None for yrPt fixes the error of wrong boundaries. Is this the way to move forward now?

Edit: This setting still creates a "Gap between time values across files."-error in qa-dkrz.
Seems, the tests are not consistent.

[uwefladrich]

I regret to say, but I have given up on QA-DKRZ. It is not supported any more at the moment and the prospects are doubtful. See IS-ENES-Data/QA-DKRZ#25 for a recent issue I had and the response from DKRZ. It does a decent job when/where it runs, but I can't get it to work in my environment anymore.

So relying on nctime for checking the time coordinate is a reasonable strategy as far as I can see.

[nierad]

Thanks a lot Uwe,
I will then consider this issue solved and ignore time-related reports from qa-dkrz.
I check a full dataset now and then commit (send changes to @treerink).

[pabretonniere]

Hi,

I did some tries on this. With a cdo settaxis,$y-01-01,00:00 $file ${file}.new. nctime won't complain (because it modifies the time axis and removes the time bounds) but the metadata in the file will be shifted of one year compared to the real data (because the point is corresponds to the snapshot of the end of the year and not the beginning).

Not sure what's the best option: keep the cmor files with the name not corresponding to what there is in the value of the time (as currently in ece2cmor), shift the time axis one year back to have the name and the metadata match (what I did with the problem I explained), or modify LPJG to have it compute the yearly as the first time step of the year instead of the last one.

For a non expert user that doesn't know the problem, both options 1 and 2 can be misleading...