New section 4 templates to encode anomalies, significance and other products in relation with a reference period

[sebvi]

Initial request

ECMWF is engaged in migrating its operations to GRIB2 with the aim to complete the work in the following years. To support this activity, we are requesting new section 4 templates in the context of producing products that require the definition of a reference period.

There is a need to precisely define how the climatology is obtained, some of the information include the sample size of the reference (number of values used in the calculation of the reference), the number of years used (outer loop, for instance 30years), the number of hours/days/week/months used within each year (inner loop, for instance 5 weeks centred around the present day to compare to).

Once a model climatology is constructed, one can use it to compute for instance anomalies and significance.
It can also be used to derive cumulative distribution functions (CDFs) and probability density functions (PDFs), see CDF, PDF, EFI and SOT, and compute EFIs and SOTs which are very popular products where the CDF of a forecast run is compared to the CDF of a reference period ( model climate CDF ).

Amendment details

ADD the following templates entries in code table 4.0

Code	Description
105	Anomalies, significance and other derived products from an analysis or forecast in relation to a reference period at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval
106	Anomalies, significance and other derived products from an individual ensemble Forecast, control and perturbed in relation to a reference period at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval
107	Anomalies, significance and other derived products from derived Forecasts based on all ensemble members in relation to a reference period at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval
112	Probability of anomalies, significance and other derived products from an ensemble forecast in relation to a reference period at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval

ADD New Code Table 4.100 – Type of reference dataset

Code	Description
0	Analysis
1	Forecast
2	Reforecast (Hindcast)
3	Reanalysis
4	Climate projection
5	Gridded observations
6-191	Reserved
192-254	Reserved for local use
255	Missing

ADD New Code Table 4.101 – Type of relationship to reference dataset

Code	Description
0	Anomaly (see Note 1)
1	Standardized anomaly (see Note 1)
2	Significance (Wilcoxon-Mann-Whitney) (see Note 2)
3-19	Reserved
20	Extreme Forecast Index (EFI) (see note 3)
21	Shift of Tails (SOT) (see Note 4)
22-191	Reserved
192-254	Reserved for local use
255	Missing

Note 1: no additional parameter needed, NA in templates 4.105, 4.106 and 4.107 should be set to 0

Note 2: Confidence interval for the significance test is needed (NA=1)

Note 3: EFI is defined as described in https://doi.org/10.1256/qj.02.152. No additional parameter needed (NA=0)

Note 4: SOT is defined as described in https://doi.org/10.1256/qj.02.152. Lower and upper quantile used for the SOT need to be defined (NA=2) in this order: start of the tail (10 or 90 usually) then end of tail (1 or 99 respectively)

ADD New Code Table 4.102 – Statistical processing of reference period

Code	Description
0	Average
1	Accumulation
2	Maximum
3	Minimum
4	Median
5-19	Reserved
20	Model Climate
21-191	Reserved
192-254	Reserved for local use
255	Missing

ADD Template 4.105 - Anomalies, significance and other derived products from a analysis or forecast in relation to a reference period at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval

Octet	Description
10	Parameter Category (see code table 4.1)
11	Parameter Number (see code table 4.2)
12	Type of Generating Process (see code table 4.3)
13	Background Process
14	Generating Process Identifier
15-16	Hours After Data Cut-off
17	Minutes After Data Cut-off
18	Indicator of Unit of Time Range (see code table 4.4)
19-22	Forecast Time
23	Type of First Fixed Surface (see code table 4.5)
24	Scale Factor of First Fixed Surface
25-28	Scaled Value of First Fixed Surface
29	Type of Second Fixed Surface (see code table 4.5)
30	Scale Factor of Second Fixed Surface
31-34	Scaled Value of Second Fixed Surface
35-36	Year of End of Overall Time Interval
37	Month of End of Overall Time Interval
38	Day of End of Overall Time Interval
39	Hour of End of Overall Time Interval
40	Minute of End of Overall Time Interval
41	Second of End of Overall Time Interval
42	Number of Time Range
43-46	Number of Missing in Statistical Process
	The next six entries are repeated NT times nt=1:NT
47 + (nt-1)*12	Type of Statistical Processing
48 + (nt-1)*12	Type of Time Increment
49 + (nt-1)*12	Indicator of Unit for Time Range
50 + (nt-1)*12 - 53 + (nt-1)*12	Length of Time Range
54 + (nt-1)*12	Indicator of Unit for Time Increment (see code table 4.4)
55 + (nt-1)*12 - 58 + (nt-1)*12	Time Increment
	End of sequence repeated NT times
59 + (NT-1)*12	Type of Reference Dataset (see code table 4.100)
60 + (NT-1)*12	Type of Relation to Reference Dataset (see code table 4.101)
61 + (NT-1)*12	Number of Additional Parameter for Reference Period – NA (see note 1)
	The next two entries are repeated NA times na=0:NA – can be 0 times if no parameters are necessary
62 + (NT-1)*12 + (na-1)*5	Scale Factor of Additional Parameter for Reference Period
63 + (NT-1)*12 + (na-1)*5 – 66 + (NT-1)*12 + (na-1)*5	Scaled Value of Additional Parameter for Reference Period
	end of repeated NA sequence
62 + (NT-1)*12 + NA*5 - 63 + (NT-1)*12 + NA*5	Year of Start of Reference Period
64 + (NT-1)*12 + NA*5	Month of Start of Reference Period
65 + (NT-1)*12 + NA*5	Day of Start of Reference Period
66 + (NT-1)*12 + NA*5	Hour of Start of Reference Period
67 + (NT-1)*12 + NA*5	Minute of Start of Reference Period
68 + (NT-1)*12 + NA*5	Second of Start of Reference Period
69 + (NT-1)*12 + NA*5 – 72+ (NT-1)*12 + NA*5	Sample Size of Reference Period
73 + (NT-1)*12 + NA*5	Number of Reference Period Time Range – NR (see note 2)
	The next three entries are repeated NR times nr=1:NR
74 + (NT-1)*12 + NA*5 + (nr-1)*6	Type of Statistical Processing for Time Range for Reference Period (see code table 4.102)
75 + (NT-1)*12 + NA*5 + (nr-1)*6	Indicator of Unit of Time Range for Reference Period (see code table 4.4)
76 + (NT-1)*12 + NA*5 + (nr-1)*6 - 79 + (NT-1)*12 + NA*5 + (nr-1)*6	Length of Time Range for Reference Period
	end of repeated NR sequence

Note 1: see notes under table 4.101 for details

Note 2: "Number of Reference Period Time Range" is used to define an outer and if needed one or more inner loops. The outer loop gives the overall date and time range of the reference period and the inner loop(s) are used to specify a sub-period over all time instances defined in the outer loop in terms of the time unit defined by "Reference Period Indicator of Unit for Time Increment". Typically the outer loop defines a number of years and inner loop defines the date and time range within each year: for instance "every January for 30 years"

ADD Template 4.106 - Anomalies, significance and other derived products from an individual ensemble Forecast, control and perturbed in relation to a reference period at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval

Octet	Description
10	Parameter Category (see code table 4.1)
11	Parameter Number (see code table 4.2)
12	Type of Generating Process (see code table 4.3)
13	Background Process
14	Generating Process Identifier
15-16	Hours After Data Cut-off
17	Minutes After Data Cut-off
18	Indicator of Unit of Time Range (see code table 4.4)
19-22	Forecast Time
23	Type of First Fixed Surface (see code table 4.5)
24	Scale Factor of First Fixed Surface
25-28	Scaled Value of First Fixed Surface
29	Type of Second Fixed Surface (see code table 4.5)
30	Scale Factor of Second Fixed Surface
31-34	Scaled Value of Second Fixed Surface
35-36	Year of End of Overall Time Interval
37	Month of End of Overall Time Interval
38	Day of End of Overall Time Interval
39	Hour of End of Overall Time Interval
40	Minute of End of Overall Time Interval
41	Second of End of Overall Time Interval
42	Number of Time Range
43-46	Number of Missing In Statistical Process
	The next six entries are repeated NT times nt=1:NT
47 + (nt-1)*12	Type of Statistical Processing
48 + (nt-1)*12	Type of Time Increment
49 + (nt-1)*12	Indicator of Unit for Time Range
50 + (nt-1)*12 - 53 + (nt-1)*12	Length of Time Range
54 + (nt-1)*12	Indicator of Unit for Time Increment (see code table 4.4)
55 + (nt-1)*12 - 58 + (nt-1)*12	Time Increment
	End of sequence repeated NT times
59 + (NT-1)*12	Type of Ensemble Forecast (see code table 4.6)
60 + (NT-1)*12	Perturbation Number
61 + (NT-1)*12	Number of Forecasts in Ensemble
62 + (NT-1)*12	Type of Reference Dataset (see code table 4.100)
63 + (NT-1)*12	Type of Relation to Reference Dataset (see code table 4.101)
64 + (NT-1)*12	Number of Additional Parameter for Reference Period – NA (see note 1)
	The next two entries are repeated NA times na=1:NA– can be 0 times if no parameters are necessary
65 + (NT-1)*12 + (na-1)*5	Scale Factor of Additional Parameter for Reference Period
66 + (NT-1)*12 + (na-1)*5 – 69 + (NT-1)*12 + (na-1)*5	Scaled Value of Additional Parameter for Reference Period
	end of repeated NA sequence
65 + (NT-1)*12 + NA*5 - 66 + (NT-1)*12 + NA*5	Year of Start of Reference Period
67 + (NT-1)*12 + NA*5	Month of Start of Reference Period
68 + (NT-1)*12 + NA*5	Day of Start of Reference Period
69 + (NT-1)*12 + NA*5	Hour of Start of Reference Period
70 + (NT-1)*12 + NA*5	Minute of Start of Reference Period
71 + (NT-1)*12 + NA*5	Second of Start of Reference Period
72 + (NT-1)*12 + NA*5 – 75 + (NT-1)*12 + NA*5	Sample Size of Reference Period
76 + (NT-1)*12 + NA*5	Number of Reference Period Time Range – NR (see note 2)
	The next three entries are repeated NR times nr=1:NR
77 + (NT-1)*12 + NA*5 + (nr-1)*6	Type of Statistical Processing for Time Range for Reference Period (see code table 4.102)
78 + (NT-1)*12 + NA*5 + (nr-1)*6	Indicator of Unit of Time Range for Reference Period (see code table 4.4)
79 + (NT-1)*12 + NA*5 + (nr-1)*6 - 82 + (NT-1)*12 + NA*5 + (nr-1)*6	Length of Time Range for Reference Period
	end of repeated NR sequence

Note 1: see notes under table 4.101 for details

Note 2: "Number of Reference Period Time Range" is used to define an outer and if needed one or more inner loops. The outer loop gives the overall date and time range of the reference period and the inner loop(s) are used to specify a sub-period over all time instances defined in the outer loop in terms of the time unit defined by "Reference Period Indicator of Unit for Time Increment". Typically the outer loop defines a number of years and inner loop defines the date and time range within each year: for instance "every January for 30 years"

ADD Template 4.107 - Anomalies, significance and other derived products from derived forecasts based on all ensemble members in relation to a reference period at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval

Octet	Description
10	Parameter Category (see code table 4.1)
11	Parameter Number (see code table 4.2)
12	Type of Generating Process (see code table 4.3)
13	Background Process
14	Generating Process Identifier
15-16	Hours After Data Cut-off
17	Minutes After Data Cut-off
18	Indicator of Unit of Time Range (see code table 4.4)
19-22	Forecast Time
23	Type of First Fixed Surface (see code table 4.5)
24	Scale Factor of First Fixed Surface
25-28	Scaled Value of First Fixed Surface
29	Type of Second Fixed Surface (see code table 4.5)
30	Scale Factor of Second Fixed Surface
31-34	Scaled Value of Second Fixed Surface
35-36	Year of End of Overall Time Interval
37	Month of End of Overall Time Interval
38	Day of End of Overall Time Interval
39	Hour of End of Overall Time Interval
40	Minute of End of Overall Time Interval
41	Second of End of Overall Time Interval
42	Number of Time Range
43-46	Number of Missing In Statistical Process
	The next six entries are repeated NT times nt=1:NT
47 + (nt-1)*12	Type of Statistical Processing
48 + (nt-1)*12	Type of Time Increment
49 + (nt-1)*12	Indicator of Unit for Time Range
50 + (nt-1)*12 - 53 + (nt-1)*12	Length of Time Range
54 + (nt-1)*12	Indicator of Unit for Time Increment (see code table 4.4)
55 + (nt-1)*12 - 58 + (nt-1)*12	Time Increment
	End of sequence repeated NT times
59 + (NT-1)*12	Derived Forecast (see code table 4.7)
60 + (NT-1)*12	Number of Forecasts in Ensemble
61 + (NT-1)*12	Type of Reference Dataset (see code table 4.100)
62 + (NT-1)*12	Type of Relation to Reference Dataset (see code table 4.101)
63 + (NT-1)*12	Number of Additional Parameter for Reference Period – NA (see note 1)
	The next two entries are repeated NA times na=1:NA – can be 0 times if no parameters are necessary
64 + (NT-1)*12 + (na-1)*5	Scale Factor of Additional Parameter for Reference Period
65 + (NT-1)*12 + (na-1)*5 - 68 + (NT-1)*12 + (na-1)*5	Scaled Value of Additional Parameter for Reference Period
	end of repeated NA sequence
64 + (NT-1)*12 + NA*5 - 65 + (NT-1)*12 + NA*5	Year of Start of Reference Period
66 + (NT-1)*12 + NA*5	Month of Start of Reference Period
67 + (NT-1)*12 + NA*5	Day of Start of Reference Period
68 + (NT-1)*12 + NA*5	Hour of Start of Reference Period
69 + (NT-1)*12 + NA*5	Minute of Start of Reference Period
70 + (NT-1)*12 + NA*5	Second of Start of Reference Period
71 + (NT-1)*12 + NA*5 – 74 + (NT-1)*12 + NA*5	Sample Size of Reference Period
75 + (NT-1)*12 + NA*5	Number of Reference Period Time Range – NR (see note 2)
	The next three entries are repeated NR times nr=1:NR
76 + (NT-1)*12 + NA*5+ (nr-1)*6	Type of Statistical Processing for Time Range for Reference Period (see code table 4.102)
77 + (NT-1)*12 + NA*5 + (nr-1)*6	Indicator of Unit of Time Range for Reference Period (see code table 4.4)
78 + (NT-1)*12 + NA*5 + (nr-1)*6 - 81 + (NT-1)*12 + NA*5 + (nr-1)*6	Length of Time Range for Reference Period
	end of repeated NR sequence

Note 1: see notes under table 4.101 for details

Note 2: "Number of Reference Period Time Range" is used to define an outer and if needed one or more inner loops. The outer loop gives the overall date and time range of the reference period and the inner loop(s) are used to specify a sub-period over all time instances defined in the outer loop in terms of the time unit defined by "Reference Period Indicator of Unit for Time Increment". Typically the outer loop defines a number of years and inner loop defines the date and time range within each year: for instance "every January for 30 years"

ADD Template 4.112 - Probability of anomalies, significance and other derived products from an ensemble forecast in relation to a reference period at a horizontal level or in a horizontal layer in a continuous or non-continuous time interval

Octet	Description
10	Parameter Category (see code table 4.1)
11	Parameter Number (see code table 4.2)
12	Type of Generating Process (see code table 4.3)
13	Background Process
14	Generating Process Identifier
15-16	Hours After Data Cut-off
17	Minutes After Data Cut-off
18	Indicator of Unit of Time Range (see code table 4.4)
19-22	Forecast Time
23	Type of First Fixed Surface (see code table 4.5)
24	Scale Factor of First Fixed Surface
25-28	Scaled Value of First Fixed Surface
29	Type of Second Fixed Surface (see code table 4.5)
30	Scale Factor of Second Fixed Surface
31-34	Scaled Value of Second Fixed Surface
35-36	Year of End of Overall Time Interval
37	Month of End of Overall Time Interval
38	Day of End of Overall Time Interval
39	Hour of End of Overall Time Interval
40	Minute of End of Overall Time Interval
41	Second of End of Overall Time Interval
42	Number of Time Range - NT
43-46	Number of Missing in Statistical Process
	Repeats NT times for nt=1:NT
47+(nt-1)*12	Type of Statistical Processing
48+(nt-1)*12	Type of Time Increment
49+(nt-1)*12	Indicator of Unit for Time Range
50+(nt-1)*12 - 53+(nt-1)*12	Length of Time Range
54+(nt-1)*12	Indicator of Unit for Time Increment (see code table 4.4)
55+(nt-1)*12-58+(nt-1)*12	Time Increment
	End of repeated NT sequence
59 + (NT-1)*12	Forecast probability number
60 + (NT-1)*12	Total Number of Forecast Probabilities
61 + (NT-1)*12	Probability type (see code table 4.9.table)
62 + (NT-1)*12	Scale factor of lower limit
63 + (NT-1)*12 - 66 + (NT-1)*12	Scaled value of lower limit
67 + (NT-1)*12	Scale factor of upper limit
68 + (NT-1)*12 - 71 + (NT-1)*12	Scaled value of upper limit
72 + (NT-1)*12	Type of Reference Dataset (see code table 4.100)
73 + (NT-1)*12	Type of Relation to Reference Dataset (see code table 4.101)
74 + (NT-1)*12	Number of Additional Parameter for Reference Period – NA (see note 1)
	The next two entries are repeated NA times na=1:NA – can be 0 times if no parameters are necessary
75 + (NT-1)*12 + (na-1)*5	Scale Factor of Additional Parameter for Reference Period
76 + (NT-1)*12 - 79 + (NT-1)*12 + (na-1)*5	Scaled Value of Additional Parameter for Reference Period
	end of repeated NA sequence
75 + (NT-1)*12 + NA*5 - 76 + (NT-1)*12 + NA*5	Year of Start of Reference Period
77 + (NT-1)*12 + NA*5	Month of Start of Reference Period
78 + (NT-1)*12 + NA*5	Day of Start of Reference Period
79 + (NT-1)*12 + NA*5	Hour of Start of Reference Period
80 + (NT-1)*12 + NA*5	Minute of Start of Reference Period
81 + (NT-1)*12 + NA*5	Second of Start of Reference Period
82 + (NT-1)*12 + NA*5 - 85 + (NT-1)*12 + NA*5	Sample Size of Reference Period
86 + (NT-1)*12 + NA*5	Number of Reference Period Time Range – NR (see note 2)
	The next three entries are repeated NR times
87 + (NT-1)*12 + NA*5 + (nr-1)*6	Type of Statistical Processing of Reference Dataset (see code table 4.102)
88 + (NT-1)*12 + NA*5 + (nr-1)*6	Reference Period Indicator of Unit for Time Increment (see code table 4.4)
89 + (NT-1)*12 + NA*5 + (nr-1)*6 - 92 + (NT-1)*12 + NA*5 + (nr-1)*6	Length of Reference Period
	end of repeated NR sequence

Note 1: see notes under table 4.101 for details

Note 2: "Number of Reference Period Time Range" is used to define an outer and if needed one or more inner loops. The outer loop gives the overall date and time range of the reference period and the inner loop(s) are used to specify a sub-period over all time instances defined in the outer loop in terms of the time unit defined by "Reference Period Indicator of Unit for Time Increment". Typically the outer loop defines a number of years and inner loop defines the date and time range within each year: for instance "every January for 30 years"

Comments

No response

Requestor(s)

Olivier Iffrig, Ivan Tsonevsky, Robert Osinski and Sebastien Villaume (all ECMWF)

Stakeholder(s)

ECMWF

Publication(s)

Manual on Codes (WMO-No. 306), Volume I.2, GRIB code table 4.0 (new entries)
Manual on Codes (WMO-No. 306), Volume I.2, GRIB code table 4.100 (new table)
Manual on Codes (WMO-No. 306), Volume I.2, GRIB code table 4.101 (new table)
Manual on Codes (WMO-No. 306), Volume I.2, GRIB code table 4.102 (new table)
Manual on Codes (WMO-No. 306), Volume I.2, GRIB template 4.105 (new template)
Manual on Codes (WMO-No. 306), Volume I.2, GRIB template 4.106 (new template)
Manual on Codes (WMO-No. 306), Volume I.2, GRIB template 4.107 (new template)
Manual on Codes (WMO-No. 306), Volume I.2, GRIB template 4.112 (new template)

Expected impact of change

MEDIUM

Collaborators

No response

References

No response

Validation

No response

[sebvi]

Here is a link to the experimental version of ecCodes able to encode/decode the new templates: eccodes

Let me know if you do not have access.

EDIT with correct sample:
reference_period.zip

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference-1-and-2.11.2022 notes:

@sebvi samples are ready for validation; @amilan17 review for consistency

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference-21.11.2022 notes:
@amilan17 will review this week.

[amilan17]

@sebvi Please change template numbers to 99 - 102. I think you were trying to accomodate the numbering of the templates for issue #173, but this is probably going to get published first. The branch is ready for you to update.

[sebvi]

@amilan17 : it is the opposite, issue #173 is the issue that should change their numbering, this is what was decided at the meeting

[sebvi]

after checking, the number here are perfectly fine!! issue #167 uses 99-104, then this issue uses 105-107 and issue #171 uses 108-111 so everything is ok :D

[amilan17]

I see. I stand corrected. There are multiple templates being proposed in multiple issues ((#167, #169 and #171) and I think you have the numbers correct...

[sebvi]

yes exactly! sorry for the confusion :D

[sebvi]

After finalising the branch we realized we are missing a template. Would it be possible to add one more template for probabilities? (see template 4.112 in the proposal)

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference-12-and-13.12.2022 notes:

@amilan17 will make sure the extra template number (4.112) is not in use, @sebvi add note about the units for the new parameters in 4.112

[sebvi]

@amilan17 I forgot which "note about units" I need to add :S Could you please remind me?

[amilan17]

@sebvi -- I don't remember either. Maybe somebody at the meeting tomorrow will know....

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference-10.01.2023 notes:

@sebvi will double check the note options 

@amilan17 will review consistency

[marijanacrepulja]

@sebvi and @amilan17

In template 4.112
42 Number of Time Range - NT

Should octet 42 in all other templates 4.105, 4.106, 4.107 has the same definition. I mean to add -NT

Thanks

[sebvi]

I agree, it is the same entry so it should be consistent in all templates and it will make it clearer that it is the key determining the number of NT loops. Thank you for this comment.