New section 4 template to encode random fields for stochastic parametrizations and associated code table 4.X entries

[sebvi]

Initial request

Random fields for stochastic parametrizations like those in model uncertainty representations such as the Stochastically Perturbed Parametrization Tendencies scheme (SPPT) or the Stochastically Perturbed Parametrizations (SPP) scheme are used in ensemble forecasts and Ensemble data assimilation. To be able to use the same random fields during the cycling procedure in the data assimilation, ECMWF stores the random fields in the GRIB2 format to read it in sub-sequent cycling steps. This request proposes the implementation of a specific section 4 template to store random fields used for different stochastic perturbation methods and new table entries which allow to encode random fields in GRIB2. 

 The Template introduces the following new entries to describe the metadata:

Number of octets	Description
2	Random Field Number
2	Total Number of Random Fields
2	Spatio-temporal Scale Number
2	Total number of spatio-temporal scales
4	Scaled Value of Spatial Scale
1	Scale Factor of Spatial Scale
4	Scaled Value of Temporal Scale
1	Scale Factor of Temporal Scale

The first 2 entries are used to index the fields: random field 1 out of N, random field 2 out of N, etc.
The next 2 entries are used to index the spatial and temporal scales used in the perturbation.
The last 4 entries are used to encode the scale of the perturbation in space and time.

The new template is obtained by inserting these entries in the existing template 4.1.

To use this template we also need a set of metadata elements: a type of level in code Table 4.5 and parameters decribing the type of parameterization in code Table 4.2. To encode these new parameters, we propose a new discipline "Computational parameters" in Code Table 0.0, a new category "Stochastic parametrizations" within that new discipline in code Table 4.1.

Amendment details

ADD to code table 4.0 Product definition template

Code	Description
143	Random fields used in an ensemble forecast, at a horizontal level or in a horizontal layer at a point in time

ADD TEMPLATE 4.143 Random fields used in an ensemble forecast, at a horizontal level or in a horizontal layer at a point in time

Octet	Number of octets	Description
10	1	Parameter Category (see code table 4.1)
11	1	Parameter Number (see code table 4.2)
12	1	Type of Generating Process (see code table 4.3)
13	1	Background Process
14	1	Generating Process Identifier
15 to 16	2	Hours After Data Cut-off
17	1	Minutes After Data Cut-off
18	1	Indicator of Unit of Time Range (see code table 4.4)
19 to 22	4	Forecast Time
23-24	2	Random Field Number
25-26	2	Total Number of Random Fields
27-28	2	Spatio-temporal Scale Number
29-30	2	Total number of spatio-temporal scales
31-34	4	Scaled Value of Spatial Scale
35	1	Scale Factor of Spatial Scale
36-39	4	Scaled Value of Temporal Scale
40	1	Scale Factor of Temporal Scale
41	1	Type of First Fixed Surface (see code table 4.5)
42	1	Scale Factor of First Fixed Surface
43- 46	4	Scaled Value of First Fixed Surface
47	1	Type of Second Fixed Surface (see code table 4.5)
48	1	Scale Factor of Second Fixed Surface
49-52	4	Scaled Value of Second Fixed Surface
53	1	Type of Ensemble Forecast (see Code table 4.6)
54-57	4	Perturbation Number
58-61	4	Number of Forecasts in Ensemble

ADD to code table 4.5 Fixed surface types and units

Code	Description	Unit
191	Abstract level with no vertical localization (see note)	-

Note: This level has no defined location along the vertical axis. Scale factor and scaled values of first and second fixed surface should be set to "missing" if not used.

ADD to code table 0.0: Discipline of processed data in the GRIB message, number of GRIB Master table

Code	Description
191	Computational parameters

ADD to code table 4.1 , Discipline 191

Discipline	Code	Description
191	0	Stochastic parametrizations

CREATE code table 4.2.191.0: Product discipline 191 Computational parameters, parameter category 0: Stochastic parametrizations

Code	Description	Unit
0	Stochastically Perturbed Parametrization Tendency (SPPT) (see Note 1)	Numeric
1	Stochastically Perturbed Parameterizations (SPP) (see Note 2)	Numeric
2	Stochastic Kinetic Energy Backscatter  (SKEB) (see Note 3)	Numeric
3	Stochastic Trigger of Convection (STC) (see Note 4)	Numeric
4	Stochastic boundary-layer Humidity (SHUM) (see Note 5)	Numeric
5	Stochastic Total Tendency Perturbations (STTP) (see Note 6)	Numeric
6-191	Reserved
192-254	Reserved for local use
255	Missing

Notes:

1. Buizza, R., M. Miller, and T. N. Palmer, 1999: Stochastic representation of model uncertainties
   in the ECMWF ensemble prediction system. Quart. J. Roy. Meteor. Soc., 125, 2887-2908

2. Lang STK, Lock S-J, Leutbecher M, Bechtold P, Forbes RM. Revision of the Stochastically Perturbed Parametrisations model uncertainty scheme in the Integrated Forecasting System. Q J R Meteorol Soc. 2021; 147: 1364–1381. https://doi.org/10.1002/qj.3978

3. Shutts, G., 2004. A stochastic kinetic energy backscatter algorithm for use in ensemble
   prediction systems. Technical Memorandum 449, ECMWF.
   Shutts, G., 2005: A kinetic energy backscatter algorithm for use in ensemble prediction systems.
   Quart. J. Roy. Meteor. Soc., 131, 3079-3102.

4. Li J., J. Du and Y. Liu, 2015: A comparison of initial condition-, multi-physics- and stochastic
   physics-based ensembles in predicting Beijing “7.21” excessive storm rain event. Acta
   Meteorologica Sinica, 73(1), 50-71, DOI: 10.11676/qxxb2015.008

5. Du, Jun & Berner, Judith & Buizza, R. & Charron, Martin & Houtekamer, Pieter & Hou, Dingchen & Jankov, Isidora & Mu, Mu & Wang, Xuguang & Wei, Mozheng & Yuan, Huiling. (2018). Ensemble Methods for Meteorological Predictions. 10.1007/978-3-642-40457-3_13-1.

• Hou, D., Z. Toth, and Y. Zhu, 2006: A Stochastic Parameterization Scheme within NCEP Global
  Ensemble Forecast System. 18th AMS conference on Probability and Statistics. Atlanta, GA,
  Jan. 29-Feb. 2, 2006. [Available on line at
  http://www.emc.ncep.noaa.gov/gmb/ens/ens_info.html]

• Hou, D., Z. Toth, Y. Zhu, and W. Yang, 2008: Impact of a Stochastic Perturbation Scheme on
  NCEP Global Ensemble Forecast System. 19th AMS conference on Probability and Statistics.
  New Orleans, LA, 20-24 Jan. 2008. [Available on line at
  http://www.emc.ncep.noaa.gov/gmb/ens/ens_info.html}

Comments

No response

Requestor(s)

Sebastien Villaume (ECMWF) 
Robert Osinski (ECMWF) 
Martin Leutbecher (ECMWF)

Stakeholder(s)

ECMWF

Publication(s)

Manual on Codes (WMO-No. 306), Volume I.2, GRIB code table 4.0 (update) 
Manual on Codes (WMO-No. 306), Volume I.2, GRIB Template 4.143 (create) 
Manual on Codes (WMO-No. 306), Volume I.2, GRIB Template 0.0 (update) 
Manual on Codes (WMO-No. 306), Volume I.2, GRIB Template 4.1 (update) 
Manual on Codes (WMO-No. 306), Volume I.2, GRIB Template 4.5 (update) 
Manual on Codes (WMO-No. 306), Volume I.2, GRIB Template 4.2.191.0 (create) 

Expected impact of change

None

Collaborators

No response

References

No response

Validation

No response

[amilan17]

https://github.com/wmo-im/tt-tdcf/wiki/2024.11.13.tt.tdcf notes:

• Sebastien introduced the proposal, noted that will be useful for ML training
• @sebvi please add references for computational parameters
• from GRIB1
• @wmo-im/tt-tdcf - please review and comment on

[sebvi]

Please find attached some sample files and a version of ecCodes capable of encode/decode the new proposed templates.
eccodes-2.40.0-Source.tar.gz
FT2025-1_281.tar.gz

[amilan17]

https://github.com/wmo-im/tt-tdcf/wiki/2024.11.12.tt.tdcf notes:
@sebvi branch is ready for updates; Robert provided an overview; there were no further comments from team

[sebvi]

branch is updated

[amilan17]

https://github.com/wmo-im/et-data/wiki/2025.01.14.et.data notes:
@SibylleK will validate

[rdosinski]

Please find attached an ASCII file with the section 4 grib_dump as discussed in the meeting from 14/01/2025 :
dumps_281.tar.gz

[SibylleK]

The GRIB example could be read with a DWD "gribdump" software using the PDTs of the branch.
The values of section 4 with Product Definition Template Number 143 are the same as in the provided PDTN_143.dump.

The validation of PDT 143 is done.