BUFR Table D, New BUFR template for lightning observations from satellite

[marijanacrepulja]

Initial request

ECMWF is proposing a new BUFR template for representation of lighting observations from satellite . This to support assimilation of lighting observations in NWP models.

Amendment details

1. Add a new elements in table B

F X Y	ELEMENT NAME	UNIT	SCALE	REFERENCE VALUE	DATA WITH (bits)
0-20-139140	Lightning area	m2	0	0	30

F X Y	ELEMENT NAME	UNIT	SCALE	REFERENCE VALUE	DATA WITH (bits)
0-20-140141	Lightning energy	J	19	0	30

F X Y	ELEMENT NAME	UNIT	SCALE	REFERENCE VALUE	DATA WITH (bits)
0-20-141142	Lighting detection efficiency	%	0	0	7

2. Add a new template to table D

3 40 027		Lightning observations from satellite
	001007	SATELLITE IDENTIFIER
	002019	SATELLITE INSTRUMENTS
	002020	SATELLITE CLASSIFICATION
	005077	SUB SATELLITE LATITUDE
	006043	SUB SATELLITE LONGITUDE
	001033	IDENTIFICATION OF ORIGINATING/GENERATING CENTRE
	001034	IDENTIFICATION OF ORIGINATING/GENERATING SUB-CENTRE
	301011	YEAR, MONTH, DAY
	301012	HOUR,MINUTE
	004007	Seconds within a minute (microsecond accuracy)
	301021	Latitude/longitude (high accuracy)
	033003	QUALITY INFORMATION
	008090	DECIMAL SCALE OF FOLLOWING SIGNIFICANDS
	020140	LIGHTNING AREA
	008090	DECIMAL SCALE OF FOLLOWING SIGNIFICANDS	Set to missing (cancel)
	020141	LIGHTNING ENERGY

Comments

No response

Requestor(s)

Marijana Crepulja, Philippe Lopez (both from ECMWF)

Stakeholder(s)

Enter list of stakeholder(s).

Publication(s)

Example: Manual on Code (WMO-No. 306), Volume I.2, BUFR table D, table B

Expected impact of change

None

Collaborators

No response

References

No response

Validation

No response

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference.25.26.April.2023 notes:

• (Marijana) ECMWF wants to assimilate lightning from GOES satellites. Source data is NetCDF. (Anna) there is an open issue right now in WIGOS Metadata to improve codes lightning observations (Marijana) maybe we should place them in a class that can be used for satellite and ground stations? (Jeff) I was going to suggest class 20 for 'other phenomena', because there are some other lightning parameters there. 
• (Jeff) Do the units for flash density really need 30 bits? (Marijana) thousand flashes per km2 per day and it's going down to per second and wanted to have the maximum available; (ACTION) Marijana will provide the conversion formula and it can be tested in the BUFR examples
• (Sibylle) There is also already a lightning density parameter available <020129>; (Marijana) I think the units are different  (Jeff) sounds more like a lightning density rate? (Sibylle) it's not GRIB. [...] (Marijana) will compare the old value to the new value and discuss with scientists; (Antoine) (1) I think we need this in per seconds, because of the way the observations are coming from satellite; (2) We need to double check the test data from NetCDF.
• (Abderrazak) density is not an instantaneous value, but a calculation on a large area and a large time period; we calculate all types of flashes for km and day, but when we go down to meters and seconds we don't have all the information; (Sibylle) agreed density is more relevant at km2. (Marijana) will double check
• (Sergio) there are two points to check; 1) the range of the data; 2) the units.

[marijanacrepulja]

After discussion with colleague from research, we are not proposing lightning density with units flash m-2 s-1
We'll use already defined descriptor 020129 following time period.

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference.2&3May.2023 notes:
will only include flash area and energy; team agrees to use Table D 40, because it is from satellites and keep descriptors under 20 in Table B; @marijanacrepulja will ask if it's "flash" or it can be generalized for all types of lightning observations (stroke or... )

[marijanacrepulja]

Feedback from Philippe Lopez

"Regarding the naming convention, I think "lightning flash" and "lightning stroke" would be the most appropriate terminology. A single lightning flash, which typically lasts for a few hundreds of milliseconds, is usually made of several strokes (= electric discharges), which are much shorter in time (typically a few tens of microseconds), as explained for instance in the following link:

http://hyperphysics.phy-astr.gsu.edu/hbase/electric/lightning2.html

Besides, we could have new descriptors for "lightning area" and "lightning energy", which could be combined with existing descriptor 20124 to decide whether the information to be encoded relates to either stroke (1) or flash (2). 

One issue is that satellite and ground-based instruments work in very different ways. Satellite lightning imagers detect the optical signal coming from lightning discharges (at near-infrared frequency), while ground-based sensors detect the electromagnetic emissions from lightning discharges at very-low, low or very-high frequencies (VLF, LF or VHF). The physics of the measurements is therefore very different. 

Luckily, the term "flash" has the same meaning among the satellite and ground-based communities. On the other hand, when referring to the elements that make a given flash, the satellite community prefers to use the term (optical) "pulses", while "strokes" is the preferred term used by the ground-based community. In addition, "pulse" often seems to be the preferred term for lightning that does not reach the ground, while "stroke" usually implies some connection of lightning to the ground."

[amilan17]

https://github.com/wmo-im/CCT/wiki/16.May.2023.satellite.issues notes:
Marijana talked with Jeff other NOAA colleagues and there is no problem with this proposal moving forward; @SibylleK can validation in July;

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference.6.7.June.2023 notes:
branch is updated, Sibylle will validate in July

[marijanacrepulja]

lighting_sample_BUFR.tar.zip

Hi @SibylleK, thank you for your help in supporting validation. Hope you will be able to decode the samples provided. I have also updated the branch.

[SibylleK]

Here the output of the DWD bufr reading software:
OR_GLM-L2-LCFA_G16_s20211722239200_e20211722239405_c20211722239424.nc.bufr.DWDreadbufrxOutput.zip

I used the BUFR tables from the branch. The sample BUFR could be read and the data values in the output are the same as in the eccoced output, except for the slightly different rounding of the numbers.
Therefore, in a technical point of view, the validation was successful.

[amilan17]

https://github.com/wmo-im/CCT/wiki/Teleconference.13.July.2023 notes:

validated and ready

[amilan17]

@marijanacrepulja 

I merged the branch into FT2023-2 and I had to change the sequence numbers. (see issue summary)

I don't see the following used in Table D 40 in this branch... Is it used somewhere else? Or are you adding it now for use later? 

0-20-141	Lighting detection efficiency

[marijanacrepulja]

@amilan17 Many thanks for looking into this.

Yes, 0-20-141 is for later use.