problem with jpeg encoding of constant fields

[edwardhartnett]

On Mon, Feb 27, 2023 at 3:32 PM Wesley Ebisuzaki - NOAA Federal wesley.ebisuzaki@noaa.gov wrote:
All,

| Warning: g2lib/g2clib jpeg encode/deocde may differ from WMO standard, use -g2clib 0 for WMO standard

There is an error in the g2lib/g2clib library in the way that they encode
certain constant value grib2 files.

https://www.cpc.ncep.noaa.gov/products/wesley/wgrib2/g2clib.html

Both wgrib2 and ncep post avoid this error message by checking for a constant
field, and writing field in a format that can be read by g2lib and WMO-standard
decoders.

If you see this error message, and you created it with g2lib/g2clib, then it can
not be read correctly by a WMO-standard grib decoder. If you created the
grib message with ECcodes, for example, it cannot be read correctly by
g2lib.

Wesley

On Mon, Feb 27, 2023 at 3:04 PM Geoffrey Manikin - NOAA Federal geoffrey.manikin@noaa.gov wrote:

Hi Saeideh-

    My apologies - I see now that you were previously involved in this thread, and I effectively circled right back to you.    I agree that the warnings are odd if those products haven't been changed in v2.

    I've added Andrew Benjamin, who is a VPPPGB products engineer, to this thread.     Andrew, any chance that you're familiar with the odd wgrib2 errors that are being seen?

    I am also cc'ing Wesley from CPC who develops wgrib2.    Wesley, there are more details in the thread below, but the key takeaway is that running wgrib2 on new Great Lakes Wave model output yields the error
               Warning: g2lib/g2clib jpeg encode/deocde may differ from WMO standard, use -g2clib 0 for WMO standard

    and none of us are familiar with it.    Any help you can provide to help us understand the error (especially since the output has not changed) would be greatly appreciated.

                  -Geoff

On Mon, Feb 27, 2023 at 2:44 PM Saeideh Banihashemi - NOAA Affiliate saeideh.banihashemi@noaa.gov wrote:
Hi Geoff,

To the best of my knowledge, the VPPG branch has not worked with the GLWUv2 products. If I know who to contact, I could send the details of the WW3 grib2 file generation to them and we could double check the source of this warning.

Right now if I wanted to summarize the variables that we use to generate the grib2 files:
variable name : value
center ID: 7
generating process ID: 131 for 6day forecast, 133 for 2 day forecasts
grid definition: 255
GDS/BMS flag : 0
grid definition template number : 30 (for Lambert conformal) 0 for regular grid

It is also worth noting that none of these values have been changed compared to the current production model. I am not sure how relative this information might be, but it could be a starting point. Thank you all for your help,

Saeideh

On Mon, Feb 27, 2023 at 2:09 PM Geoffrey Manikin - NOAA Federal geoffrey.manikin@noaa.gov wrote:

Hi Brian-

   I'm not sure whether the products team worked much (if at all) with the GLWUv2 products.      Hui-ya, I suspect that you have an insane amount of emails to sort through today, so I am cc'ing the GLWU project lead (Saeideh) here for comment.

                             -Geoff

On Mon, Feb 27, 2023 at 1:42 PM Brian Gross - NOAA Federal brian.gross@noaa.gov wrote:
Hui-ya and Geoff, seeking your help with suggestions on who with grib2 expertise in VPPPG/product generation might be able to help diagnose the issues below in the GLWU application.

Did VPPPG work on GLWU?

Thanks,

Brian D. Gross
Director, NOAA/NWS/NCEP/Environmental Modeling Center

5830 University Research Court, Room 2050
College Park, MD 20740

Email: Brian.Gross@noaa.gov
Phone: 301-683-3748
Cell: 609-439-9702

---------- Forwarded message ---------
From: Avichal Mehra - NOAA Federal avichal.mehra@noaa.gov
Date: Mon, Feb 27, 2023 at 1:06 PM
Subject: Fwd: DF GLWU v2.0 - grib2 variables
To: Brian Gross - NOAA Federal brian.gross@noaa.gov
Cc: Jason Levit - NOAA Federal jason.levit@noaa.gov

Hi Brian,

We need someone from VPPG  to help us with this issue re. GLWU upgrade.

Thanks, Avichal.

---------- Forwarded message ---------
From: Arun Chawla - NOAA Federal arun.chawla@noaa.gov
Date: Mon, Feb 27, 2023 at 12:54 PM
Subject: Re: DF GLWU v2.0 - grib2 variables
To: William Boll - NOAA Federal william.boll@noaa.gov
Cc: Saeideh Banihashemi - NOAA Affiliate saeideh.banihashemi@noaa.gov, Samy Kamal - NOAA Affiliate samy.kamal@noaa.gov, Steven Earle - NOAA Federal steven.earle@noaa.gov, Anne Myckow - NOAA Federal anne.myckow@noaa.gov, Avichal Mehra - NOAA Federal avichal.mehra@noaa.gov, Jason Levit - NOAA Federal jason.levit@noaa.gov, Hui-Ya Chuang - NOAA Federal hui-ya.chuang@noaa.gov, Rahul Mahajan - NOAA Federal rahul.mahajan@noaa.gov

VPPG is the post processing branch at EMC. They have the grib2 expertise

Arun Chawla
Chief
Engineering & Implementation Branch
Room 2083
National Center for Weather & Climate Prediction
5830 University Research Court
College Park, MD 20740
Phone: 301-683-3740
Mobile: 240-564-5675
Fax: 301-683-3703

On Mon, Feb 27, 2023 at 12:44 PM William Boll - NOAA Federal william.boll@noaa.gov wrote:
Saeideh,
Thank you for confirming the new variables. Are you able to see if the code has changed in any way to account for the wmo grib2 errors? I do not see these errors currently in production. The new files are:

glwu.grlc_2p5km_lc_sr.tHHz.grib2
glwu.grlc_2p5km_lc.tHHz.grib2
glwu.grlr_500m_lc.tHHz.grib2
grib2.tHHz.awipsww3_grlc_2p5km_lc
grib2.tHHz.awipsww3_grlc_2p5km_lc_sr

The least two file types do contain headed data.

Arun,
Thank you for that information. Who/what is VPPG and how do we get into contact with them?

Thanks,
William

On Mon, Feb 27, 2023 at 11:13 AM Arun Chawla - NOAA Federal arun.chawla@noaa.gov wrote:
Those errors are an indication that you may be using non-standard wmo grib2 headers when generating the grib files. It would be good to note how the code creating these files have changed. We may need to bring in VPPG expertise here to help diagnose this issue

Arun Chawla
Chief
Engineering & Implementation Branch
Room 2083
National Center for Weather & Climate Prediction
5830 University Research Court
College Park, MD 20740
Phone: 301-683-3740
Mobile: 240-564-5675
Fax: 301-683-3703

On Mon, Feb 27, 2023 at 11:06 AM Saeideh Banihashemi - NOAA Affiliate saeideh.banihashemi@noaa.gov wrote:
William,

I discussed the subjects you had pointed out in this email internally with my team and here are some updates:

1. The 3 added variables in the grib2 files are expected and are coming from the third swell partition outputs. This comes with the grib2 updates and we would add this for sure in the SCN.

2. In terms of the grib2 warning since I am not familiar with the wmo files, I have cc'd Arun here to see if he could help us out or point us to who has more information on this topic.

Regards,

Saeideh

On Fri, Feb 24, 2023 at 9:38 AM William Boll - NOAA Federal william.boll@noaa.gov wrote:
Hi,

I am seeing a change in the number of variables between prod and para data.

of Files File Unique Prod Vars Prod Vars Unique Para Vars Para Vars

24 glwu.grlc_2p5km_sr.tHHz.grib2 784 16 931 19
4 glwu.grlc_2p5km.tHHz.grib2 2400 16 2850 19
24 glwu.grlr_500m.tHHz.grib2 784 16 931 19
4 grib2.tHHz.awipsww3_grlc_2p5km 2400 16 2400 16
24 grib2.tHHz.awipsww3_grlc_2p5km_sr 784 16 784 16

In particular, it looks like the variables below are the new variables being added. Are they expected? If so, they will need to be added to the SCN.
SWDIR:3 in sequence
SWELL:3 in sequence
SWPER:3 in sequence

Also, I am getting many warnings for all of the new LC data grib2 when using wgrib2. The files are:
glwu.grlc_2p5km_lc_sr.tHHz.grib2
glwu.grlc_2p5km_lc.tHHz.grib2
glwu.grlr_500m_lc.tHHz.grib2
grib2.tHHz.awipsww3_grlc_2p5km_lc
grib2.tHHz.awipsww3_grlc_2p5km_lc_sr
With the warning:
Warning: g2lib/g2clib jpeg encode/deocde may differ from WMO standard, use -g2clib 0 for WMO standard
The first three types of files are not WMO data. I do not see this warning with the current production data.

To add, I don't think I have ever seen this many unique variables for a single file. Each variable seems to have entries for each forecast hour and is the reason for the unique variable. This is currently in production. I am guessing this may assist in some way of the generation of the netrc files? Are the high number of variables expected?

Thanks,
William

--
NCEP Central Operations - Dataflow
College Park, Maryland 20740
Office: (381)-683-0511

--
NCEP Central Operations - Dataflow
College Park, Maryland 20740
Office: (381)-683-0511

--
Dr. Avichal Mehra
Chief, Dynamics and Coupled Modeling Group
Modeling and Data Assimilation Branch
NOAA/NWS/NCEP/EMC
Ph. 301-683-3746 Fax: 301-683-3703
Avichal.Mehra@noaa.gov