degrib2 not correctly interpreting negative forecast time

[edwardhartnett]

Ed,

The original WMO documentation didn't specify that the forecast time

could be negative. So some people assumed it was an unsigned integer.
Some people just said it was an integer and ignored the problem when
the integer was greater than 2G. In 2015, WMO issued a clarification that
forecast time could be negative integer. By the grib2 rules, the integer is
stored as a sign bit followed by the unsigned integer value.

Wesley

On Wed, Nov 2, 2022 at 11:59 AM Jianping Huang - NOAA Affiliate <jianping.huang@noaa.gov> wrote:

Hi Jun,

Kai will create one-day forecast products for CS, AK, and HI at 06z and 12z cycles and share the product files with Andrew for WMO header test, MDL for website display, and other users for test.  

Meanwhile, I have one question for Ed,

As Wesley pointed out, Wgrib2 lib stores signed integers as (1-bit sign) + (31-bits unsigned number) but Degrib2 is probably reading and writing the forecast hour as a "normal" integer (2-complement).  
Is the 2nd part of his statement correct? How can we solve the problem?

Thanks,

Jianping

On Tue, Nov 1, 2022 at 8:17 PM Jun Wang - NOAA Federal <[jun.wang@noaa.gov](mailto:jun.wang@noaa.gov)> wrote:

    Jianping,

    I am including Ed from the library team. Can you confirm the post code changes from Kai resolved the issue? Thanks

    Jun

    On Tue, Nov 1, 2022 at 4:59 PM Jianping Huang - NOAA Affiliate <[jianping.huang@noaa.gov](mailto:jianping.huang@noaa.gov)> wrote:

        Hi Fanglin,

        I am worrying that aqm.v7 is not able to reproduce the identical format to the current operational product in terms of display with degrib2 utility and failure with verification package. 

        Now Ho-Chun is able to read in the grib2 product after Kai made some changes to the post code. I am not sure the change is ok for other downstream users.

        We will get more details back to you later.

        Jianping

        On Tue, Nov 1, 2022 at 2:42 PM Fanglin Yang - NOAA Federal <[fanglin.yang@noaa.gov](mailto:fanglin.yang@noaa.gov)> wrote:

            Jianping,

            My understanding is that we will make the best effort to produce the same products from arm.v7 as swab.v6.   If everything is working fine in current operation, why an update to grib2 utilities is required?   I apologize if this has been explained before and I did not follow it closely. 

            Fanglin

            Sent from my iPhone

            On Nov 1, 2022, at 4:59 PM, Jianping Huang - NOAA Affiliate <[jianping.huang@noaa.gov](mailto:jianping.huang@noaa.gov)> wrote:

            
            Hi Jason,

            Thanks for the quick response. 

            Hi Jun,

            Do you know whom we should contact to get the support from the NCEP libs team? 

            Thanks,

            Jianping





            On Tue, Nov 1, 2022 at 12:20 PM Jason Levit - NOAA Federal <[jason.levit@noaa.gov](mailto:jason.levit@noaa.gov)> wrote:

                Hi Jianping,

                VPPPGB doesn't support wgrib2 or the g2 libraries, I believe this is supported by the NCEP libs team. Thanks,

                Jason

                On Tue, Nov 1, 2022 at 12:11 PM Jianping Huang - NOAA Affiliate <[jianping.huang@noaa.gov](mailto:jianping.huang@noaa.gov)> wrote:

                    Hi Jason,

                    We are working on generation of grib2 products for daily 8-hr maximum and 24-hr average PM2.5 in support of AQM v7.0 implementation.  Now we met an issue of reading the negative starting forecast hour(s) in the grib2 product with degrib2 utility. The details can be found from Kai's original email.  This may affect the verification and downstream users like MDL to display AQ forecasts on the [website](https://www.weather.gov/sti/stimodeling_airquality_predictions). 

                    According to Wesley, the g2lib and g2clib are probably at fault and probably need to be fixed. Can your branch provide help on this?

                    Please let us know if you need more information.

                    Thanks,

                    Jianping 


                    On Tue, Nov 1, 2022 at 11:32 AM Jun Wang - NOAA Federal <[jun.wang@noaa.gov](mailto:jun.wang@noaa.gov)> wrote:

                        Jianping,

                        Please check with Jason. Thank

                        Jun

                        On Tue, Nov 1, 2022 at 11:00 AM Jianping Huang - NOAA Affiliate <[jianping.huang@noaa.gov](mailto:jianping.huang@noaa.gov)> wrote:

                            Including Jun's email now.

                            Jianping

                            On Tue, Nov 1, 2022 at 11:00 AM Jianping Huang - NOAA Affiliate <[jianping.huang@noaa.gov](mailto:jianping.huang@noaa.gov)> wrote:

                                Thanks Wesley,

                                Let me include Jun for the possible issue(s) with g2lib and g2clib.

                                Jun,

                                Should we contact your branch or Jason's branch?

                                Thanks,

                                Jianping



                                On Tue, Nov 1, 2022 at 10:22 AM Wesley Ebisuzaki - NOAA Federal <[wesley.ebisuzaki@noaa.gov](mailto:wesley.ebisuzaki@noaa.gov)> wrote:

                                    Kai,

                                        The problem is that the forecast hour is a signed integer.  Grib
                                    stores signed integers as (1-bit sign) + (31-bits unsigned number).  Degrib2 is 
                                    probably reading and writing the forecast hour as a "normal" integer (2-complement).  
                                    The g2lib and g2clib are probably at fault and probably need to be fixed.  Anyways
                                    this is my guess without looking or using the code. (This was the wgrib2 problem that 
                                    was fixed 11/2015.)

                                      Getting wgrib2_api or wgrib2 to fix the files is not easy.  It is better that someone
                                    fixes the ncep library/tools so that this problem doesn't happen again.  Since the
                                    data assimilation systems are going to IAU, this will become a problem.  Isn't
                                    easy to submit a ticket about this problem?

                                    Wesley




                                    On Mon, Oct 31, 2022 at 6:18 PM Kai Wang - NOAA Affiliate <[kai.wang@noaa.gov](mailto:kai.wang@noaa.gov)> wrote:

                                        Hi Wesley,

                                        Thank you for your response! When we used the degrib2 command to show the header information. It showed differently.

                                        For example, degrib2 tmp_-1-22/aqm.t06z.ave_24hr_pm25.793.grib2, showed
                                        FIELD: PMTF    1 hybrid lvl (2147483647 -) valid  2147483647 hour before 2022081506:00:00 to 2022081604:00:0

                                        While degrib2 oper/aqm.t06z.ave_24hr_pm25.148.grib2, showed
                                        FIELD: PMTF     1 sigma (1 -) valid  1 hour before 2022081506:00:00 to 2022081604:00:00

                                        I'm curious if wgrib2 command or wgrib2api (e.g., grb2_wrt) can help to fix it (e.g., change 2147483647 hour before to 1 hour before)?

                                        Thanks,

                                        Kai

                                        On Mon, Oct 31, 2022 at 2:19 PM Wesley Ebisuzaki - NOAA Federal <[wesley.ebisuzaki@noaa.gov](mailto:wesley.ebisuzaki@noaa.gov)> wrote:

                                            Kai,

                                               Sorry for the slow reply, I was away last week on business.

                                            wesley.ebisuzaki@clogin04:/lfs/h2/emc/physics/noscrub/kai.wang/Diag_UPP/forissue28/c3/tmp_-1-22> wgrib2 aqm.t06z.ave_24hr_pm25.793.grib2
                                            1:0:d=2022081506:PMTF:1 hybrid level:-1-22 hour ave fcst:
                                            2:172095:d=2022081506:PMTF:1 hybrid level:23-46 hour ave fcst:
                                            3:376183:d=2022081506:PMTF:1 hybrid level:47-70 hour ave fcst:

                                            wesley.ebisuzaki@clogin04:/lfs/h2/emc/physics/noscrub/kai.wang/Diag_UPP/forissue28/oper> wgrib2 aqm.t06z.ave_24hr_pm25.148.grib2
                                            1:0:d=2022081506:PMTF:1 sigma level:-2147483647--2147483624 hour ave fcst:
                                            2:107890:d=2022081506:PMTF:1 sigma level:23-46 hour ave fcst:
                                            3:214501:d=2022081506:PMTF:1 sigma level:47-70 hour ave fcst:

                                            I am 99.99% certain that wgrib2 is right.  (Wgrib2 was fixed when the subject
                                            of negative forecast hours was proposed by the ECMWF.)

                                            Wesley






                                            On Tue, Oct 25, 2022 at 6:01 PM Kai Wang - NOAA Affiliate <[kai.wang@noaa.gov](mailto:kai.wang@noaa.gov)> wrote:

                                                Hi Wesley,

                                                I'm Kai from the EMC, who's working with Jianping on the implementation of AQMv7.0 forecasting system. We encountered an issue when using grb2_wrt (from wgrib2api) to generate the grib2 products and would like to seek some advice from you. The issue is that when we set up the starting time/ending time as -1-22 (see lines 91-95 in the Fortran code at /lfs/h2/emc/physics/noscrub/kai.wang/Diag_UPP/forissue28/PM25-stat.f90.orig on WCOSS2), the grib2 header shows (an example product grib2 file is at /lfs/h2/emc/physics/noscrub/kai.wang/Diag_UPP/forissue28/c3/tmp_-1-22/aqm.t06z.ave_24hr_pm25.793.grib2):

                                                "  PRODUCT TEMPLATE 4. 8: ( PARAMETER = PMTF     0 13 193 )  13 193 2 0 89 0 0 1 -2147483647 105 0 1 255 -127 -2147483647 2022 8 16 4 0 0 1 0 0 2 1 23 255 0
                                                  FIELD: PMTF    1 hybrid lvl (2147483647 -) valid  2147483647 hour before 2022081506:00:00 to 2022081604:00:0
                                                "
                                                The header encoding from the current AQMv6 product generated by a different approach (an example file is at /lfs/h2/emc/physics/noscrub/kai.wang/Diag_UPP/forissue28/oper/aqm.t06z.ave_24hr_pm25.148.grib2) is as follows:

                                                "  PRODUCT TEMPLATE 4. 8: ( PARAMETER = PMTF     0 13 193 )  13 193 2 0 211 0 0 1 -1 104 4 10000 255 0 0 2022 8 16 4 0 0 1 0 0 2 1 23 255 0
                                                  FIELD: PMTF     1 sigma (1 -) valid  1 hour before 2022081506:00:00 to 2022081604:00:00
                                                "
                                                One of the major differences is highlighted with red color. It looks like wgrib2 can't deal with the negative hour values (e.g. -1 in the above example). Since this AQM product will need two hours of records from the previous cycle for 06z forecast and seven hours from previous cycles for 12z forecast, we would like to get encoding like "1 hour before" for 06z cycle or "7 hour before" for 12z cycle as shown in the AQMv6 product. Sorry for a rather long email. We would really appreciate it if you could take a look at this issue for us. Please let me know if you need any clarification.

[edwardhartnett]

Hi Ed,

An example file can be found in /lfs/h2/emc/physics/noscrub/kai.wang/Diag_UPP/forissue28/c3/tmp_-1-22/aqm.t06z.ave_24hr_pm25.793.grib2

wgrib2 -v aqm.t06z.ave_24hr_pm25.793.grib2
"1:0:d=2022081506:PMTF:1 hybrid level:-1-22 hour ave fcst:
2:172095:d=2022081506:PMTF:1 hybrid level:23-46 hour ave fcst:
3:376183:d=2022081506:PMTF:1 hybrid level:47-70 hour ave fcst:"

degrib2 aqm.t06z.ave_24hr_pm25.793.grib2 | more
" PRODUCT TEMPLATE 4. 8: ( PARAMETER = PMTF 0 13 193 ) 13 193 2 0 89 0 0 1 -2147483647 105 0 1 255 -127 -2147483647 2022 8 16 4 0 0 1 0 0
2 1 23 255 0
FIELD: PMTF 1 hybrid lvl (2147483647 -) valid 2147483647 hour before 2022081506:00:00 to 2022081604:00:0
"
The encoding with issues is highlighted above when using wgrib2 and degrib2 to display the file.

Thanks,

Kai

[KaiWang-NOAA]

@edwardhartnett, just curious if there's a timeline on when this issue could be fixed? Thanks!
@JianpingHuang-NOAA @HaixiaLiu-NOAA

[edwardhartnett]

I cannot get to the sample file on acorn and need it moved to hera... Thanks, Ed

…

On Thu, Nov 17, 2022 at 2:35 PM KaiWang-NOAA ***@***.***> wrote: @edwardhartnett <https://github.com/edwardhartnett>, just curious if there's a timeline on when this issue could be fixed? Thanks! @JianpingHuang-NOAA <https://github.com/JianpingHuang-NOAA> @HaixiaLiu-NOAA <https://github.com/HaixiaLiu-NOAA> — Reply to this email directly, view it on GitHub <#175 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJIOMMCXIT37QKLVDHWHVJ3WI2QK5ANCNFSM6AAAAAARWM4WFA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[edwardhartnett]

Can someone please put this sample file on hera somewhere so I can get to it?

[KaiWang-NOAA]

I just put a set of sample files in /scratch2/NCEPDEV/naqfc/Kai.Wang/sample_grib2 on Hera. Thanks!

[edwardhartnett]

OK, I have released a new version of NCEPLIBS-g2, version 2.4.6, which solved the negative forecast time issue.

I will soon do a new release of grib_util to work with the new g2 release...

[KaiWang-NOAA]

Hi Ed, Thank you for the update! Looking forward to the new release of grib_util.