Segfaults when verifying ECMWF data

[mollybsmith-noaa]

Hi all,

GSL has recently gotten access to some realtime ECMWF grids, and are trying to integrate them into our global model verification. We plan on verifying the GFS, ECMWF, and the GSL in-house models against both the GFS analysis and the ECMWF analysis using GridStat (we've been already verifying the GFS and GSL in-house models against the GFS analysis for years).

At this point I'm trying to add the step to verify the ECMWF forecasts against the GFS analysis, regridding both to the NCEP G002 grid. However, I keep getting segfaults around the regridding step, and I can't figure out why.

The log for my test case (verbosity set to 10) is on Hera at /scratch2/BMC/amb-verif/global_g2g_verif/g2g_reprocess/ECMWF/realtime/METplus/logs/metplus.log.20231207205422
The model file is at /home/role.amb-verif/GSL_verif-global/verif/g2g_reprocess/ECMWF/realtime/20231205-06z/ecmwf_2023120506_012.grib2
The analysis file is at /home/role.amb-verif/GSL_verif-global/verif/g2g_reprocess/GFS_anl/realtime/20231205-18z/gfs_anl_2023120518_000.grib2
The configuration is at /home/role.amb-verif/GSL_verif-global/verif/g2g_reprocess/ECMWF/realtime/METplus/metplus_final.conf.20231207205422

This is with METplus 5.1.0 and MET 11.1.0. When you have a moment, could you take a look and let me know if you see anything obviously wrong with the way I have set things up?

Thanks!
Molly

[JohnHalleyGotway]

Hi @mollybsmith-noaa, I see you're getting a segfault when processing GRIB2 ECMWF data. Thanks for pointing me to that sample data on hera.

I did pull the ECMWF file you specified and am able to replicate the segfault on my laptop, using only the plot_data_plane tool.

bin/plot_data_plane ecmwf_2023120506_012.grib2 ecmwf_TMP_P1000.ps 'name="TMP"; level="P1000";'
getdrstemplate: DRS Template 5.42 not defined. # 	REPEATED MANY TIMES
FATAL ERROR (SEGFAULT): Process 6865 got signal 11 @ local time = 2023-12-08 19:26:30Z

However, if I first process that file through wgrib2, essentially just reading it in and then back out again, then MET processes it without error:

wgrib2 ecmwf_2023120506_012.grib2 | wgrib2 -i -grib_out ecmwf_2023120506_012_wgrib2.grib2 ecmwf_2023120506_012.grib2
bin/plot_data_plane ecmwf_2023120506_012_wgrib2.grib2 ecmwf_TMP_P1000.ps 'name="TMP"; level="P1000";'
DEBUG 1: Opening data file: ecmwf_2023120506_012_wgrib2.grib2
DEBUG 1: Creating postscript file: ecmwf_TMP_P1000.ps
DEBUG 1: Finish plot_data_plane by johnhg(6088) at 2023-12-08 19:40:00Z

This indicates that there's some issue in how the ECMWF GRIB2 data (in hera:/home/role.amb-verif/GSL_verif-global/verif/g2g_reprocess/ECMWF/realtime/20231205-06z/ecmwf_2023120506_012.grib2) is encoded. It is curious that wgrib2 reads the data without complaint but MET does not. I do note the following warning being printed by the GRIB2C library, to which MET links:

getdrstemplate: DRS Template 5.42 not defined.

Doing some Googling, I strongly suspect it's related to this issue noted for the NCL software:
NCAR/ncl#204
I tested the grib_set step described in this comment. And that also produces the desired effect:

grib_set -r -w packingType=grid_ccsds -s packingType=grid_simple ecmwf_2023120506_012.grib2 ecmwf_2023120506_012_simple.grib2
bin/plot_data_plane ecmwf_2023120506_012_simple.grib2 ecmwf_TMP_P1000.ps 'name="TMP"; level="P1000";'

So preprocessing using wgrib2 or grib_set is a temporary workaround until we can get a direct solution in MET and/or the GRIB2C library to which MET links.

[JohnHalleyGotway]

@mollybsmith-noaa (and tagging @jprestop for her awareness as well), I tracked this issue down a bit further to the NOAA-EMC G2C repository (https://github.com/NOAA-EMC/NCEPLIBS-g2c). Issue NOAA-EMC/NCEPLIBS-g2c#461 from last week identifies this as being a bug in the G2C library.

I'd recommend that we wait for an updated version of G2C, and then test to confirm that this fixes the segfault being thrown by MET.

[mollybsmith-noaa]

@JohnHalleyGotway Interesting! Thank you for taking a look at this, I'll try the fix of running it thought wgrib2 first and see what happens. ecmwf_2023120506_012.grib2 is not the raw file we're getting from the ECMWF, it was missing a couple of variables (RH, total wind, etc) that we wanted, so I'm using pygrib on Hera to generate them. Pygrib must be doing something weird with the encoding.

[mollybsmith-noaa]

@JohnHalleyGotway The wgrib2 workaround was successful, so that solves my immediate problem. I'll stay tuned for the G2C update!

[JohnHalleyGotway]

@mollybsmith-noaa thanks for confirming the workaround was successful. FYI, I've documented this as an issue to upgrade the G2C library version with which MET is linked, once an updated version becomes available:
dtcenter/MET#2759

I'm going to mark this discussion as being answered and recommend that we move any future conversation about it over the comments of that new issue.