Convert gives unexpected results with python embedding in Grid-Stat

[CPKalb]

Hi:

I'm reading data into Grid-Stat using python embedding and then performing a units conversion using convert. In this case, I'm not performing the conversion in the python embedding script because the script is used to read other fields that do not need to be converted. However, I am getting some unexpected results in Grid-Stat that are inconsistent with the results from plot_data_plane.

I've attached 2 images below. These show the output from plot_data_plane using python embedding. The first is the original data, and the second is with the units converted, using the commands listed. The results are as expected.
plot_data_plane PYTHON_NUMPY testMaxcth_orig.ps 'name="read_extrap_byte_netcdf.py /data1/work/aspire/mdvNc/radar/CTH/20210917/050000.mdv.cf.nc MaxCTH kFT obs";'

plot_data_plane PYTHON_NUMPY testMaxcth_units.ps 'name="read_extrap_byte_netcdf.py /data1/work/aspire/mdvNc/radar/CTH/20210917/050000.mdv.cf.nc MaxCTH kFT obs"; convert(x) = x*0.0032808399;'

However, the next 2 images show the output from Grid-Stat. These are the plots done on the netCDF output file from Grid-Stat. In this case, the data is put on a coarser and smaller grid, so some differences are unexpected. The first image is without using convert in Grid-Stat, and the data is consistent with the first plot above. However, the second shows the output from Grid-Stat with convert. In this case, the data values are orders of magnitude smaller.

It almost looks like the conversion is happening twice, which I know was an issue previously discussed (convert being performed twice with regridding). This is MET version 11.0.1. Is there a specific setting that's needed in METplus to ensure the conversion is not performed twice?

Thanks!
Tina

[DanielAdriaansen]

Hi @CPKalb. Do you have MET_PYTHON_EXE set or not? Can you point me to your Python script somewhere?

[CPKalb]

Hi Dan:

I do not have MET_PYTHON_EXE set. The python script is on aoawsru1, but I can move it somewhere else. I'm wondering if this is related to this other issue, but I thought that had been resolved.

Tina

[JohnHalleyGotway]

Tina, it looks to like that conversion operation is being applied twice. Checking the max value from the colorbar of the Grid-Stat output...
11315.3 * 0.0032808399 = 37.123
37.123 * 0.0032808399 = 0.121796

And this same issue was noted in discussion #2033 and addressed in issue dtcenter/MET#2437.

This doesn't provide a complete answer to your question, but is just to say that this isn't the first time it's come up. We'll need to look very closely at the context in which your conversion function is defined.

What version of MET are you using? 11.0.0 or 11.0.2 which would include the bugfix for issue dtcenter/MET#2437.

[CPKalb]

I'm posting the METplus configuration files here. Should I put a sample model/obs file somewhere for testing?

The command to run is the following:
run_metplus.py -c GridStat_StatAnalysis.conf -c CTREC_leads.conf -c CTREC_MaxCTH.conf -c obs_MaxCTH.conf -c system.conf

GridStat_StatAnalysis.conf.txt
CTREC_leads.conf.txt
CTREC_MaxCTH.conf.txt
obs_MaxCTH.conf.txt
read_extrap_byte_netcdf.py.txt

Tina

[JohnHalleyGotway]

Tina, no that's OK. I was able to use a python embedding with Grid-Stat to replicate this issue by including convert(x) within the fcst variable setting. I'll look more closely into this an do think it'll require a bugfix in MET.

Checking the log output, I do see it being applied twice:

DEBUG 3: Running user's python script (read_CPC_binary).
DEBUG 3: Applying conversion function.
...
DEBUG 3: Reading forecast data for UNKNOWN/Surface.
DEBUG 3: Applying conversion function.

[CPKalb]

Ah, yes. It is being applied twice.

Tina

[JohnHalleyGotway]

Good news, I found the bug in src/libcode/vx_data2d_python/data2d_python.cc.

MET runs the python script once and stores the result in a DataPlane object. Future calls to retrieve data from that python script just serve up the data it reads the first time. The bug is that the conversion and censoring logic are being applied to that stored version. And Grid-Stat calls that python script twice, once to get the grid definition and once to get the data. So the conversion function was being applied to that stored version 2 times.

The fix is to leave the data read from the python script as-is. Apply the conversion and censoring logic to a copy of the DataPlane object rather than modifying it in place.

This is a bug in the python library code that would affect many MET tools when the convert or censoring operations are used with python embedding.

[JohnHalleyGotway]

FYI, I'm fixing this via bugfix issue dtcenter/MET#2575 and will include this in an 11.0.3 bugfix release, along with the upcoming 11.1.0 release.

[CPKalb]

Fantastic! Feel free to add me as a reviewer when the bugfix is ready!

Thanks,
Tina