RAiDER build the data cube failed when the HRES weather model contains NaNs [BUG]

[xhuang-jpl]

Describe the bug
When the HRES weather model contains NaN values, the RAiDER bulids the data cube failed with errors shown below,

Output SNWE: [1820916.1440531414, 2130260.971614293, 203886.3636198498, 541353.448232015]
Output cube spacing: 7030.564262753442
ERROR: Weather model scratch/weather_model_files/HRES_2023_06_19_T00_00_00_17N_20N_79E_82E.nc failed, may contain NaNs
Traceback (most recent call last):
  File "/scratch/xhuang/python/miniconda3/envs/isce3/lib/python3.11/site-packages/RAiDER/delay.py", line 147, in _get_delays_on_cube
    ifWet, ifHydro = getInterpolators(weather_model_file, "total")
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/xhuang/python/miniconda3/envs/isce3/lib/python3.11/site-packages/RAiDER/delayFcns.py", line 36, in getInterpolators
    raise RuntimeError('Weather model {} contains NaNs'.format(wm_file))
RuntimeError: Weather model scratch/weather_model_files/HRES_2023_06_19_T00_00_00_17N_20N_79E_82E.nc contains NaNs
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/scratch/xhuang/tools/isce/install/packages/nisar/workflows/troposphere.py", line 419, in <module>
    run(tropo_runcfg.cfg, gunw_hdf5 = out_paths['GUNW'])
  File "/scratch/xhuang/tools/isce/install/packages/nisar/workflows/troposphere.py", line 401, in run
    tropo_delay_datacube = compute_troposphere_delay(cfg, gunw_hdf5)
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/xhuang/tools/isce/install/packages/nisar/workflows/troposphere.py", line 235, in compute_troposphere_delay
    tropo_delay_reference, _ = raider_tropo_delay(dt=acquisition_time_ref,
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/xhuang/python/miniconda3/envs/isce3/lib/python3.11/site-packages/RAiDER/delay.py", line 84, in tropo_delay
    ds = _get_delays_on_cube(dt, weather_model_file, wm_proj, aoi.bounds(), height_levels,
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/scratch/xhuang/python/miniconda3/envs/isce3/lib/python3.11/site-packages/RAiDER/delay.py", line 156, in _get_delays_on_cube
    [ifWet, ifHydro])
     ^^^^^
UnboundLocalError: cannot access local variable 'ifWet' where it is not associated with a value

I have tested on my side, if the NaNs are interpolated, this issue is gone. Here is the simple code I run to interpolate the NaNs if it is helpful.

         def _interpolate_NaNs_in_raider_NetCDF(weather_model_file):
                '''
                Interpolate the  NaNs in the RAiDER NetCDF

                Parameters
                ----------
                 weather_model_file: str
                    HRES NetCDF weather model file
                Returns
                ----------
                     the path of the RAiDER internal NetCDF file with the NaNs interpolated
                 '''
                ds = rioxr.open_rasterio(weather_model_file)
                # Interpolate NaN values using linear method
                ds_interp = ds.rio.interpolate_na()
                # Sort the dataset in ascending order following the order conventions of the RAiDER NetCDF
                for dim in ['x', 'y', 'z']:
                    ds_interp = ds_interp.sortby(dim)
                ds.close()
                ds_interp.to_netcdf(weather_model_file)
                ds_interp.close()
                return weather_model_file

To Reproduce
Steps to reproduce the behavior:

1. I can share with you HRES weather model containing NaNs if you necessary.
2. the tropo_delay from RAiDER.delay is used
3. RuntimeError: Weather model scratch/weather_model_files/HRES_2023_06_19_T00_00_00_17N_20N_79E_82E.nc contains NaNs
   Traceback (most recent call last):

Expected behavior
A clear and concise description of what you expected to happen.

Screenshots
If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

• RAiDER 0.4.2
• OS

[bbuzz31]

This should be fixed in the current Release 0.4.5. Unfortunately you'll have to install from GitHub as the latest releases aren't available on conda

[bbuzz31]

Although, I don't think HRES should contain Nans

[xhuang-jpl]

Thanks @bbuzz31, I checked the ECMWF HRES NetCDF data, it indeed has the NaN values as shown below,

[hfattahi]

@bbuzz31 would you please clarify what the latest release does when there is NAN in the input data?

[bbuzz31]

@hfattahi NANS in the input file no longer raise an error during RAiDER and are propagated such that the final delays are NAN at those locations. However, this should only happen for regional models (like HRRR) when you specify an AOI that is not covered by the model domain. So I wouldn't expect this in global HRES.

@xhuang-jpl I ran RAiDER with a bounding box based on yours (SNWE=[17, 20, 79, 82]) on the same date and get weather model files without NAN... I suspect you have an old version of the actual weather model file?
You could re-download HRES using RAiDER from the command line. Alternatively I can send you the weather files I just made, but I'm on conference travel and my response will be delayed.

[xhuang-jpl]

Thanks @bbuzz31 , I have no account to download the HRES from the RAiDER, can you share with me your downloaded file?
The NetCDF we use is created by the grid_to_neccdf tools from the ECWMF not directly downloaded from the ECWMF.

[hfattahi]

@dbekaert @jhkennedy this issue is blocking a delivery on NISAR SDS. The flow is not to install from source. Would you please push this release to conda? How long do you think it will take to have this release on conda?

[jhkennedy]

@hfattahi I've been working on the v0.4.5 conda-forge release this week. You can follow this PR to see when it's merged and available on conda:
conda-forge/raider-feedstock#12

Should be this afternoon.

[hfattahi]

Thank you @jhkennedy. We will follow the PR on conda-forge.

[jhkennedy]

@hfattahi v0.4.5 PR was merged and should be available on conda in the next couple of hours once the CDN refreshes

[xhuang-jpl]

Thank you @jhkennedy ! I will install and test it when it is ready.

[xhuang-jpl]

Hi @jhkennedy and @bbuzz31, when I convert the ECMWF NetCDF (white dots are NaNs) to RAiDER internal NetCDF, the NaNs seems to be replaced with 0 as shown below, which may be because of this function? Thanks.

RAiDER/tools/RAiDER/models/weatherModel.py

Line 265 in e9cac85

self._checkForNans()

[xhuang-jpl]

Would it be possible in the next release, the user can pass their own named weather model file when this function (prepareWeatherModel) is used? Thank you

https://github.com/dbekaert/RAiDER/blob/4b7f4a1139b3a44ae4ece2bbd3f6d070d0a1f143/tools/RAiDER/processWM.py#L20C4-L20C4