[BUG] geo2rdr fails to converge on Track 106

[cmarshak]

On track 106, where each input granule in ref is being given different orbits, our application of geo2rdr in ISCE2 and ISCE3 is having issues (in ISCE3, a failure to converge).

Here are some sample inputs and logs:

{'job_id': '804dd1ef-9c7f-4635-8056-8d3a1bb1f28c',
  'job_type': 'INSAR_ISCE_TEST',
  'request_time': '2023-08-25T03:36:55+00:00',
  'status_code': 'FAILED',
  'user_id': 'access_cloud_based_insar',
  'name': 'track106-n3',
  'job_parameters': {'compute_solid_earth_tide': True,
   'estimate_ionosphere_delay': True,
   'frame_id': 16421,
   'granules': ['S1A_IW_SLC__1SDV_20230323T225950_20230323T230017_047778_05BD70_1E8E',
    'S1A_IW_SLC__1SDV_20230323T230015_20230323T230042_047778_05BD70_0772'],
   'secondary_granules': ['S1A_IW_SLC__1SDV_20230215T225950_20230215T230017_047253_05ABAA_15E6',
    'S1A_IW_SLC__1SDV_20230215T230015_20230215T230042_047253_05ABAA_5261'],
   'weather_model': 'HRRR'},
  'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/804dd1ef-9c7f-4635-8056-8d3a1bb1f28c/804dd1ef-9c7f-4635-8056-8d3a1bb1f28c.log'],
  'expiration_time': '2023-10-25T00:00:00+00:00',
  'processing_times': [8909.324, 372.572]}

The link to the relevant input GUNW is here.

Please see ACCESS-Cloud-Based-InSAR/DockerizedTopsApp#148 for more details for this highly related problem.

I suspect that deduplication of orbit files via builtin set is acting differently for each step. As we show in previous notebook, one of the orbit files is able to successfully complete an azimuth time grid generation, the other not.

We will investigate this further on Monday. It amounts to figuring out how to filter out orbit files for the application of our azimuth time grid generation. We will have to discuss with individuals linked on related issue ticket. Or maybe it requires better initialization of the orbit object in ISCE.

[piyushrpt]

Likely related to midnight crossing.

[cmarshak]

It appears the fix for ensuring success of geo2rdr for azimuth time interpolation worked in ISCE2 but not ISCE3 (and therefore the raider step). All the failures are concentrated track 106 and track 91 and are related to 835 SLCs.

Here is log of output

k3 = 3750.0
Humidity type = q
=====================================
Class name: hrrr
Dataset: hrrr
=====================================
A: []
B: []
Number of points in Lon/Lat = 113/143
Total number of grid points (3D): 921063
=====================================
Extent of the weather model is (xmin, ymin, xmax, ymax):-79.48, 40.65, -73.50, 44.45
Extent of the input is (xmin, ymin, xmax, ymax): -78.05, 41.60, -74.40, 43.55
Weather model HRRR is available from 2016-07-15 to Present
Weather model HRRR is available from 2016-07-15 to Present
✅ Found ┊ model=hrrr ┊ �[3mproduct=nat�[0m ┊ �[38;2;41;130;13m2020-Feb-08 00:00 UTC�[92m F00�[0m ┊ �[38;2;255;153;0m�[3mGRIB2 @ aws�[0m ┊ �[38;2;255;153;0m�[3mIDX @ aws�[0m
/opt/conda/envs/RAiDER/lib/python3.10/site-packages/herbie/core.py:797: UserWarning: This pattern is interpreted as a regular expression, and has match groups. To actually get the groups, use str.extract.
  logic = df.search_this.str.contains(searchString)
👨��� Created directory: [/home/raider/weather_files/hrrr/20200208]
Note: Returning a list of [12] xarray.Datasets because cfgrib opened with multiple hypercubes.
Number of weather model nodes: 921063
Shape of weather model: (113, 143, 57)
Bounds of the weather model: 479693.85/815693.85/1493479.86/1919479.86 (SNWE)
Weather model: HRRR
Mean value of the wet refractivity: 8.754181
Mean value of the hydrostatic refractivity: 184.977432
======Weather Model class object=====
Weather model time: 2020-02-08 00:00:00
Latitude resolution: 0.02702702702702703
Longitude resolution: 0.02702702702702703
Native projection: PROJCRS["unknown",BASEGEOGCRS["unknown",DATUM["unknown",ELLIPSOID["unknown",6371229,0,LENGTHUNIT["metre",1,ID["EPSG",9001]]]],PRIMEM["Greenwich",0,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8901]]],CONVERSION["unknown",METHOD["Lambert Conic Conformal (2SP)",ID["EPSG",9802]],PARAMETER["Latitude of false origin",38.5,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8821]],PARAMETER["Longitude of false origin",262.5,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8822]],PARAMETER["Latitude of 1st standard parallel",38.5,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8823]],PARAMETER["Latitude of 2nd standard parallel",38.5,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8824]],PARAMETER["Easting at false origin",0,LENGTHUNIT["metre",1],ID["EPSG",8826]],PARAMETER["Northing at false origin",0,LENGTHUNIT["metre",1],ID["EPSG",8827]]],CS[Cartesian,2],AXIS["(E)",east,ORDER[1],LENGTHUNIT["metre",1,ID["EPSG",9001]]],AXIS["(N)",north,ORDER[2],LENGTHUNIT["metre",1,ID["EPSG",9001]]]]
ZMIN: -100.0
ZMAX: 26000.0
k1 = 0.776
k2 = 0.233
k3 = 3750.0
Humidity type = q
=====================================
Class name: hrrr
Dataset: hrrr
=====================================
A: []
B: []
Number of points in Lon/Lat = 113/143
Total number of grid points (3D): 921063
=====================================
Extent of the weather model is (xmin, ymin, xmax, ymax):-79.48, 40.65, -73.50, 44.45
Extent of the input is (xmin, ymin, xmax, ymax): -78.05, 41.60, -74.40, 43.55
Weather model HRRR is available from 2016-07-15 to Present
Weather model HRRR is available from 2016-07-15 to Present
✅ Found ┊ model=hrrr ┊ �[3mproduct=nat�[0m ┊ �[38;2;41;130;13m2020-Feb-07 22:00 UTC�[92m F00�[0m ┊ �[38;2;255;153;0m�[3mGRIB2 @ aws�[0m ┊ �[38;2;255;153;0m�[3mIDX @ aws�[0m
/opt/conda/envs/RAiDER/lib/python3.10/site-packages/herbie/core.py:797: UserWarning: This pattern is interpreted as a regular expression, and has match groups. To actually get the groups, use str.extract.
  logic = df.search_this.str.contains(searchString)
Note: Returning a list of [12] xarray.Datasets because cfgrib opened with multiple hypercubes.
Number of weather model nodes: 921063
Shape of weather model: (113, 143, 57)
Bounds of the weather model: 479693.85/815693.85/1493479.86/1919479.86 (SNWE)
Weather model: HRRR
Mean value of the wet refractivity: 9.284787
Mean value of the hydrostatic refractivity: 183.895157
======Weather Model class object=====
Weather model time: 2020-02-07 22:00:00
Latitude resolution: 0.02702702702702703
Longitude resolution: 0.02702702702702703
Native projection: PROJCRS["unknown",BASEGEOGCRS["unknown",DATUM["unknown",ELLIPSOID["unknown",6371229,0,LENGTHUNIT["metre",1,ID["EPSG",9001]]]],PRIMEM["Greenwich",0,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8901]]],CONVERSION["unknown",METHOD["Lambert Conic Conformal (2SP)",ID["EPSG",9802]],PARAMETER["Latitude of false origin",38.5,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8821]],PARAMETER["Longitude of false origin",262.5,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8822]],PARAMETER["Latitude of 1st standard parallel",38.5,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8823]],PARAMETER["Latitude of 2nd standard parallel",38.5,ANGLEUNIT["degree",0.0174532925199433],ID["EPSG",8824]],PARAMETER["Easting at false origin",0,LENGTHUNIT["metre",1],ID["EPSG",8826]],PARAMETER["Northing at false origin",0,LENGTHUNIT["metre",1],ID["EPSG",8827]]],CS[Cartesian,2],AXIS["(E)",east,ORDER[1],LENGTHUNIT["metre",1,ID["EPSG",9001]]],AXIS["(N)",north,ORDER[2],LENGTHUNIT["metre",1,ID["EPSG",9001]]]]
ZMIN: -100.0
ZMAX: 26000.0
k1 = 0.776
k2 = 0.233
k3 = 3750.0
Humidity type = q
=====================================
Class name: hrrr
Dataset: hrrr
=====================================
A: []
B: []
Number of points in Lon/Lat = 113/143
Total number of grid points (3D): 921063
=====================================
Extent of the weather model is (xmin, ymin, xmax, ymax):-79.48, 40.65, -73.50, 44.45
Extent of the input is (xmin, ymin, xmax, ymax): -78.05, 41.60, -74.40, 43.55
Traceback (most recent call last):
  File "/opt/conda/envs/RAiDER/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/conda/envs/RAiDER/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/conda/envs/RAiDER/lib/python3.10/site-packages/RAiDER/cli/__main__.py", line 46, in <module>
    main()
  File "/opt/conda/envs/RAiDER/lib/python3.10/site-packages/RAiDER/cli/__main__.py", line 42, in main
    process_entry_point.load()()
  File "/opt/conda/envs/RAiDER/lib/python3.10/site-packages/RAiDER/cli/raider.py", line 620, in calcDelaysGUNW
    cube_filenames = calcDelays([path_cfg])
  File "/opt/conda/envs/RAiDER/lib/python3.10/site-packages/RAiDER/cli/raider.py", line 364, in calcDelays
    time_grid = get_s1_azimuth_time_grid(lon,
  File "/opt/conda/envs/RAiDER/lib/python3.10/site-packages/RAiDER/s1_azimuth_timing.py", line 165, in get_s1_azimuth_time_grid
    az_arr = get_azimuth_time_grid(lon_mesh, lat_mesh, hgt_mesh, orb)
  File "/opt/conda/envs/RAiDER/lib/python3.10/site-packages/RAiDER/s1_azimuth_timing.py", line 96, in get_azimuth_time_grid
    aztime, sr = isce.geometry.geo2rdr(
RuntimeError: geo2rdr failed to converge

Below is a sample of 10 input granules:

{'expiration_time': '2023-10-30T00:00:00+00:00',
 'job_id': '0850e26b-61a8-439e-81c3-2cba6c5519d8',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 16421,
                    'granules': ['S1A_IW_SLC__1SDV_20200207T225933_20200207T230000_031153_0394CE_BFC6',
                                 'S1A_IW_SLC__1SDV_20200207T225958_20200207T230025_031153_0394CE_C996'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20190131T225927_20190131T225954_025728_02DC14_487D',
                                           'S1A_IW_SLC__1SDV_20190131T225952_20190131T230019_025728_02DC14_E783'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/0850e26b-61a8-439e-81c3-2cba6c5519d8/0850e26b-61a8-439e-81c3-2cba6c5519d8.log'],
 'name': 'geo2rdr_2023-08-30',
 'processing_times': [9481.409, 376.915],
 'request_time': '2023-08-30T19:53:38+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'}
{'expiration_time': '2023-10-30T00:00:00+00:00',
 'job_id': '1636c7a9-1473-4240-be60-616f116b1609',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 16422,
                    'granules': ['S1A_IW_SLC__1SDV_20181003T225955_20181003T230022_023978_029E80_3F17',
                                 'S1A_IW_SLC__1SDV_20181003T230019_20181003T230057_023978_029E80_D8F2'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20170222T225922_20170222T225950_015403_019475_4049',
                                           'S1A_IW_SLC__1SDV_20170222T225948_20170222T230015_015403_019475_5AE4'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/1636c7a9-1473-4240-be60-616f116b1609/1636c7a9-1473-4240-be60-616f116b1609.log'],
 'name': 'geo2rdr_2023-08-30',
 'processing_times': [10520.87, 359.522],
 'request_time': '2023-08-30T19:53:38+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'}
{'expiration_time': '2023-10-30T00:00:00+00:00',
 'job_id': '1781e249-3c54-4ff6-9854-c7aebd94e299',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 16420,
                    'granules': ['S1A_IW_SLC__1SDV_20220115T225921_20220115T225948_041478_04EEAA_2E0D',
                                 'S1A_IW_SLC__1SDV_20220115T225946_20220115T230013_041478_04EEAA_FA55'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20201215T225917_20201215T225944_035703_042D7B_A0B7',
                                           'S1A_IW_SLC__1SDV_20201215T225942_20201215T230009_035703_042D7B_49EB'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/1781e249-3c54-4ff6-9854-c7aebd94e299/1781e249-3c54-4ff6-9854-c7aebd94e299.log'],
 'name': 'geo2rdr_2023-08-30',
 'processing_times': [9071.726, 371.337],
 'request_time': '2023-08-30T19:53:38+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'}
{'expiration_time': '2023-10-30T00:00:00+00:00',
 'job_id': '969c77eb-3f42-43db-943b-52853c1526cc',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 16421,
                    'granules': ['S1A_IW_SLC__1SDV_20210213T225939_20210213T230006_036578_044BED_28D0',
                                 'S1A_IW_SLC__1SDV_20210213T230004_20210213T230031_036578_044BED_6B34'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20200207T225933_20200207T230000_031153_0394CE_BFC6',
                                           'S1A_IW_SLC__1SDV_20200207T225958_20200207T230025_031153_0394CE_C996'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/969c77eb-3f42-43db-943b-52853c1526cc/969c77eb-3f42-43db-943b-52853c1526cc.log'],
 'name': 'geo2rdr_2023-08-30',
 'processing_times': [9272.986, 371.098],
 'request_time': '2023-08-30T19:53:38+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'}
{'expiration_time': '2023-10-30T00:00:00+00:00',
 'job_id': '42ebc658-502a-400c-b14f-b3e48d9ec36d',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 16420,
                    'granules': ['S1A_IW_SLC__1SDV_20220328T225920_20220328T225947_042528_05129B_50E0',
                                 'S1A_IW_SLC__1SDV_20220328T225945_20220328T230012_042528_05129B_FFDF'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20210225T225914_20210225T225941_036753_045200_8DB1',
                                           'S1A_IW_SLC__1SDV_20210225T225939_20210225T230006_036753_045200_E26F'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/42ebc658-502a-400c-b14f-b3e48d9ec36d/42ebc658-502a-400c-b14f-b3e48d9ec36d.log'],
 'name': 'geo2rdr_2023-08-30',
 'processing_times': [9388.022, 349.908],
 'request_time': '2023-08-30T19:53:38+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'}
{'expiration_time': '2023-10-30T00:00:00+00:00',
 'job_id': '1bf2d968-bcdb-474c-8d00-ce3934527281',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 16421,
                    'granules': ['S1A_IW_SLC__1SDV_20171101T225924_20171101T225951_019078_020453_154F',
                                 'S1A_IW_SLC__1SDV_20171101T225948_20171101T230015_019078_020453_856C'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20161025T225926_20161025T225953_013653_015E31_3684',
                                           'S1A_IW_SLC__1SDV_20161025T225951_20161025T230018_013653_015E31_AE0D'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/1bf2d968-bcdb-474c-8d00-ce3934527281/1bf2d968-bcdb-474c-8d00-ce3934527281.log'],
 'name': 'geo2rdr_2023-08-30',
 'processing_times': [10218.788, 354.964],
 'request_time': '2023-08-30T19:53:38+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'}
{'expiration_time': '2023-10-30T00:00:00+00:00',
 'job_id': 'e8181351-cf31-4582-8a15-a1f45b8fab69',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 16421,
                    'granules': ['S1A_IW_SLC__1SDV_20181214T225929_20181214T225956_025028_02C2B7_D9FF',
                                 'S1A_IW_SLC__1SDV_20181214T225954_20181214T230021_025028_02C2B7_18A4'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20171113T225923_20171113T225950_019253_0209C1_D661',
                                           'S1A_IW_SLC__1SDV_20171113T225948_20171113T230015_019253_0209C1_7488'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/e8181351-cf31-4582-8a15-a1f45b8fab69/e8181351-cf31-4582-8a15-a1f45b8fab69.log'],
 'name': 'geo2rdr_2023-08-30',
 'processing_times': [8809.812, 366.195],
 'request_time': '2023-08-30T19:53:38+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'}
{'expiration_time': '2023-10-30T00:00:00+00:00',
 'job_id': '2fe03b13-d2c9-4174-a9f8-2a9e427bc746',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 16421,
                    'granules': ['S1A_IW_SLC__1SDV_20190119T225928_20190119T225955_025553_02D5AF_2B9E',
                                 'S1A_IW_SLC__1SDV_20190119T225953_20190119T230019_025553_02D5AF_E761'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20171231T225922_20171231T225949_019953_021FA1_46E6',
                                           'S1A_IW_SLC__1SDV_20171231T225946_20171231T230013_019953_021FA1_83A0'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/2fe03b13-d2c9-4174-a9f8-2a9e427bc746/2fe03b13-d2c9-4174-a9f8-2a9e427bc746.log'],
 'name': 'geo2rdr_2023-08-30',
 'processing_times': [9249.954, 355.201],
 'request_time': '2023-08-30T19:53:38+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'}
{'expiration_time': '2023-10-30T00:00:00+00:00',
 'job_id': '5cc77c84-e01f-4a45-b19b-ee76cf9e69d3',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 16420,
                    'granules': ['S1A_IW_SLC__1SDV_20220127T225920_20220127T225947_041653_04F48D_A9D5',
                                 'S1A_IW_SLC__1SDV_20220127T225945_20220127T230012_041653_04F48D_EDA8'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20201227T225916_20201227T225943_035878_04338F_D28F',
                                           'S1A_IW_SLC__1SDV_20201227T225941_20201227T230008_035878_04338F_17A0'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/5cc77c84-e01f-4a45-b19b-ee76cf9e69d3/5cc77c84-e01f-4a45-b19b-ee76cf9e69d3.log'],
 'name': 'geo2rdr_2023-08-30',
 'processing_times': [9205.364, 373.795],
 'request_time': '2023-08-30T19:53:38+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'}
{'expiration_time': '2023-10-30T00:00:00+00:00',
 'job_id': '5b7ad51d-89a3-4fea-8abf-de23a060c39c',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 16420,
                    'granules': ['S1A_IW_SLC__1SDV_20230323T225926_20230323T225953_047778_05BD70_2095',
                                 'S1A_IW_SLC__1SDV_20230323T225950_20230323T230017_047778_05BD70_1E8E'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20220316T225920_20220316T225947_042353_050CA8_C57F',
                                           'S1A_IW_SLC__1SDV_20220316T225944_20220316T230011_042353_050CA8_6839'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/5b7ad51d-89a3-4fea-8abf-de23a060c39c/5b7ad51d-89a3-4fea-8abf-de23a060c39c.log'],
 'name': 'geo2rdr_2023-08-30',
 'processing_times': [9483.706, 365.466],
 'request_time': '2023-08-30T19:53:38+00:00',
 'status_code': 'FAILED',
 'user_id': 'access_cloud_based_insar'}

1,975 jobs failed related to this.

[cmarshak]

Just double checked these job failures - all are on track 106. Some that were previously grouped in this were on track 91 but all these succeeded. For reference, here are a few.

{'browse_images': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/15cd8b3f-d3fe-442d-886d-3aa8afc003dd/S1-GUNW-A-R-091-tops-20201202_20201027-221941-00069W_00045N-PP-c570-v3_0_0.png'],
 'expiration_time': '2023-10-30T00:00:00+00:00',
 'files': [{'filename': 'S1-GUNW-A-R-091-tops-20201202_20201027-221941-00069W_00045N-PP-c570-v3_0_0.nc',
            's3': {'bucket': 'hyp3-tibet-jpl-contentbucket-81rn23hp7ppf',
                   'key': '15cd8b3f-d3fe-442d-886d-3aa8afc003dd/S1-GUNW-A-R-091-tops-20201202_20201027-221941-00069W_00045N-PP-c570-v3_0_0.nc'},
            'size': 108152775,
            'url': 'https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/15cd8b3f-d3fe-442d-886d-3aa8afc003dd/S1-GUNW-A-R-091-tops-20201202_20201027-221941-00069W_00045N-PP-c570-v3_0_0.nc'}],
 'job_id': '15cd8b3f-d3fe-442d-886d-3aa8afc003dd',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 14061,
                    'granules': ['S1A_IW_SLC__1SDV_20201202T221914_20201202T221943_035513_0426F0_F52D',
                                 'S1A_IW_SLC__1SDV_20201202T221941_20201202T222009_035513_0426F0_EFA7'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20201027T221915_20201027T221944_034988_0414C3_DA43',
                                           'S1A_IW_SLC__1SDV_20201027T221942_20201027T222010_034988_0414C3_36ED'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': [],
 'name': 'geo2rdr_2023-08-30',
 'processing_times': [9074.64, 770.832],
 'request_time': '2023-08-30T19:53:38+00:00',
 'status_code': 'SUCCEEDED',
 'thumbnail_images': [],
 'user_id': 'access_cloud_based_insar'}
{'browse_images': ['https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/f6d0917d-8aed-42b3-94da-fcacfffdea67/S1-GUNW-A-R-091-tops-20190319_20180324-221952-00069W_00046N-PP-249d-v3_0_0.png'],
 'expiration_time': '2023-10-30T00:00:00+00:00',
 'files': [{'filename': 'S1-GUNW-A-R-091-tops-20190319_20180324-221952-00069W_00046N-PP-249d-v3_0_0.nc',
            's3': {'bucket': 'hyp3-tibet-jpl-contentbucket-81rn23hp7ppf',
                   'key': 'f6d0917d-8aed-42b3-94da-fcacfffdea67/S1-GUNW-A-R-091-tops-20190319_20180324-221952-00069W_00046N-PP-249d-v3_0_0.nc'},
            'size': 113268119,
            'url': 'https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/f6d0917d-8aed-42b3-94da-fcacfffdea67/S1-GUNW-A-R-091-tops-20190319_20180324-221952-00069W_00046N-PP-249d-v3_0_0.nc'}],
 'job_id': 'f6d0917d-8aed-42b3-94da-fcacfffdea67',
 'job_parameters': {'compute_solid_earth_tide': True,
                    'estimate_ionosphere_delay': True,
                    'frame_id': 14062,
                    'granules': ['S1A_IW_SLC__1SDV_20190319T221926_20190319T221954_026413_02F4BE_671A',
                                 'S1A_IW_SLC__1SDV_20190319T221952_20190319T222019_026413_02F4BE_8A92'],
                    'secondary_granules': ['S1A_IW_SLC__1SDV_20180324T221920_20180324T221948_021163_024616_EE85',
                                           'S1A_IW_SLC__1SDV_20180324T221946_20180324T222013_021163_024616_D62D'],
                    'weather_model': 'HRRR'},
 'job_type': 'INSAR_ISCE',
 'logs': [],
 'name': 'geo2rdr_2023-08-30',
 'processing_times': [8981.085, 764.475],
 'request_time': '2023-08-30T19:53:38+00:00',
 'status_code': 'SUCCEEDED',
 'thumbnail_images': [],
 'user_id': 'access_cloud_based_insar'}

[cmarshak]

Given the wonkiness of Track 106 - will also want to ensure the convergence on these jobs that never got to RAiDER: ACCESS-Cloud-Based-InSAR/DockerizedTopsApp#152

[cmarshak]

@jhkennedy - I am struggling with this; fairly confident it's an environment/build issue along with isce3. Would love your input on how to proceed because 😵‍💫.

Here is a sample product link (see the last of the most recent batch above): https://hyp3-tibet-jpl-contentbucket-81rn23hp7ppf.s3.us-west-2.amazonaws.com/5b7ad51d-89a3-4fea-8abf-de23a060c39c/S1-GUNW-A-R-106-tops-20230323_20220316-225951-00078W_00041N-PP-3769-v3_0_0.nc

The relevant command I am using is:

raider.py ++process calcDelaysGUNW -f S1-GUNW-A-R-106-tops-20230323_20220316-225951-00078W_00041N-PP-3769-v3_0_0.nc -m HRRR -interp azimuth_time_grid

Here are two different scenarios (maybe 2.5 as you will see) that I tried:

1. Use latest docker container and are able to regenerate geo2rdr error documented above
   • docker run -it -v ${PWD}:/home/raider/work --rm --entrypoint /bin/bash ghcr.io/dbekaert/raider:latest
   • cd work
   • conda activate RAiDER
   • Command above
2. Use rosetta on ARM64 mac and the job completes successfully. New install of RAiDER using pip install -e .

Here are the environments.

env-conda-docker.txt
env_conda_arm64.txt

Upon first inspection, I see that our container usesisce3=0.14.0 where as on my mac, i have a minor version bump. So, I installed 0.15.0 in the container hoping that would improve this. However, I get the same error.

[cmarshak]

Furthermore, if I try to install the latest RAiDER on the linux server without docker, I run into dependency heck.

(RAiDER) --- leffe2-cmarshak/RAiDER ‹dev› » pip install .
Processing /mnt/leffe-data2/cmarshak/RAiDER
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Building wheels for collected packages: RAiDER
  Building wheel for RAiDER (pyproject.toml) ... /
done
  Created wheel for RAiDER: filename=RAiDER-0.4.4-cp310-cp310-linux_x86_64.whl size=485864 sha256=b5d7a34746773581aee0468ea426c140fd06e04b2ce5920ee4b4fbfa49a3a9e8
  Stored in directory: /tmp/pip-ephem-wheel-cache-yjfu_9h6/wheels/91/b6/a0/e344f07a068562c639f85f28d93469f2d1d491553d61151382
Successfully built RAiDER
Installing collected packages: RAiDER
  Attempting uninstall: RAiDER
    Found existing installation: RAiDER 0.4.4
    Uninstalling RAiDER-0.4.4:
      Successfully uninstalled RAiDER-0.4.4
Successfully installed RAiDER-0.4.4
(RAiDER) --- leffe2-cmarshak/RAiDER ‹dev› » cd ../recent_crop_of_failed-geo2rdr
(RAiDER) --- leffe2-cmarshak/recent_crop_of_failed-geo2rdr » raider.py ++process calcDelaysGUNW -f S1-GUNW-A-R-106-tops-20230323_20220316-225951-00078W_00041N-PP-3769-v3_0_0.nc -m HRRR -interp azimuth_time_grid

Traceback (most recent call last):
  File "/mnt/leffe-data2/cmarshak/mambaforge/envs/RAiDER/bin/raider.py", line 8, in <module>
    sys.exit(main())
  File "/mnt/leffe-data2/cmarshak/mambaforge/envs/RAiDER/lib/python3.10/site-packages/RAiDER/cli/__main__.py", line 32, in main
    from RAiDER.cli.raider import calcDelays, downloadGNSS, calcDelaysGUNW
  File "/mnt/leffe-data2/cmarshak/mambaforge/envs/RAiDER/lib/python3.10/site-packages/RAiDER/cli/raider.py", line 14, in <module>
    import RAiDER.aria.prepFromGUNW
  File "/mnt/leffe-data2/cmarshak/mambaforge/envs/RAiDER/lib/python3.10/site-packages/RAiDER/aria/prepFromGUNW.py", line 13, in <module>
    import rasterio
  File "/mnt/leffe-data2/cmarshak/mambaforge/envs/RAiDER/lib/python3.10/site-packages/rasterio/__init__.py", line 28, in <module>
    from rasterio._version import gdal_version, get_geos_version, get_proj_version
ImportError: libabsl_cord.so.2301.0.0: cannot open shared object file: No such file or directory

Again, this is reinstalling the environment. Although this is using the dev branch, it should be the same currently (no new merges to dev since the latest release).

conda_env_server.txt

You can see in this particular environment that we have isce3 at 0.15.0. Oy!