HYP3 GUNW Delivery

[bbuzz31]

Moving discussion of tropo correction in GUNW started in #404 here.

Goal: to provide RAiDER with a GUNW[s], calculate the tropospheric delay, and optionally embed it in the product.

Methodology:

1. Provide entry hooks to main cli, e.g., raider.py ++prepGUNWs S1*.nc (ongoing in Setup initial structure for working hooks #432)
2. Extract / calculate necessary RAiDER parameters from GUNW

• for initial release this will write a config file
• subsequent release will store parameters in memory (ongoing in initial attempt at library interface #431)

3. Calculate delays with call to delay.py

• initial release will use config file
• subsequent release will call delay.py as a library

4. Write interferometric delay to disk as netcdf
5. [optionally] Write interferometric delay into GUNW layer

Initial release will be finished by next HYP3 meeting, library prototype dependent on #431. No reason it can't be delivered by Dec 21.

[cmarshak]

So the discussion from #404 moved to here - unsure if this complete or closed. I will tag @jhkennedy so he is aware.

If this really depends on #431, you might want more time. You mentioned that you could deliver on this task by writing a config file, running that config file, and then packaging it as discussed in the issue ticket. You can quickly prototype this and then swap out the "better way", which can be improved later. Please clarify given it's better to have more of a buffer than running behind.

Also, please make the modification to the GUNW the default as that will be what this script is used for in hyp3 cloud processing.

Also, the regex processing seems unnecessary here and let's really hone in on what we want - processing a single gunw. I would add this feature later particularly because hyp3 will not require it and processing is going to happen serially anyways (isce3 is going to want its own cpu). More focused here, the better IMHO.

[bbuzz31]

1. I'm also happy to have more time :D. More of a buffer may actually be good on the first task (the config), as I forgot that RAiDER cannot yet do calculations on the cube.

2. Sure

3. For regex I'm just using glob and iterating over the product[s]. I think this should be left as it'll be easier for users running corrections on products they already have (including us during testing)

[cmarshak]

For more time, communicate clearly to @dbekaert and on this thread - I expect errors to be exposed when hyp3 are used. What I mean to say, build in some additional buffers based on the initial release as you see it now and how it might be slightly modified when it's used in the cloud.

[dbekaert]

Raider does calculations on the cube already in existing release. Its one of the config cases that was released. i.e. support for slant using specific user-set the posting of the grid, the bounds; or the native model extends, and then projection and use of an orbit file.

These are configurable parameters and should not be holding back the prototyping. I would just make the grid configurable as well as the height layers. (I would say we go with native posting of the model in x,y and then set z based on min/max topogrpahy and native model resolution in z)

[cmarshak]

To be clear, the motivation for this plugin is to demonstrate raider at scale. Such a demonstration will just use HRRR over US and Alaska because of its low latency (within aws s3) - so the library exposure is not as important here (yet). When we want to use era5 for example, then there could potentially be more workflow modifications and/or additional step functions utilizing different workflows. This would be to deal with issues with high data latency of other weather models and to lower cloud costs. Regardless, this is very far down the line since we have to see how this works first and would require a considerable amount of coordination (and Joe's/Andrew's expertise). Just want to be extra clear.

[cmarshak]

You cannot overwrite group types in xarray without deleting the group first. See this.

I would use netcdf4 directly as indicated here:

Attributes can be deleted from a netCDF Dataset, Group or Variable using the python del statement (i.e. del grp.foo removes the attribute foo the the group grp).

[cmarshak]

Lastly, one last thing - all the data should be localized in this script, except for the input GUNW.

The GUNW will only have dates of acquisition and will require localization of orbit files at least as indicated here: #381. Unless I am mistaken due to raider updates.

ASF tools has to do this. You mentioned other tools. I like this library: https://github.com/scottstanie/sentineleof.

[dbekaert]

Something we can do in future: Can we check if we have S1 orbit downloader in ISCE3 or on COMPASS repo? ISCE3 is already a dependency now and in future I suggest we use their orbit readers through import statement too.

[cmarshak]

I don't think so - but a developer of Compass maintains the library I suggested.

I would just ask @scottstanie if ISCE3 or compass can be used to download orbit files.

[dbekaert]

I dont like adding many dependancies which we dont know are maintained. We can stick with those used for ISCE or OPERA or NISAR would be better. @scottstanie indicated this repo: https://github.com/opera-adt/s1-reader/blob/main/src/s1reader/s1_orbit.py

[cmarshak]

https://github.com/opera-adt/s1-reader#usage - that's pretty clear how this could be used to download the relevant orbit files from the SLC ids. Nice.

import s1reader

zip_path = "S1A_IW_SLC__1SDV_20190909T134419_20190909T134446_028945_03483B_B9E1.zip"

# read orbits
orbit_dir = '/home/user/data/sentinel1_orbits'
orbit_path = s1reader.get_orbit_file_from_dir(zip_path, orbit_dir, auto_download=True)

[bbuzz31]

we have to build this package from source though

[cmarshak]

This should not be an issue.

[cmarshak]

@bbuzz31 - what's your update on a PR and timeline?