Design of the SAS for Sentinel-1 CSLC in radar coordinates

[hfattahi]

I just looked at PR #1 and I though its easier to start a discussion before getting to more details. Also an excuse to explore discussions feature for this repo :) .

What is the exact need/problem that we want to solve:

Let's assume we have N S1 frames of SLCs in the archive which cover an area of interest. We need to coregister each burst of the stack with geometrical algorithm + ancillary corrections

• The coregistration is done in radar coordinates
• The first version of the workflows does not need to be worried about the ancillary corrections

Let's assume we have a static local stack with full control (ignoring the real world with data systems component for a second). In this case the efficient approach would be to

1. select reference burst, run rdr2geo for the refence burst and archive that burst and its geometry
2. Loop over all secondary bursts ,
   a. use geometry of the reference burst and run geo2rdr to estimate offsets
   b. resample the secondary burst and archive it

This is essentially what we have done in the isce2 stack processor (with geometry coregistration approach)

How it may be operationalized within our data system with current DACC API functionality that provides zip files and not bursts.

Let's assume a burst map of Sentinel-1 exists. A burst map is simply a database of burst IDs with their geometries such that one can query the IDs extract the geometry and query the ASF archive to get the required SLC zip files. A prototype to create such database exists here

1. a control manager such as PCM chooses a list of burst IDs
2. from the busr ID static databse, the bounding box that covers the bursts is chosen
3. The archived coregistered burst is checked to find out the dates to query. if any coregistered burst found, then return the reference coregistered burst. If no archived coregistered burst is found, then the isReference=True is set indicating that this is the reference date of the stack.
4. PCM queries the archive and returns the zip files for the date that covers the bounding box
5. PCM queries the orbits and returns the orbit files which cover the zipl file(s) for the date of interest
6. PCM writes the SAS/PGE configuration file which @vbrancat has started a design for already here Co-registered S1 SLC schema #1
7. PGE starts and then SAS runs

In this scenario, the SAS will be given:
a. one or more zip files for one date
b. the reference coregisterd burst (empty for the reference burst)
b. the orbit file that covers the date of the zip file(s)
c. the DEM file
d. the list of burst IDs to be processed
e. the list of output coregistered burst product names
f. the isReference flag
g. other possible input parameters as already started by @vbrancat here #1

   Loop over the list of busrts passed by the run config:
       If the input is for the reference date (i.e., ```if isReference==True```) then:
            run rdr2geo and archive the burst, its metadata and its geometry files. 
       else (i.e., the input is secondary images to be coregistered)
            extract geometry from the refernce burst passed by config file
            run geo2rdr
            run resample
            archive the coregistered burst and metadta

Note: in this design the SAS always runs on one date**

How to test the end2end workflow on a stack

If we agree with the design explained above, we should develop the SAS as explained above. Basically no reference and secondary. always one date is provided in the config file.
Since we don't have a PCM system during the development we should simulate one with simple scripts. Similar to what is done in isce2 stack processor, we can write high level scripts to manage processing for testing purposes. These scripts should create the SAS run configuration files with the final format of the schema as suggested here here #1 .

@vbrancat @yunjunz @LiangJYu this is how I have the whole flow in mind. We don't need to necessarily enforce this design but I hope this will start a discussion and will lead to a final design.

[vbrancat]

Sounds good.
Regarding the input section of the runconfig, I am not entirely sure why we need to have an entry for the co-registered burst/s and a flagisReference. It should be sufficient to assign the co-registered burst (IDs?) to have the workflow running on the secondary bursts.

[hfattahi]

Yes I agree. The flag is redundant. I think I was trying to make it very clear in the discussion above about the reference date. As far as we are fine with the overall end2end flow, the details may still change during development/implementations.

[vbrancat]

Summarizing the discussion on the design of the S1 CSLC workflow.

The interface (#1) of the S1 CSLC workflow for a control manager like PCM will have:

• List of file path to one or more SAFE (zip) files (required)
• List of file path to one or more orbit (EOF) files (required)
• List of (unique) burst IDs to process (not required)
• A schema subgroup named reference_burst with a required Boolean flag (is_reference) indicating if the burst to process is or not a reference burst. The subgroup also includes the file_path (not required) to where the reference burst is stored in case the flag is_reference is False.

One SAS will handle the processing of reference and secondary bursts.

If the flag is_reference is set to True, then given the SAFE file, orbits, andburst_id, the SAS will:

• extract the reference burst and store it to sas_output_file
• run rdr2geo (topo) to get the x, y, z topo rasters and store them.
  Still to be decided if the topo rasters will be stored with the reference burst or in a separate location.

If the flag is_reference is set to False, the workflow will use the archived x, y, z topo layers to run

• geo2rdr
• resample and stored the coregistered secondary burst to sas_output_file

[hfattahi]

@LucaCinquini @riverma @hookhua @hhlee445 and @collinss-jpl, the discussion above is very relevant to the data system and in particular to PCM and to the topic that algorithms with shorter run times reduce risks during operation when using AWS on the spot market.

In the design that our team has considered and discussed here the SAS will accept a list of burst IDs. The data system can choose to feed only one burst at a time. Therefore I think for coreg-SLC-Sentinel-1 the SAS design is already compatible with the idea of processing individual bursts at a time.

Regardless if the data system will get access to individual bursts during operation or if the access will be on zip files, the design above should still allow to run one burst at a time. However, I think if we want to scale jobs with individual bursts, then the value of an API which allows burst access instead of frame access will be evident vs the current ASF's API which only allow for a frame query and access. Imagine if we will only have access to a frame. That way if we pull one frame, we can generate ~30 burst CSLC-S1 on one AWS instance or on 30 instance. With current ASF API, the farmer will result in one time data movement of one frame (~30 bursts) to one instance and the latter would mean 30 times moving the same frame (i,e, 30 bursts) to 30 instances.