Sentinel-2 Collection-1 questions?

[vudongpham]

Hi David,

The Senintel-2 Collection-1 comes with baseline processing 05.XX:
https://sentiwiki.copernicus.eu/web/s2-processing

I wonder if you plan to add a parameter in the force-level1-csd to allow user to specify the baseline version when downloading Sentinel-2?

And most importantly, the new processing baseline products have improved geolocation accuracy, then is co-registration with Landsat data still necessary?

Dong

[davidfrantz]

Hi Dong,

why would introducing such a switch be helpful?

For the second question: good question... I am still unsure about this myself. I am hearing ambiguous things about this. What are your experiences with this?

David

[vudongpham]

Hi David,

Here is an example of the same image available for download with two processing baselines, how does FORCE handle this with force-level1-csd? Will both be downloaded from the google cloud storage?

I am not sure about the new baseline either. However, I tried to downloaded some images and visually compared with VHR images , the new ones seem to be geometrically corrected. This could be a major change for Sentinel-2 level2 processing, should a meeting with more people might be helpful?

Dong

[davidfrantz]

I see.

However, your screenshot is from CDSE, not from GCS, I assume. We would actually need a new downloader for CDSE.

Nevertheless, the issue is probably also apparent on GCS. Although it is likely only temporary, as I assume that they would purge the old data at some point (after the whole reprocessing is complete)

Have you simply tried it?

There is this comment in the code that kind of suggests that it could just work?

  # Filter metadata and extract download links
  # sort by processing date (+ baseline for s2), drop duplicates entries of tile + sensing time

Best,
David

[ernstste]

The comment is somewhat misleading, since we don't order by baseline. We decided to just go with the processing date since later processing date = later processing baseline.

For some reason, I'm getting the same duplicates from Dong's example when I run the command in docker. When running the script locally, the file produced by the older processing baseline is not in the results.

[ernstste]

Seems like the docker image uses mawk, which is a more minimal version of awk. The comparison in the command we are using uses associative arrays, maybe that's a feature that mawk is missing? I'm not sure.
Unfortunately I don't have the resources here to just build another docker with GNU awk (the more powerful version that I used locally) and see what happens.

[vudongpham]

Hi David,

I tested with the scene I showed above, here is bash script:

first_date=20170801
second_date=20170831
singularity exec ~/force_3.7.12.sif force-level1-csd \
-c 0,75 -d $first_date,$second_date -s S2A,S2B \
/home/phamv/edc_test/level1/metadata \
/home/phamv/edc_test/level1/s2 \
"/home/phamv/edc_test/level1/queue_$first_date"_"$second_date.txt" \
T33UVU

And here are downloaded results

S2A_MSIL1C_20170805T102031_N0205_R065_T33UVU_20170805T102535.SAFE
S2A_MSIL1C_20170809T100031_N0205_R122_T33UVU_20170809T100028.SAFE
S2A_MSIL1C_20170815T102021_N0205_R065_T33UVU_20170815T102513.SAFE
S2A_MSIL1C_20170825T102021_N0205_R065_T33UVU_20170825T102114.SAFE
S2A_MSIL1C_20170829T100031_N0205_R122_T33UVU_20170829T100026.SAFE
S2B_MSIL1C_20170807T101029_N0205_R022_T33UVU_20170807T101349.SAFE
S2B_MSIL1C_20170810T102019_N0205_R065_T33UVU_20170810T102245.SAFE
S2B_MSIL1C_20170817T101019_N0205_R022_T33UVU_20170817T101021.SAFE
S2B_MSIL1C_20170820T102019_N0205_R065_T33UVU_20170820T102346.SAFE
S2B_MSIL1C_20170824T100019_N0205_R122_T33UVU_20170824T100022.SAFE
S2B_MSIL1C_20170827T101019_N0205_R022_T33UVU_20170827T101312.SAFE
S2B_MSIL1C_20170830T102019_N0205_R065_T33UVU_20170830T102531.SAFE

So probably GCS has not updated the new data

Dong

[davidfrantz]

.. or the filter does not work as intended

You should have downloaded the metadata csv when doing the above. Can you grep for the ID of the new baseline to confirm the file is not available?

In any case, I am a bit worried that GCS won't update the whole collection (remember that they discontinued Landsat when collection 2 was released), which is why we probably should think about a new downloader from CDSE

[vudongpham]

Hi David,
A quick:

grep -i 'S2B_MSIL1C_.*_N0500_.*_T33UVU_.*'

returns only three images:

L1C_T33UVU_A005010_20180220T100520,S2B_MSIL1C_20180220T100029_N0500_R122_T33UVU_20231013T101345,GS2B_20180220T100029_005010_N05.00,33UVU,2018-02-20T10:06:06.835000Z,62132426,67.3302244622259,,2023-10-13T10:13:45.000000Z,52.84834368639136,52.262549788123366,14.89274266180146,15.144776536660824,gs://gcp-public-data-sentinel-2/tiles/33/U/VU/S2B_MSIL1C_20180220T100029_N0500_R122_T33UVU_20231013T101345.SAFE
L1C_T33UVU_A023743_20210922T100024,S2B_MSIL1C_20210922T100029_N0500_R122_T33UVU_20230120T120952,GS2B_20210922T100029_023743_N05.00,33UVU,2021-09-22T10:06:12.332000Z,49447099,99.9931610370603,,2023-01-20T12:09:52.000000Z,52.83539861811328,52.262549788123366,14.899335161881817,15.144730520801755,gs://gcp-public-data-sentinel-2/tiles/33/U/VU/S2B_MSIL1C_20210922T100029_N0500_R122_T33UVU_20230120T120952.SAFE
L1C_T33UVU_A004910_20180213T101118,S2B_MSIL1C_20180213T101119_N0500_R022_T33UVU_20230805T112149,GS2B_20180213T101119_004910_N05.00,33UVU,2018-02-13T10:16:00.229000Z,671728418,0.948421806137907,,2023-08-05T11:21:49.000000Z,53.24969857188909,52.287183926668725,13.50078932347948,15.14612756680773,gs://gcp-public-data-sentinel-2/tiles/33/U/VU/S2B_MSIL1C_20180213T101119_N0500_R022_T33UVU_20230805T112149.SAFE

[vudongpham]

A search in 2018:

first_date=20180201
second_date=20180220
singularity exec ~/force_3.7.12.sif force-level1-csd \
-c 0,75 -d $first_date,$second_date -s S2A,S2B \
/home/phamv/edc_test/level1/metadata \
/home/phamv/edc_test/level1/s2 \
"/home/phamv/edc_test/level1/queue_$first_date"_"$second_date.txt" \
T33UVU

returns dublicated images with two baselines:

S2B_MSIL1C_20180216T102059_N0206_R065_T33UVU_20180216T122659.SAFE
*S2B_MSIL1C_20180213T101119_N0500_R022_T33UVU_20230805T112149.SAFE
*S2B_MSIL1C_20180213T101119_N0206_R022_T33UVU_20180213T135340.SAFE
S2A_MSIL1C_20180208T101151_N0206_R022_T33UVU_20180208T135551.SAFE
S2B_MSIL1C_20180206T102209_N0206_R065_T33UVU_20180206T140356.SAFE
S2A_MSIL1C_20180205T100211_N0206_R122_T33UVU_20180205T152114.SAFE
S2A_MSIL1C_20180201T102241_N0206_R065_T33UVU_20180201T140542.SAFE

[davidfrantz]

Have you tried 05.11?

grep N0500 metadata_sentinel2.csv | wc -l
43802
grep N0511 metadata_sentinel2.csv | wc -l
444156

[vudongpham]

Yep, I produced similar numbers. But we still have the dublicate issue mentioned above

[vudongpham]

And here is an example of this area (52.4925413,13.8593035):

S2B_MSIL1C_20170830T102019_N0205_R065_T33UVU_20170830T102531 (B04, scaled to 1000 - 3000)

S2B_MSIL1C_20170830T102019_N0500_R065_T33UVU_20231014T053638 (B04, scaled to 1000 - 3000)

[davidfrantz]

yep, I am aware. But the question is still: is it good enough, especially seen through time

[vudongpham]

Hi @davidfrantz and @ernstste

Just want to open this discussion again. I actually did some small experiments, and found that the new baseline Sentinel-2 data is quite robust and FORCE co-registration might not be necessary for this data. You can see the results here:

https://vudongpham.github.io/blog/2024/Sentinel2-processing-baseline/

Now, this indicates that we need to find a way to download the newer version data. I somewhat managed to download data directly from https://dataspace.copernicus.eu/ using OdataAPI, but it is not very straightforward. Basically, registered users can request token for download, but the token expires every 60 minutes, so I have to tweak this to download large number of images.

Anyway, hope this could be some food four thoughts :D, and I am looking forward to the next FORCE meeting!

Best,
Dong

[davidfrantz]

Hi Dong,

thanks for this analysis. This is looking very interesting! Although I have the feeling that the co-registered new baseline still looks more stable than the original, no?

Regarding CDSE: we urgently need to do something with regards to the downloader.

I will open a new issue on Monday on this topic and explain.

Cheers,
David