Debug LS and S2 STAC iteration

[banesullivan]

Follow up to #44

[codecov-commenter]

Codecov Report

Merging #50 (1b13b45) into main (4b2f7b9) will decrease coverage by 0.06%.
The diff coverage is 50.00%.

@@            Coverage Diff             @@
##             main      #50      +/-   ##
==========================================
- Coverage   74.03%   73.96%   -0.07%     
==========================================
  Files          34       34              
  Lines         697      699       +2     
==========================================
+ Hits          516      517       +1     
- Misses        181      182       +1     

Impacted Files	Coverage Δ
rgd-watch-client/rgd_watch_client/plugin.py	91.66% <50.00%> (-3.79%)	⬇️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4b2f7b9...1b13b45. Read the comment docs.

[banesullivan]

Update: we need to reduce our queries because stac-server has some serious limitations.

The problem and the error that we are experiencing (which index.max_result_window controls) is that stac-server is improperly using the elastic search scroll API. This seems like a serious flaw in stac-server's implementation and a significant limitation of using stac-server.

Specifically, from the Elastic Search API docs, there is this note:

We no longer recommend using the scroll API for deep pagination. If you need to preserve the index state while paging through more than 10,000 hits, use the search_after parameter with a point in time (PIT).

ref https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results

This error is indeed missing functionality / a bug in stac-server itself and has already been filed with stac-server here: stac-utils/stac-server#111

To work around this until stac-server can resolve the issue is partition the data using a filter on the DateTime (e.g. per day) with https://api.stacspec.org/v1.0.0-beta.4/item-search/#operation/getItemSearch

[banesullivan]

I've updated the LS and S2 ingest scripts to use pystac-client - we'll see it's able to iterate over the whole collection in a reasonable amount of time... if not, will try to parallelize in a follow-up PR as we parallelize the POST requests to RGD

[banesullivan]

@matt-bernstein, any thoughts on a minimum ingest date for Landsat?

I'm waiting to see how long it takes to ingest, so assuming the last 8 years of data doesn't take more than an hour or so to ingest, there shouldn't be much of a limitation on our end

[banesullivan]

stac-server's response times are proving to be slow. We're seeing ~5 seconds to retrieve a single day's items. If we want to iterate over the last 5 years (arbitrarily chosen but we'll def want more data than that), it would take over 2.5 hours (365 * 5 * 5 / 60 / 60 = 2.535) if done serially just to retrieve the items