STAC harvesting

[pandzel-zz]

This pull request provides support for STAC harvesting.

STAC repositories are organized in catalogs which in turn could hold other sub-catalogs and/or items. Entire structure is expressed through JSON files i.e. one catalog which is defined as JSON has an array of URL's pointing to sub catalogs and separate array of URL's pointing to items. STAC broker crawls catalog structure and fetches fetches items, then it publishes it accordingly.

Testing

Current implementation has been tested with the following repositories:

• https://landsat-stac.s3.amazonaws.com/landsat-8-l1/catalog.json,
• https://sentinel-stac.s3.amazonaws.com/sentinel-2-l1c/catalog.json,
• https://s3.amazonaws.com/spacenet-stac/spacenet-repository.json,
• https://raw.githubusercontent.com/cholmes/sample-stac/master/stac/catalog.json,
• https://3dep-stac.s3-us-west-2.amazonaws.com/catalog.json,
• https://ebirdst-data.s3-us-west-2.amazonaws.com/catalog.json.

Note, some other mentioned catalogs were unavailable at the time of development/testing:

• https://gptogc.esri.com/stac/catalog/c34fd380d16f40a7bb7995ac4d7ab8de/catalog.json,
• https://cbers-stac-0-7.s3.amazonaws.com/catalog.json,
• https://iserv-stac.s3.amazonaws.com/0.6.1/catalog.json.

Important !!!

By default Elastic Search refuses to accept documents with more than 1000 properties (fields) in it. More about that feature could be found: https://www.elastic.co/guide/en/elasticsearch/reference/master/mapping-settings-limit.html. If this is a case then the workaround is to increase that limit. It could be done using freely available tools like POSTMAN or curl. The following example would increase that limit:

POST http://localhost:9200/metadata/_settings
{
  "index.mapping.total_fields.limit": 3000
}

[pandzel-zz]

@zguo Please, review