Align earthaccess with STAC

[betolink]

Based on what the GIS/Pangeo communities are doing, I think earthaccess could be more useful if it was aligned with the STAC specification. An idea would be to support both CMR and CMR-STAC access patterns, @scottyhq suggested something like this already in #167

import earthaccess 

# we could even use NOAA's catalog or any other STAC endpoint, the default will still be NASA CMR (no STAC)
earthaccess.configure({
 "search": "https://cmr.earthdata.nasa.gov/cloudstac/",
 "protocol": "STAC"
})

# when we use STAC, earthaccess will be wrapping pystac_client
items = earthaccess.search_data(
    collections = ['ATL06', 'ATL08', '3IMERGHH'],
    intersects=dict(type="Point", coordinates=[-65.78, 45.79]),
    datetime = '2018/2024',
)

# earthacces will have to know how to deal with STAC results
earthaccess.download(items, "./data")

It would be interesting to explore intake-stac to stream the results and earthaccess will only provide the authenticated fsspec sessions.

This integration will make earthaccess more flexible but there are some things that we need to verify before trying to implement this:

• Will CMR-STAC results be consistent with the results provided by CMR?
• There is no top level NASA catalog, a user needs to know that ATL is in the NSIDC catalog and 3IMERGHH in GESDISC, can there be a way to avoid this so the users don't need to know where each dataset is?
• Can we interchangeable use DOI / short_name to search for data collections?
• What is the difference between say
  • https://cmr.earthdata.nasa.gov/stac/LPCLOUD vs https://cmr.earthdata.nasa.gov/cloudstac/LPCLOUD

[BriannaLind]

That makes a lot of sense to me! I have no idea of how to go about implementing that though!

[betolink]

Some answers from the CMR-STAC team

---

1. Results on CLOUDSTAC will have the cloudHosted attribute for their parent collections

   • /cloudstac/<provider> can be considered a subset of /stac/<provider>
   • in the case of LPCLOUD, if there are any collections NOT marked as cloudHosted, they should not show up in /cloudstac/LPCLOUD, but would be accessible in /stac/LPCLOUD

2. CMR-STAC is a STAC compliant wrapper for queries into CMR. There may still be some minor inconsistencies due to currently unsupported query mechanisms

   • e.g. a POST search {"query": {"eo:cloud_cover": {"gte": 50 } } may yield unexpected results as CMR only supports a min and max value, but not other comparison operators
   • in intersects queries with polygons containing holes, the holes are not honored presently and you would need to use Shapefile searches against CMR directly

3. Searching across all providers is something that has been discussed, but it not currently supported. Knowing there is a demand for it will help gauge the priority for adding that.

---

Based on this I think aligning earthaccess with CMR-STAC is doable and will bring great flexibility and re-usability to existing workflows.

[abbottry]

1. CMR STAC and CLOUDSTAC should be merged and a parameter should allow filtering for cloudHosted -- we're working towards that.
2. You are correct
3. This is also correct, its difficult to allow searching across ~2 Billion records so we're looking into ways to allow this for uses like STAC where its more necessary.

[scottyhq]

Thanks for documenting all these ideas @betolink !

@scottyhq suggested something like this already in #167

Yes, my suggestion was to support the same search keywords and values as the STAC API (e.g. intersects=dict(type="Point", coordinates=[lon, lat]) and translate those to whatever CMR needs if necessary, rather than supporting endpoints other than CMR

CMR-STAC is a STAC compliant wrapper for queries into CMR. There may still be some minor inconsistencies due to currently unsupported query mechanisms

Based on this, I can see why the above suggestion might be hard!

In practice I find this lack of full compliance super difficult as a user. I’ve tried to use CMR-STAC multiple times over the last couple years but unfortunately each time find that it falls short and I stop using it:

1. It’s really slow compared to directly searching CMR and getting JSON outputs
2. The API inconsistencies manifest as issues when people come from other STAC API endpoints like USGS Collection2 or Planetary Computer (see https://github.com/nasa/cmr-stac/issues)
3. STAC is most useful with a bunch of metadata properties per item (and extensions like eo,proj,sar,etc) but CMR-STAC only returns the basic datetime and polygon for most items

So I wonder instead of the proxy approach, what is preventing hosting STAC metadata directly in CMR (at least just for the new cloudHosted datasets)?

It would be interesting to explore intake-stac to stream the results and earthaccess will only provide the authenticated fsspec sessions.

Providing fsspec objects is nice for consumption by downstream tools. I'll caution that intake-stac worked well for the case of a arbitrary single file (CSV, HDF, TIF, etc) to give back a python object, but because searches can return pretty complicated collections of data, the process of returning a composite data object in an efficient and accurate way is pretty hard. You start getting into fsspec cache settings, and other settings based on the tool that ultimately does I/O (e.g. rasterio/GDAL). This is where development of intake-stac basically stalled. If you restrict the possible data formats it does become much more tractable. For example just dealing with COGs is complicated enough with stackstac and odc-stac :)

[scottyhq]

Hmmm, I just noticed the CMR API lists STAC as a supported return type https://cmr.earthdata.nasa.gov/search/site/docs/search/api.html#stac , is that actually using cmr-stac?

[abbottry]

Hi @scottyhq we worked on a feature a bit ago to support this but ultimately the format isn't fully compliant, for (mostly) that reason we're looking at removing that support in favor of a more concerted effort to improve CMR Stac.

[cboettig]

Hi @abbottry -- is the a decent place to report other issues about the EarthData STAC? I wanted to note that in addition to the issues mentioned above, the static JSON versions could be improved, e.g. for better consistency with stac-browser.

Note that currently stac-browser does not recognize either the pagination or the stac API (rendered example) makes it appear like there are only 10 entries.

As NASA is already providing both static JSON and dynamic API endpoints, I understand the API should be mentioned in the JSON using the conformance tag:

{
    "rel": "conformance",
      "type": "application/json",
      "title": "STAC/WFS3 conformance classes implemented by this server",
      "href": "https://planetarycomputer.microsoft.com/api/stac/v1/conformance"
    }

Examples of this can be seen in, e.g. on planetary computer stac (note the API box at top and the added search abilities that come with this; see source json for comparison).

[scottyhq]

It's been a while and I thought I'd revive this discussion. I'm still a bit confused by 1. NASA CMR already directly supporting STAC as an output type versus 2. The NASA-CMR STAC Proxy effort.

If the cmr-stac proxy is the future of development that would be good to know - as then earthaccess could use pystac_client as a dependency as @betolink describes above!

However, another possible approach is to just expose the STAC output type from CMR. I just opened an issue about that here nasa/python_cmr#81 . This would make it straightforward to take advantage of other great tools like pystac and geopandas for spatial operations, map exploration (#305) etc...

The example below would require a small change to python_cmr, but since earthaccess uses python_cmr should be easy to integrate?...

import geopandas as gpd
from cmr import GranuleQuery
import ast

api = GranuleQuery() 
search = api.parameters(
    point=(-105.78, 35.79),
    temporal=('2021-02-01','2021-03-01'),
    collection_concept_id='C2021957657-LPCLOUD'  # Required for STAC search
)
items = search.format("stac").get()
features = ast.literal_eval(items[0]) # python_cmr returns list with string-wrapped dict
gf = gpd.GeoDataFrame.from_features(features, crs='EPSG:4326')

[jhkennedy]

I'm still a bit confused by 1. NASA CMR already directly supporting STAC as an output type versus 2. The NASA-CMR STAC Proxy effort.

From the comment above, #221 (reply in thread), I think this implies that the STAC output type is (or will be) deprecated, and future efforts will revolve around improving the STAC proxy

@abbottry or @doug-newman-nasa can you confirm?

[doug-newman-nasa]

Any effort to use STAC with CMR should use the CMR STAC API. I'm loathed to use the word 'proxy' when describing it, this just adds to the confusion. :)

[scottyhq]

thanks for the insight @doug-newman-nasa!

I'm loathed to use the word 'proxy' when describing it, this just adds to the confusion. :)

Could you clarify why? Perhaps the readme needs updating, as it says up front "CMR-STAC acts as a proxy between the CMR repository and STAC API queries." https://github.com/nasa/cmr-stac?tab=readme-ov-file#cmr-stac

[doug-newman-nasa]

The README does need updating. Why use the word proxy? It is CMR's STAC API. It's a design detail.

[jhkennedy]

@doug-newman-nasa, thanks for weighing in! I agree, CMR STAC API is more appropriate as the proxy part is largely irrelevant to end-users.

[cboettig]

Perhaps orthogonal to the issue of a stac API, but this sounds really compelling to me: https://cloudnativegeo.org/blog/2024/08/introduction-to-stac-geoparquet/

Specifically, it's really nice to be able to see the entire catalog metadata as data like this, rather than having to pound the heck out of the API...