Community meeting on May 5th at 1 Eastern

[abkfenris]

Our next community meeting is coming up on May 5th at 1 PM Eastern.

If you wish to join and don't have an invite, please reply and we can add you.

[xaviernogueira]

I have some stuff to share briefly!

Essentially I've completed a catalog to server prototype (Intake implementation fully done, STAC will be shortly) that digests a catalog and spins-up a FastAPI server via Xpublish that only accesses data once requested (as opposed to loading into xarray upon initialization) while maintaining catalog pathing (i.e., 'catalog/sub_catalog1/sub_catalog2/datasets/{dataset_id}').

The interface with catalog types is abstracted, so one could implement other catalog schemas interchangeably.

While I am happy with the code I've written, I do think a Xpublish native catalog plugin could be better. I was forced to mount different Xpublish.Rest.app instances on-top of an existing FastAPI app instance. I have some thoughts along these lines...

Is there a place to sign up for a spot?

[abkfenris]

I haven't started an agenda yet, but it'll end up in the same doc: https://hackmd.io/@LIrAVhP6Rzm1d_vhTynHZQ/ryx7rvx5-h

I think this should be manageable to build a native plugin with app routers, though it may be messier to handle varying levels of catalogs.

• One way to do it would be via path parameters
• A better way might be if the catalogs could be iterated on startup, and to make explicit paths for each one of them, or at least the levels of catalogs.

Both would then load dataset routers on those paths and replace the datasets dependency. I've drafted a bit more on how that might be done here: xpublish-community/xpublish#92 (comment)

[xaviernogueira]

The second option is sort of what I implemented just with mounting multiple servers. Basically I used recursion to search the catalog, wrapped each level in a custom dataclass that stores sub-catalog names, any datasets available, parent catalog, and the raw catalog object (i.e., Intake.Catalog or pySTAC.Catalog). Then where there are datasets I instantiate a dataset provider plugin that knows how to convert to xarray from the catalog object and add that to a xpublish.Rest.app instance (along with the opendap plugin). Where there are no datasets a FastAPI router is made to serve up the list of sub-catalogs and other info. Then I just mount the xpublish apps and FastAPI routers using the appropriate pathing.

Not the idea solution but it works.

I agree a native solution seems 100% possible, and path parameters are likely the way to go. I'd be happy to help to the extent I can with some design constraints in place.

[abkfenris]

Attendees:

• Alex Kerney / Gulf of Maine Research Institute / @abkfenris
• Jonathan Joyce / RPS Group / @jonmjoyce
• Joe Hamman / Earthmover / @jhamman
• Xavier Nogueira / LimnoTech / @xaviernogueira
• Kyle Wilcox / Axiom Data Science / @kwilcox
• Shane St Savage / Axiom Data Science / @srstsavage

Agenda and Notes:

• Xavier demo
  • Build partially as a learning exercise, trying to figure out how things work
  • Take a intake catalog or a static STAC catalog
  • Geared towards working with Zarr data, so it requires the cloud storage STAC extension
  • Working on USGS projects
    • Modernizing and standardizing infrastructure
  • Findings
    • Transforming dataset provider
• Working towards releasing 0.3 #183
  • Docs: Docs revamp xpublish#180
  • Release cadence? When stuff happens, release on little changes
• Axiom update
  • DatasetConfig plugin, loaders, metrics
  • SFBOFS - Full CO-OPS OFS model
  • SFBOFS - Last 24 hours of CO-OPS OFS model for mobile phone app
  • CIOFS - Actively running forecast model (auto updates with new data)
  • https://github.com/axiom-data-science/xpublish-host
    • Includes lots of metrics work
  • Put it into production for two APIs
    • Serving models to a mobile app
      • Last 24 hours of data
    • Returning binary parquet
    • Getting 400 Mb/s
  • How do we configure servers in a standard way?
    • DatasetConfig plugin
    • Auto reloading datasets
      • Avoiding reloading in the dask cluster
      • Cache invalidation
      • Scales to multiple workers
  • Currently running a 30 year hindcast, but researchers wanted data as soon as possible
• Volunteers
  • Answer Q&As
  • review PRs,
    • Joe
  • Work on documentation/examples
    • Jonathan
    • Xavier (OpenDap)
  • ...
  • both on Xpublish and across the ecosystem
• Bigger issues to wrangle
  • Caching
    • RPS trying to cache tiles for WMS
  • Dask
    • How is async and dask requests working?
      • Axiom: all sync endpoints
      • In theory that should help with concurrent reads or requests
      • Most useful in Zarr for high latency stores 'listing all the keys in a S3 bucket'
  • Catalogs