Community meeting - June 2nd at 1 PM Eastern

[abkfenris]

Our next community meeting is coming up! Folks seem to have a good number of questions on how to use Dask effectively with Xpublish, so how about we talk about that?

If you wish to join and don't have an invite, please reply and we can add you.

[mpiannucci]

Darn, I can't make this one either as I am traveling again. Hoping to follow any discussion because I think the dask and xpublish integration is really important for building web services

[abkfenris]

Attendees:

• Alex Kerney / Gulf of Maine Research Institute / @abkfenris
• Joe Hamman / Earthmover / @jhamman
• Xavier Nogueira / LimnoTech / @xaviernogueira
• Jonathan Joyce / RPS Group / @jonmjoyce
• Kyle Wilcox / Axiom Data Science / @kwilcox

Agenda and Notes:

• OpenDAP weirdness depending on the client
  • Opening .dods with OPeNDAP clients xpublish-opendap#18
• Dask
  • Links:
    • https://examples.dask.org/applications/async-web-server.html
    • https://examples.dask.org/applications/async-await.html
  • Jonathan: Hard/slow to connect to a remote cluster and keeping the dependencies in sync
  • Kyle
    • Spin up an Xpublish server with a sidecar cluster per dataset
    • Throwing big servers at the problem
    • We'll need to figure out how to work with Kubecluster and other ways
  • Joe: Has anyone tried async computation? We should be able to delgate it
  • Xavier: Async is the next thing he's trying
  • Joe: FastAPI and Dask can be async, but there isn't much info on how to use Xarray async
  • Jonathan: How do you manage remote state
  • Xavier: Mixing sync/async
    • https://anyio.readthedocs.io/en/stable/threads.html#calling-synchronous-code-from-a-worker-thread
    • https://simonwillison.net/2020/Sep/2/await-me-maybe/
  • Joe:
    • Could await Dask tasks as coroutines. Xarray has no native async code for
    • It would be great to make a blog post about using Dask with Xpublish
    • Describing the deployment methods
      • Single server with Dask sidecar
      • Autoscaling servers and dask clusters
    • Hard to keep the cache alive with ephemeral dask clusters
  • Jonathan
    • Is there enough calcuations for caching in Dask, or should it be on the IO path?
  • Kyle
    • Two main access patterns
      • Maps
      • Timeseries
      • Dynamic dataset that invalidates after 10 min for map data
  • Xavier
    • Experimenting with chunk sizes in other servers, not effiencent with non-coordinate queries
  • Kyle
    • Keep a tree in memory for querying against
      • https://github.com/xarray-contrib/xoak
  • Jonathan
    • Starting to work more with irregular grids
  • Kyle
    • Not a ton of different grid types
    • Can make some perfomant code that can target each type of grid
    • Try to make everything work with cf-xarray
  • Joe
    • Haven't benefitted yet from the Xarray flexible indexes refactor
    • Should be able to write indexes that can be more perfomant and may to multiple variables at the NetCDF level
    • Benoit Bovy did the flexible index refactor and wrote xoak

[abkfenris]

After our discussion I doodled with Dask plugins: xpublish-community/xpublish#208 I'd love some feedback as I haven't needed to reach for Dask yet with my Xpublish servers.

[mpiannucci]

This looks really cool. I'll definitely give this some testing