Quilt: Reproducible Data Dependencies for Python

[rabernat]

I just stumbled across this post on the jupyter blog
https://blog.jupyter.org/reproducible-data-dependencies-for-python-guest-post-d0f68293a99

The quilt project seems to be aimed at solving many of the problems related to data discovery we have been discussing:

Quilt hides network, files, and storage behind a data package abstraction so that anyone can create durable, reproducible data dependencies for notebooks.

https://quiltdata.com/

It's a commercial product, but they have open sourced the building blocks.

By default Quilt store data in the registry at quiltdata.com. Alternatively, you can host your own registry by running the open source containers, then using quilt config to point clients to your private registry.

They seem focused on tabular-style data. But nevertheless, it's probably worth looking into this.

[shoyer]

I've chatted a bit with one of the Quilt founders.

I suspect they would be happy to add netCDF/xarray support to the open source client if there's demand for it (especially if they get a pull request!).

[akarve]

I'm a developer on Quilt. Happy to collaborate.

Unstructured and semistructured data are also supported by Quilt (e.g. large image corpuses, geojson, etc.).

In next week's quilt module we'll provide a callback that allows users to provide their own data readers (a.k.a. "deserializers"), so that packages with the xarray format (which can already be stored in Quilt) can be read out with custom libraries:

import some.geo.reader as func

from quilt.data.pangeo import ourdata

x = ourdata.foo.bar(asa=func)

We're also working on community-powered specs for reproducible data. If you'd like to be included in the discussion let me know.

[jgerardsimcock]

@akarve Can we already build netcdf-based packages in Quilt? If so, can you point to an example or documentation on how to do so?

[akarve]

In theory, yes. You can put any bits in and then use the asa= callback to call a custom deserializer. We're going to make deserialization even easier in the future. It seems people are already using Quilt + netcdf.

In practice, if you provide me with an example of data-roundtrip that you'd like to accomplish with Quilt using netcdf data, then I can try that for you and see how it might be improved.

[akarve]

To give you a concrete example, supposenetcdf() is a deserializer, you could do something like this:

from quilt.data.foo import bar
bar.baz(asa=netcdf())

[shoyer]

In practice, if you provide me with an example of data-roundtrip that you'd like to accomplish with Quilt using netcdf data, then I can try that for you and see how it might be improved.

We would typically do this with xarray, e.g.,

ds = xarray.open_dataset(path)  # netCDF file -> xarray.Dataset
ds.to_netcdf(path)  # xarray.Dataset -> netCDF file

[akarve]

Thanks. I can confirm that it is possible to round-trip xarray Datasets to and from a Quilt package. Here's a notebook.

It takes more lines of code than I'd like to complete this round trip, but future versions of Quilt will get this close to two lines of code.

[martindurant]

Intake now officially being circulated: https://www.anaconda.com/blog/developer-blog/intake-taking-the-pain-out-of-data-access/
Intake-xarray almost supports streaming from an Intake server :)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it had not seen recent activity. The issue can always be reopened at a later date.