Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Global storage transformers #182

Closed

Conversation

rabernat
Copy link
Contributor

As discussed in the ZEP call last week, this is a first pass at creating a "global storage transformers" concept, to be applied at the store (rather than array) level.

There are a few things that need cleaning up here:

  • Double check that all RST references point to the right place
  • Add more detail and example use-cases of both global and array storage transformers
  • Add an example of a global storage transformer in zarr.json

But before doing that, I thought I would solicit some feedback on this. Does anyone see any fundamental problems with this concept? My motivating use case is #154.

My main open question is: when both array and global storage transformers are defined, what determines the order in which they are applied? The only feasible choices appear to be to apply all the global transformers either before or after the array transformers. Should we make this an option in the protocol?

@jbms
Copy link
Contributor

jbms commented Nov 25, 2022

I think array storage transformers should definitely apply before global storage transformers, since the array metadata itself is obtained through the global storage transformers.

In fact though it might make sense to also support group-level storage transformers, which would stack with the nesting of groups on top of any global storage transformers.

docs/core/v3.0.rst Outdated Show resolved Hide resolved
@jstriebel jstriebel added the core-protocol-v3.0 Issue relates to the core protocol version 3.0 spec label Nov 28, 2022
Co-authored-by: Stephan Hoyer <shoyer@google.com>
@jstriebel
Copy link
Member

Depends on #192

@joshmoore
Copy link
Member

Now conflicting.

@jstriebel
Copy link
Member

@rabernat Could you adapt this PR to cover group storage transformers for the current spec?

@rabernat
Copy link
Contributor Author

On today's call, we decided that we would defer this to an extension after the core v3 spec has been finalized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core-protocol-v3.0 Issue relates to the core protocol version 3.0 spec
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

5 participants