-
Notifications
You must be signed in to change notification settings - Fork 28
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Extension for separate user attributes #72
Comments
Some discussion around this on today's community meeting. Further discussion on the spec meeting needed. |
I want to put in a word for keeping the attributes in a separate document. |
I'm wondering if it should be a core feature to have the |
If there are no objections, I'd leave this as an extension (TBD) for now. |
@DennisHeimbigner - do you think that your issues could be addressed via an extension? |
The meaning of "extension" is completely undefined, so I have no idea if it is possible or not. |
Thanks for your feedback Dennis. Are you saying that the extension points in the V3 spec are not well defined enough? Any suggestions on how to make that better? |
Short answer: no they are not well defined enough.
In the worst case, we are talking about an arbitrary rewrite of the JSON metadata, which is not desirable, so we also need to somehow limit what kind of transformations are needed. Needs more thought. |
Another possibility to address this. |
Another thought about extensions. |
This might be relevant: https://softwareengineering.stackexchange.com/questions/264706/xslt-equivalent-for-json |
Agreed that none of the existing "extension points" are applicable for this purpose. However, at least as I see it, an "extension" more generally is really any future change to the spec that maintains some amount of compatibility with the existing spec (as otherwise we would just consider it an independent spec rather than an extension), and that would certainly be able to support this feature. It sounds like you are proposing a more general mechanism for moving arbitrary subtrees of the JSON to separate keys in the underlying store; that is an interesting possibility. I can also imagine a more limited feature specifically to allow user-defined attributes to be specified separately. |
My worry is that the notion of "extension" has no current definition. It needs at least a semi-formal |
Yes, perhaps it should be clarified. It seems to me that a typical approach for creating an extension would be to first create an (experimental) implementation, then create a ZEP proposal to standardize it. For extensions outside the realm of the indicated "extension points", any changes to the zarr.json metadata file should take into account interoperability issues:
|
Actually, this raises a question. Suppose I want to add a non-standard key to, say, a configuration dictionary. |
That is a good point. Currently the spec does not specify this, and therefore it is left up to the definition of the specific configuration, and currently those aren't very explicit about this. Tensorstore currently only supports |
In the current v3 protocol spec draft, an array metadata document contains both core metadata (like array data type, shape, chunk shape, etc.) and user attributes. E.g.:
The same is true for groups, i.e., both core metadata and user attributes are stored together in a single group metata document, e.g.:
The zarr v3 approach of storing core metadata and user attributes together is similar to N5.
Note that this is different from the zarr v2 protocol, where the core metadata and user attributes are stored in separate documents.
Raising this issue to surface any comments or discussion on this approach.
Some possible reasons for bringing core metadata and user attributes together into a single document:
Some possible reasons for separating core metadata and user attributes into different documents:
The text was updated successfully, but these errors were encountered: